Why Mandarin ? (Updated)

Strangers that I meet like to talk to me in Mandarin. They always assume that i can speak it. Like today, i went to a grocery shop near my house to buy a loaf of bread, the boss who has known me since i was 13 suddenly spoke to me in Mandarin. When i asnwered in Cantonese, he look quite pissed off, maybe coz i was suppose to reply in Mandarin ?

Do i have a chinese-educated type of look or do i look like i'm from China or issit because of my dressing? Or what ....???? I am actually quite stumped. This has happen to me for like ages....

Hmmm.....maybe people just assume that Chinese are suppose to know Mandarin...
Nah, i don't think it's that.....

There was once when I was in a chinese restaurant with a group of friends, and the lady taking our orders was happily conversing in Cantonese with my friends but when she turn to me, she started ching chong ching chong in mandarin. Oiiiii................ !! WHY ???

Can anyone please enlighten me ??? I don't want to piss people off lah....
They probably think that I am not proud that my ancestors are from China (err...i think they are from there) or I want to act high-class and answer in English.

Is it my hairstyle ?? Should i change it ?

Updated:

I am not annoyed at all at being perceived in this way. In fact, I am quite amused and at the same time, baffled.

I agree that it is a huge advantage to be able to read/write/speak Mandarin. If i had known the language, I wouldn't have missed out on a few job opportunities that would have taken me to HK/China/Shanghai (all expenses paid somemore - sigh...bummer...)

Perhaps like blue commented, it's time to take up Mandarin lessons. Any good tutor to recommend?