Hi Ruchi,
Was wondering what your thoughts were on this. 
The woman that I work for, who is an unaffiliated Jew, went into the local kosher takeout place yesterday to pick up an order. I go out socially with her and some other friends once a month. They are so respectful and accommodating and want me to be able to eat. They either order from a kosher restaurant or check with me before they buy something from the grocery, and serve on all paper/plastic.

So she asks me in front of the other women last night to explain to her why observant Jews seem to be so unfriendly. She goes on to say that she was waiting at the restaurant to get her order and there was a man with his wife and kids also waiting at the counter. She said in non-Jewish restaurants (elevators, bank lines, etc.) people say hello or might make small talk. She said the people at this kosher place were so unfriendly.
She typically dresses VERY conservatively. She happened to have a sleeveless dress on yesterday with a somewhat plunging neckline, which was out of character for her. So I explained to her that religious men try to be careful about having too much conversation with other women.
I have another friend who is the receptionist at my other office who asked the same thing about a religious man who comes in and barely (if at all) looks at her. If you are not observant, you don’t get this at all. It just seems flat out rude and then these women associate that behavior with Orthodox Jews across the board and probably mention it in conversation to their other friends.
So I understand and value men not making too much conversation with another woman (especially if she is not dressed very modestly) but it affects us religious Jewish people as a whole in such a negative way sometimes. I don’t have an answer. Do you?