I hate to say this, but no one likes a smart arse. Your witty rejoinder may make you feel good, but it just makes you look like a prat in their eyes.
If someone disses you, be polite in return.
If their poor lives are so miserable that they have to take their pathetic pleasures in giving you a hard time, they deserve your sympathy, not your venom.
Think what you like, but always speak well.
And (as I always say to my kiddies) ....
If ya can't say owt good, say nowt.