Should I help an aggressive panhandler?

I recently encountered an aggressive panhandler. To be exact, he was a peddler who was selling a small good at a profitable price. I didn't end up buying anything from him because I simply didn't need the small good he was selling; but now I'm conflicted as to whether I should've just bought the item to help him out. On one hand, I recognize that he is making a tough living and I admire him for that. I could've easily spared a few dollars to buy the item and help him out. To be honest, I really wanted to buy the item out of pity but I couldn't reason why I should treat this man differently from any other salesman I come across. All beings are suffering, and material wealth is illusory and insignificant at the end of the day. I am confused as to whether not giving (through refusing to buy from the peddler) is just me being stingy, and this is my attempt to rationalise my actions, or indeed there is no point in saying 'yes' to every salesman I come across selling something I don't need just to help them out with their livelihood. Just to clarify this man was clearly not a drug addict or alcoholic who would use the money to feed his addictions, probably just an honest man trying to make a living. I guess what I'm trying to ask in essence is from a Buddhist perspective, **should we treat those who are poorer and 'worse off' better by accommodating them than we do to those who are presumably 'better off'?** I understand that we should give whenever we can, but if the act of giving is accompanied by the moral high ground I take by deeming this man as more deserving of my sympathy, is it really the way I want to give? On another note, this man was also aggressively trying to guilt trip me by saying that I could just give him money and it wouldn't make a difference to me but I didn't think this was relevant because I'm sure he has his reasons for any malevolence. **I'm more troubled by the fact that I could've helped him, but I didn't simply because I didn't feel I should judge him differently from other salesmen and I really didn't need the small items he was selling.**