I once had a lot of trouble from dole cheat window cleaners. Undercutting me by half and claiming I had packed it in and sold my round to them, etc. I started carrying a camera in the van, and if I saw them working I would take a picture, making sure they saw me.
They would ask me what I was doing, but I didn't get involved in arguments, just walked away. They got the message, moved on and I didn't get anymore trouble.
If you get any more people threatening you, or even the same one again, make sure you have a camera with you. That way you can show the police a photograph of the person using threatening words and behaviour towards you. It's an offence, even if they don't attack you and they can be prosecuted and could even get a prison sentence if the threats are bad enough.
These people are usually cowards, very brave when they think no-one knows who they are, but will not be so brave if they think you can identify them.
Don't play their game - if you threaten them back you could be prosecuted yourself, especially if there are more than one of them to witness your threats.