His number came up even gave his name! But will going to police not make it worse!
Phone the police! Phoning the police will not make it worse; I've been in a similar - but worse situation - a couple of goons threatened Wor Lass and I face to face - not over the telephone.
Oh, him phoning you shows that he's full of urine mind, if he were serious, he'd have visited you BTW; so I wouldn't worry too much.
But the police discussed all possible angles, including the 'making the situation worse'; they were very good, I can't fault the police we dealt with.
Now, phone the police, it's mega important you do; two or more incidents can be classed as harassment (though there's probably more since you've been threatened), but for these to be counted as incidents means that you must report the first one.
And FFS keep canvassing. Don't jack it in because some coward has threatened you over the phone. Transform any negative feelings you feel into a strong determination that you'll stick with canvassing and improving your business. This is mega important. The nob 'eads that do this sort of thing don't tend to last anyway. It takes a bit of character to stick this business.