I would definitely not let it put you off.
You are as free as he is to look for customers as you see fit. If he does a good job, his customers will stick with him and he will have nothing to worry about. He's probably just mouthing off 'cos he does a lousy job, and this is about to be exposed by you doing a good job!
Stick with it, guys like him are full of wind and wee wee!
John.