HMRC do like to try and bully sole traders and small businesess. Don't fall for it, stick to your guns find out where the discrepancy is and don't let them get into your head. You are an easy target for them.
I have been dealing with a £15k liability due to the exact same circumstances as you with hmrc since 2007 and trust me they like to try to make you worry, I have been paying them back what I can and will prob contunie to do so for the next 10 years feel like i'm chasing my tail sometimes (with all the additional liabilities that have been added since). But I am paying them, and I will get back on top eventually, its this that drives me forward so good comes out of it.
It all came to a head last year, they sent someone round to me who was very persistant and serious, I politely invited him in, offered him a cup of tea and told him you got two options, make me bankrupt, and get nothing (which is fine by me as you become tax exempt during this time) or let me keep paying; and stop trying to intimidate me as it wont work. He left with tail behind legs agreeing that there isn't and awful lot that can be done and they are just going to have to accept my offer. he did say that there was still that risk of being declared Bankrupt by them, but I said OK whatever, but you cant stop me from working or continuing to grow my business if you do can you? No he said, and you cant prevent me from paying my bills or providing for my family, 'No' he said, in fact I said if you do make me bankrupt you loose out and I gain a lot by not having to pay back what I owe you, ' That's right' he said so where is the sense in it? I said. If I have a business that is growning and making money year on year, and I am offering you what I can to pay back arrears, and increasing that amount as the business develops, im doing everything I can in my powers to prove that I am not avoiding payment. he then clicked on his PDA and noticed the £400 payment I made to him last 3 days ago had arrived in their acc, and I had just done another £100 on that day too. He had another cuppa and left.
Remember they are people, just doing their job. Don't loose your cool with them, be polite and coprative. It wont do you any favors being any other way.