Nowadays, I find it much easier to just say 'no problem' and worry about it later; then they usually get dumped.
I think I have only spoken to customers on the phone, whom I have dumped, about 5 times. On each occasion, I have been polite and calm and told them why I can no longer clean for them. Locked gates on numerous occasions (including large electric gates) rude attitude, dogs mess, rubbish and crap down the side access, etc etc. On every occasion it has not ended well, with the customer getting upset, because they think they are in the right. You can explain until you're blue in the face but the custy has excuses for everything; it just causes unnecessary stress.
Now I can't be bothered to explain, so I just politely say 'no problem' and then decide if I want to keep them. If I dump them, I simply don't bother turning up anymore.
If they phone me and ask why I haven't turned up, the conversation goes like this:
Custy: 'Why haven't been for the last 6 months'
Me: 'I didn't think you wanted me to clean for you anymore'
Custy: 'Why not?'
Me: 'Because the last time I contacted you, you didn't want them done'
I then specifically don't say anything and there is a long pause (this is my favourite bit).
Custy: 'Well the last time you text me I had to go to hospital for my lumbago and it was all too much hassle' (this is usually a lie).
Me: 'Oh that's a shame as I have replaced your slot with a customer who DOES want their windows cleaned on a regular basis' so I'm afraid I can't help, thank you, bye'!