Whether you think you're being harsh or not is irrelvant...
You're obviously not happy with this customer and so by starting again you'll only start cleaning his windows with a negative outlook of said customer from the beginning.
So if anything minor happens, in your head you'll just get more miffed...
Basically i'm saying that seeing as he's already pi55ed you off, you'll just find it easier to get pi55ed off with him again so why put yourself through it?