I has something similar last week.
Small job one room, local, so £55-00. She was an older lady I guess, and quite abrubt/rude on the phone. ( On my paperwork I put Be Wary, to remind myself)
The day before the job I called to say i'll be along late morning, and again she was quite abrubt, and without saying goodby, she put the phone down. I put the phone down, and thought about it. Decided to call her, tell her that I didnt like her attitude, and even though I know I will do a top job, I didnt think I could make her happy so I will not be doing the job. £55 wont make or break me.
My instict was keep well clear so I followed it, you did the same, and im sure you did the right thing. At times, "You just know", whats the right thing for you to do. Always follow that intuition.