Sorry it's a bit long winded - if I knew how I'd draw a diagram!

The properties:-
A row of 1970-ish 3 storey town houses with a block of three flats attached to one end.
I clean 5 out of 8 of the terrace and the middle flat on the end. Plus one detached house opposite.
If the custies are in I take the hose through the house and do the backs; if the immediate neighbour is in I drop the hose and pole over the fence, walk back through and do the next property.
Every one cooperates and it works well; until today.
At the backs at the end with the block of flats attached I have the following to clean:-
The immediately adjoining town house to the block of flats I don't clean but have their permission to use my ladder to go through the garden to the next town house if the owners (Mr S) of said house are out and the backs want doing.
So I clean my custy (Mrs M in the flats) hop over two low fences and clean Mr S's rear windows.
Mrs M sees me and tells me off:-
Mrs M (Remember she is my custy too.) "You shouldn't be doing that ..."
Me "It's ok, the owners have given me permission to go through."
Mrs M "But you shouldn't climb over the fence..."
Me "I've used my gate ladder so I won't damage it and the owners have said I can go through"
Mrs M "But they only rent it ..."
Me "Do you own it?"
Mrs M "No, but you should only use our gardens (the communal garden for the block of flats I guess she means) to access our windows"
Me "What harm am I doing? Is the fence yours?"
Mrs M "No but we've (flat owners?) had a meeting and you should only use the main gate to access and exit our property, not the other gate or the fence."
Me "Mrs M, I'm just trying to do my job properly, the householder lets me go over the fence, your neighbour lets me go over the fence and it is their fence, they're low and I'm doing no harm, what is the problem?"
Mrs M "You should only use my (the communal flats) garden to clean my windows and you should stick to my path and not go onto the gravel to do that."
Me "So you want me to stand directly beneath your window, looking into your neighbour's below who doesn't have his windows done and not to stand on your gravel; and when I've done that I mustn't go over the adjoining fence which belongs to the neighbour's house to clean Mr S's windows?"
Mrs M "Yes."
Me "That doesn't seem very neighbourly to me Mrs M; I'm simply trying to carry out a service causing the least disruption and I feel you're being obstructive for no good reason."
Mrs M "Well we don't want you climbing over the fence from our property."
Me Shakes head and walks away.
So I've learned not to do an immediate dump so as I can either not go back, or carry on, or see what my mood is next time it happens.
Dan the Man advises I just carry on and wait for her to moan again and if she does, dump her. But of course when I do that I have no legitimate reason to go into the flats communal garden. That's not a huge problem as 4 times out of 5 the custy (Mr S.) is in and I can take the hose through. I could just do the fronts.
Wha-joo-lot-fink?