At the end of the day customers need to trust you as a person and believe that you can do a good and thorough job. The majority of prospects will probably not care that you are a man and a minority may be biased. It is an imperfect world unfortunately - man, woman, black, white, tall, short, foreign, local - but this should not stop you from succeeding.