He was speaking during a panel discussion with "manbassadors" from different agencies.
Bailey said: "Men need to realise what it’s like to walk into a meeting room and be immediately judged on what you’re wearing, or what your hair looks like or your body. It’s little, insidious, toxic things that create barriers for women."
The audience at the conference was primarily made up of women. The male panellists said this was embarrassing and showed that men were either scared, complacent or did not see how gender diversity was relevant for them.
Bailey said: "No-one who is successful likes to believe they are successful for any other reason than their own merit.
"Get a load of white, middle class, men together they will form a tribe and there will be bias towards other white middle class men. But implicit in that is that there are people in senior positions who, if there was a level playing field, may not be sitting here. People don’t like that."
Grey London's chairman and chief creative officer, Nils Leonard, said to change this you had to give men a clear role.
He said: "I think you have to tell them to come and meet the most powerful energised women in your industry."
The panel acknowledged that the the necessary mystic around creative departments probably contributed to the lack of women within them, because agencies were reluctant to upset the process and force diversity targets on them.
Leonard said Grey’s removal of the sign-off process helped to free up talent and place more trust in their staff.
He continued: "You can’t hire talented people, pay them lots of money but then say that the only opinion that matters is the guy in the corner office."
Leonard insisted a great culture gave a business the best competitive advantage, because it takes seven years to copy.
He said: "Too many agencies hide in a haircut or a postcode, but what does that mean? Force yourself to put your culture on paper and see who it attracts."
Be clear about what qualities you want in your staff, he continued: "For us it’s about energy – do you make the room around you better, and do you have the ability to make ideas that impact on culture. Those aren’t male or females traits at all."
Daren Rubins, the chief executive at PHD, said the media industry used to be a backroom function filled with "barrow-boy traders, aggressive, obnoxious characters".
But media has now matured and has more women leaders, he said: "We’re all dealing with talent and talent makes choices based on whether a company shares their values."
Agencies actually need to ensure they were attractive places for women to work, continued Leonard: "I’ve heard lots of people talk about hiring and promoting women, but no-one is saying why women would want to go to that agency.
"If women look at an agency and think it has the same crusty model they’ve seen for the last 25 years they’ll think, 'why the fuck would I go there? It’s going to be awful.'"