It's a description that grates with him so much that he brings it up again, mid-interview, unprompted. He admits he's not initially forthcoming.
"I don't come across as the liveliest of people, that's not me, but that doesn't make me a cold, callous, miserable bugger," he states.
He's right in that he's not immediately affable, and the fact that his office is quite bare does little to help create any warmth. But, according to his supporters, he warms up on further acquaintance.
One supporter is Jeremy Miles, who hired him as the acting managing director to run Miles Calcraft Briginshaw Duffy while Helen Calcraft was on maternity leave. He describes Whyte as "cuddly" - once you get to know him.
He's not "hail-fellow-well-met", according to the McCann executive creative director, Robert Campbell, a friend of 18 years since they worked together at Abbott Mead Vickers in the 80s.
The accusation that he's arrogant and aloof stems mainly from his last high-profile agency role as the chief executive at Leo Burnett, which has left a dark stain on his career.
Following a quick, smooth rise through Leagas Delaney and GGT, Whyte spent four-and-a half years at Leo Burnett, becoming the chief executive in 2001 after three strong years as the managing director.
It was a short-lived appointment. The agency's creative reputation was at its strongest - it picked up three gold Lions at Cannes in 2001 - but it failed to convert its creative fame into new business.
Whyte was under a lot of pressure to get new business, but his efforts were hampered by a management team dogged by politics. Contemporaries testify that instead of dealing with the problems, he withdrew.
One former colleague explains: "He found the pressure overwhelming and turned in on himself and rarely got out of his office." In fact, just two months before he was ousted, his Christmas present from staff is reputed to have been a map of the agency complete with a compass.
His tenure ended when he was "forcibly extricated" (his words) to make way for the present group chief executive, Bruce Haines. Whyte's version is that he and his boss, Steve Gatfield, couldn't agree on the future direction of the agency. Gatfield recruited Haines, and Whyte departed.
He's now taken up the same job at McCann, but the roles and the agencies are very different.
At Leo Burnett, Whyte was running an agency with 280 staff that had been thriving creatively. At McCann, he will lead 400 staff, and an agency whose creative and business reputation is in tatters.
Whyte thinks he's better suited to the latter challenge. "I'm not good at maintenance," he says. "I like changing things."
Change is certainly on the agenda at McCann. As part of the recovery plan instigated by the chairman, Rupert Howell, Campbell and the chief operating officer, Robin Price, the agency is on the brink of a restructure that will focus on the creative product, prevent further clients running out of the door and help to woo some new ones.
"Almost every piece of business McCann has lost recently has been through work that isn't good enough," Whyte explains.
Birds Eye and Glenfiddich have walked in the past six months and, last week, McCann pulled out of the pitch for Norwich Union's £11 million account on which it was the incumbent. During the same period, its wins, namely the Carbon Trust and Signet, hardly offset the losses - especially if you add the humiliation of losing lead status on Coca-Cola to Mother last year.
This flurry of reviews will test Whyte's client-services skills to the limit, but it's an area that many believe he is very good in. One senior agency source calls him a "very efficient, organised account director".
It will also help that, in his role as the head of client services, Europe, at McCann Erickson Worldgroup for the past 16 months, he has already been working with some of the network's most important multinational clients, such as Nestle, MasterCard and Cadbury Schweppes.
He was originally hired by Ben Langdon, the former chairman and chief executive of McCann Erickson Worldgroup for Europe, the Middle East and Africa, who describes him as "good at client services".
Howell said it was seeing his relationship develop with Nestle that brought Whyte's ability to his attention and says he was "clearly over- qualified" for the role he was doing.
It's just as well, because the new-business vultures from rival agencies will be circling McCann's clients, particularly the ones the outgoing chief executive, Chris Hunton, was close to - Cereal Partners and MasterCard.
Miles says Whyte made a great impression on clients when he was at MCBD.
"My current client, David Dingle at P&O, cannot speak highly enough of him in terms of his brainpower and commitment to clients," Miles says.
Howell says when he spoke to the McDonald's client, John Hawkes, the head of European marketing at the fast-food chain, he called him the "the best adman he's worked with".
But it will take much more than proficiency at client services to turn McCann around. There will be the morale of the agency to handle, an area in which Whyte did not excel while at Burnett. However, he'll have the energy of Howell and Campbell to make up for any shortfall in this area.
"He's not your typical advertising extrovert," Howell explains. "We don't need another one. We've already got Robert, Robin and me."
Whyte's former boss, Gatfield, says: "As the chief executive among a very experienced team, he stands a greater chance of success than at Leo Burnett, where he lacked that seasoned bench strength."
Whyte believes that the large team at the helm of the agency is important.
"McCann became obsessed about client service," he says. "An agency run by one person becomes very singular and focused. With Robert, Rupert, Robin and Frank Lieberman on the TV side, we're getting real breadth and diversity of opinion."
That Whyte is an internal appointment is good for the agency, which has suffered repeated management changes over the past 18 months. He will be a reasonably familiar face around its corridors and he will have a grip on McCann's cultural issues.
"Every agency needs to improve its talent base, but we also need to get the people we've got more focused and more motivated," he says.
"This agency has lost a sense of culture. People don't understand what it is about. Our ambition is to become proud of what we are - a multinational office, but also producing interesting and involving work," he explains.
Crucial to the success of his appointment will be his ability to share management decisions - and the limelight - with Howell, something that Chris Hunton, Whyte's predecessor, found trying.
Family: Son, Harry; fiancee, Lucinda
Favourite ad: Levi's "creek"
Describe yourself in three words: Decisive, forthright, passionate
Greatest extravagance: My scuba-diving obsession
Most treasured possession: My grandfather's watch
Most admired agency: Bartle Bogle Hegarty
Living person you most admire: Steve Jobs, the founder of Apple
One to watch: Jamie Kenny, an account director at MCBD
Motto: If something's worth doing, it's worth doing properly