Carroll, who joined BBH in 1991 and became chairman in 2004, is not leaving to join another agency.
He told Campaign that he had "no plans" for what he was going to do next, adding: "I’ve had a great time and worked with some amazing people."
Carroll will be replaced by Peppiatt, who is presently BBH’s deputy UK chairman, having joined the agency in 1995 as a project manager.
Ben Fennell, BBH London’s chief executive, said: "Becoming chairman will not stop anyone at BBH calling Jonathan Peppiatt 'Pep'. His unique ability to connect, with clients, with talent and with issues, will make him a brilliant UK chairman."
Carroll was a qualitative researcher before he joined BBH in 1991 to lead the strategic planning on the agency's Levi’s account. He was head of planning at BBH between 1998 and 2001.
Sir Nigel Bogle, a co-founder of BBH, said: "Occasionally, very occasionally, one has the privilege of working with someone who conforms to no stereotype whatsoever, who is outstandingly talented yet self effacing, who is immensely popular and trusted by everyone.
"Jim is one such person. If anybody embodies the spirit of the black sheep it is Jim."