The appointment begins on 1 April. Crozier has previously held non-executive roles at Debenhams, Camelot Group and G4S.
Before he joined ITV, he was the chief executive of the Football Association and then the Royal Mail.
Earlier this week, it was reported that ITV has begun the process to find Crozier's successor, although he has no imminent plans to leave the broadcaster.
Richard Baker, the chairman of Whitbread, said: "Adam is a very high quality business leader, with a strong background in business transformations, brands, communication and marketing and his expertise will be of great value to the Whitbread Board as we continue to expand our successful Premier Inn and Costa brands. "