The combined businesses will run under the umbrella company named Peak. TUI Travel and Intrepid Travel will keep their own branding while Peak will offer a platform for a multi-brand strategy.
The deal, scheduled to be completed by May, is expected to "drive cost synergies" of at least £10m a year for the next three years.
Peter Long, chief executive of TUI Travel Plc, said: "Specialist travel is an important part of our business and one of our key drivers of growth, due to strong margins and development opportunities.
"This deal with Intrepid Travel will significantly enhance our position in the fast-growing adventure travel sector.
"We believe this venture has great potential for value creation through the increased scale, synergies and growth of the combined businesses."
TUI Travel's adventure businesses, part of the specialist and activity sector, are based in the UK, Australia, North American and Netherlands and include Trek America, Exodus and Country Walkers.
Intrepid Travel is based in Melbourne, handling more than 100,000 customers a year.
In December, TUI said it was revising its marketing strategy for its Thomson travel brand, as it looked to maximise sales in the crucial January holiday booking period.