Ofcom imposed the fine on Five for announcing bogus winners on three occasions over the course of January, February and March this year. Five was also called to book for asking staff to pose as winners on two further occasions.
The regulator highlighted 11 further instances of unfair competitions in 'Brainteaser' and its spin-off programme 'Memory Bank', also broadcast on Five. Both programmes are made by Endemol-owned company Cheetah Productions.
As well as the fine, which Ofcom said was the largest financial penalty imposed on a public service broadcaster to date, Five will be obliged to air a summary of Ofcom's findings on two occasions.
Ed Richards, Ofcom chief executive, said: "We take these issues extremely seriously and will not hesitate to protect viewers' interests. Every broadcaster needs to take very careful note of this decision."
Five accepted the breaches in 'Brainteaser' but disagreed with the fine.
Jane Lighting, Five chief executive, said: "Five has always accepted that breaches occurred with regard to 'Brainteaser'. However, we are disappointed with the very high level of the financial penalty imposed, especially as we believe Five took all reasonable and appropriate steps to ensure the programme complied with the Ofcom Code."
Five also said that Endemol had acknowledged its "mistakes and fundamental errors of judgement on the occasions highlighted". The channel said that winners were only faked when the production team "had failed to find a winner within the time available and therefore wanted to bring the competition to a close".
'Brainteaser' was pulled from the schedule in March after the news of the faked competition winners broke. Five today confirmed that the show will not be returning.
The channel also vowed to find the winners of the affected competitions and donate all profits from these competitions to a cancer charity.
The inquiry into 'Brainteaser' is just one of a number of Ofcom investigations into participation TV. The regulator is currently looking 20 other call-TV breaches including competitions on BBC One's 'Blue Peter' and Channel 4's 'Richard & Judy' show.
Ofcom said it was assessing each on a case-by-case basis and will publish its finding later this summer.