The record fine was handed down to Eckoh UK, the phone service provider for the quiz, after Icstis investigators discovered that just under half of all calls, each costing £1, were received after the shortlist of winners had been chosen.
Icstis, which is changing its name to PhonePayPlus in the autumn, was particularly concerned at the scale of the problem. It was found that just under 5m viewers had entered the competition since October 2004.
Sir Alistair Graham, Icstis chairman, said: "Winners were being chosen before the competition closing deadline, while millions of additional viewers were still encouraged to phone in and pay to enter competition, but were denied the opportunity of fair consideration. Such reckless disregard for viewers is unacceptable."
Channel 4 has sought to distance itself from Eckoh with the release of a statement today which defends the broadcaster's role in the scandal.
Channel 4 said: "We engaged Eckoh in good faith as a reputable and experienced service provider and we are very disappointed by their failure to ensure that all calls to the competition were handled properly.
"At no stage did Eckoh raise concerns with Channel 4 about the way 'You Say We Pay' was run, so we are shocked to learn that management at Eckoh were aware the competition was not being operated properly at least six months before the problems were made public."
The statement added that as soon as the broadcaster became aware of the problem the competition was suspended, in February this year, and commitments were made regarding viewer refunds and donating profits to charity. In addition a new monitoring regime is now in place to ensure the debacle is not repeated.
This latest ruling, which Eckoh has 20 days to appeal against, comes amid a wave of scandals involving premium rate lines over the last year.
In May the BBC Trust launched a policy review after discovering "serious errors in judgement" regarding premium rate telephone competitions on BBC's 'Blue Peter' and 'Saturday Kitchen'.
The Blue Peter competition was particularly criticised, when 14,000 children called in to win a prize that was handed to a member of the studio audience following a phone line fault.
Graham added: "There is no doubt that the public thoroughly enjoys taking part in premium rate competitions and votes on television. However, as well as being entertaining and fun, services should be reliable and trustworthy.
"Consumer protection should be at the heart of television rather than a broadcasting philosophy of "the show must go on".
Last month Icstis appointed Paratus Communications to promote its new PhonePayPlus identity. It will also provide communications support as Icstis's remit widens to include the regulation of 0871 numbers.