Speaking at The Guardian Edinburgh Television Festival, Elstein said it was ironic that the two main ITV companies - Carlton and Granada - were now run by accountants but had still managed to haemorrhage so much money.
In an astonishing attack on the other panellists, which included the ITV joint managing director, Mick Desmond, and the former chief executives Richard Eyre and Stuart Prebble, Elstein said that as well as poor management, the federal structure of the ITV Network Centre meant it "had the worst of all worlds".
He also claimed that Carlton and Granada pressurise the Network Centre to commission new formats from them rather than from the independent sector, which was not to the benefit of the overall schedule.
However, ITV's outgoing director of channels, David Liddiment, was in the audience and interrupted saying this had never been the case.
In the sometimes fiery debate, Elstein dismissed reports that ITV's current problems are cyclical, claiming it had lost sight of its core business when it made its "deluded venture into the pay-TV market.
Answering criticisms that the ITV structure was fundamentally flawed, Prebble acknowledged that there are problems in having three shareholders in charge of the Network Centre. He said that the multi-management structure meant decisions were not made quickly enough and admitted that it could be dysfunctional.
Prebble, who was an architect of the disastrous ITV Digital venture, which spectacularly collapsed, compared the relationship between the two key shareholders Carlton and Granada as "two men fighting over an ice-cream while the sun is shining".
But he defended the decision to enter the pay-TV market, adding that a combination of factors including the advertising recession and problems with the strength of the DTT signal meant that the service was doomed.
The panellists were asked what ITV needed to do to turn its fortunes around. Elstein said ITV needed to invest an additional £200 million a year in programming while Eyre, in an apparent reference to ITV's attempts to woo Channel 5's chief executive, Dawn Airey, said it needed a figurehead to create a new vision.
Desmond refused to comment on how negotiations with Airey are progressing and observers question whether she would want to take on such a challenging job when things at Channel 5 are going well.
Desmond was confident, however, that ITV would continue to be a mass-market commercial TV channel, despite its share of audience slumping to just 22.4 per cent in July. Although he was not specific he said he was optimistic that ITV's share would "still be in the 20s by 2005.