TalkTalk and Tiscali UK were found to have wrongly issued bills to consumers charged for services they did not receive, including some who had closed their accounts.
Ofcom opened an investigation into the two providers following complaints from more than 1,000 consumers.
The regulator recorded more than 62,000 cases of wrong billing between 1 January and 1 November 2010.
It ruled that while TalkTalk and Tiscali UK took "some" important steps to comply with its rules, the two service providers still wrongly billed nearly 3,000 consumers between 2 December 2010 and 4 March 2011.
TalkTalk and Tiscali UK have been issued with a "lower penalty" than they may have otherwise been given, had the companies not made efforts to rectify their mistakes. Ofcom has made it clear, however, that it considers the issue a "serious breach of the rules".
TalkTalk and Tiscali UK must pay the £3m fine within 30 days. The two service providers have also paid over £2.5m in refunds and "good will payments" to more than 65,000 consumers affected by the mistakes.
TalkTalk claims that the group has seen a 40% year-on-year reduction in customer service calls and said Ofcom receives three times fewer calls about TalkTalk than "at the height" of the Tiscali integration.
Dido Harding, chief executive, TalkTalk Group, said that "no customer is out of pocket" and that consumers affected have been compensated, stating that the "billing issue in question is resolved".
Harding said: "We are of course disappointed at the scale of the fine and feel it is a disproportionate penalty, however we are striving to continually invest in customer experience and are pleased at the clear progress we are starting to make."
In April this year, Ofcom named TalkTalk as the most complained about fixed-line broadband provider. Three was the most complained about mobile network.