The watchdog called the breaches of consumer protection rules "serious and sustained" following two investigations.
Ofcom found that Vodafone did not credit 10,452 pay-as-you-go customers when they had topped up their account. Over 17 months, the company should have paid out £150,000.
According to Ofcom, only when it intervened did the telecoms company take action.
It added: "Vodafone also breached Ofcom’s billing rules, because the top-ups that consumers had bought in good faith were not reflected in their credit balances."
A second investigation found that Vodafone’s customer service staff was "not given sufficiently clear guidance on what constituted a complaint". Ofcom said the company’s processes were "insufficient" in dealing with complaints in a fair and timely manner.
Vodafone was also found not to explain to customers in writing that they can take "an unresolved complaint to a third-party resolution scheme after eight weeks".
Ofcom has today fined Vodafone £3,7m for taking pay-as-you-go customers’ money without providing a service in return; and £925,000 for the flaws in its complaints handling processes. The money must be paid to Ofcom within 20 working days and will be passed on to the Treasury.
Lindsey Fussell, Ofcom consumer group director, said: "Vodafone’s failings were serious and unacceptable, and these fines send a clear warning to all telecoms companies.
"Phone services are a vital part of people’s lives, and we expect all customers to be treated fairly and in good faith. We will not hesitate to investigate and fine those who break the rules."
A Vodafone spokesman said the failings were due to errors made during a complex IT migration which involved moving more than 28.5 million customer accounts and almost one billion individual customer data fields from seven legacy billing and services platforms to a single system.
He said: "We deeply regret these system and process failures. We are completely focused on serving our customers: everyone who works for us is expected to do their utmost to meet our customers' needs, day after day, and act quickly and efficiently if something goes wrong.
"It is clear from Ofcom's findings that we did not do that often enough or well enough on a number of occasions. We offer our profound apologies to anyone affected by these errors."