The ad, which features six images of the girl in various states of undress, appeared in Vice magazine, the free style magazine available from bars, nightclubs and shops. It has a UK circulation of 90,000.
The ASA ruled the shots "suggested she was stripping off for an amateur photo shoot" and that it looked like the model was under 16. American Apparel said the model was aged 23.
The ruling said: "Because the ad could be seen to sexualise a model who appeared to be a child, under the age of 16 years, we concluded that it was inappropriate and could cause serious offence to some readers.''
However the ASA did not uphold a complaint that the ads were offensive and could be seen by children.
It said the nudity "was not so overly gratuitous as to make it unsuitable for or likely to cause serious or widespread offence to the target audience".
Commenting on the ASA ruling American Apparel's UK operations manager Brent Chase said: "American Apparel is well known for its provocative advertisements.
"Our models are real girls who are often employees or friends of the company. They do their own hair and makeup and aren't Photoshop-ed.
"From time to time people are made uncomfortable by this, and it occasionally causes an unfortunate reaction."
It is not the first time the clothing company has caused controversy with its advertising.
In May American Apparel was forced to pay $5m (£3.2m) to Woody Allen after the director's image was reproduced in a poster campaign without permission.