The Geordie pop import is thought to have "lacked chemistry" with fellow judge Paula Abdul, according to US reports. Her strong accent is also believed to have caused concerned for producers who suspected the American audience might not understand her.
Other reports suggest Cole is suffering with homesickness so it is by mutual consent she is leaving the show.
Cole is expected to be replaced as a judge by the programme's co-presenter and former Pussycat Doll Nicole Scherzinger. Welsh presenter Steve Jones will present the programme on his own.
Cole was finally named as an 'X Factor' US judge earlier this month after months of speculation. She has only been filming the show for the past two weeks in Chicago and New York, alongside fellow judges Paula Abdul, Simon Cowell and record producer Antonio LA Reid.
The show is scheduled to air in the US in September. Whether Cole comes back to the UK to take up her position again on British 'X Factor' remains to be seen and some reports are suggesting she is angry at how the situation has been handled and wishes to sever all ties with the programme.
Cole's departure comes a week after UK judge Dannii Minogue revealed her contract with the show has not been renewed due to a clash of filming with 'Australia's Got Talent'.
An 'X Factor' spokesperson was unavailable for comment.