The show's average live audience across the series so far is down a million viewers on last year, according to ITV, which points out that with 11.3 million viewers it is still the most popular series on any TV channel this year.
Last weekend the drop was more pronounced, with the Saturday night show down from 12 million across ITV1, ITV1 HD and ITV1 + 1 to 10.1 million and the Sunday night show down from 13.6 million the 11.3 million.
Tabloids have seized on the angle that the show is in crisis without its guiding light Simon Cowell, who is busy with the US version, leaving a less experienced judging line-up of Gary Barlow, Kelly Rowland and Tulisa Contostavlos alongside stalwart Louis Walsh.
Following several stories about how Simon Cowell has instructed the judges to "shake it up", this Monday The Sun reported Walsh was facing the sack next year. Mixed together with a swell of opinion that the crop of talent is thinning it out in the show's eighth year, is this all translating into a turn-off for advertisers?
Luke Duffy, head of broadcast, Aegis Media central trading, "no"
"Obviously it is cause for concern that 'The X Factor' audience is dropping off week on week. There was a feeling that this would be the year that 'The X Factor' audience would at best plateau and more likely, start to diminish naturally following recent years' stellar performance.
"Additionally 'Strictly' has a really good line up this year. That said, it's still a massive success story in comparison with everything else ITV and indeed network television has to offer. This isn't the time of year you want programmes to lose ratings but it's still a very strong proposition and we've still got clients fighting to get in to it. It's as in demand as ever."
Chris Locke, group trading director, StarcomMediaVest Group, "no"
"'The X Factor' is a long way from dead. If you look at advertisers that are in it. Everyone wants to be in 'X Factor'.
It is still doing 12 million over two nights and then lots of people watch the Saturday show on Sunday afternoon or on catch-up.
"One of the problems is the artists are marginally less talented this year but the new panel is pretty good and people like to see what the girls are doing with their hair and make-up.
"It's still a massive part of people's lives through Facebook and the newspapers even if they don't realise it and brands like it because they can see the uplift on Facebook and Twitter."
Paul Rowlinson, investment director, Mindshare, "no"
"There are always ebbs and flows between 'The X Factor' and 'Strictly Come Dancing'. Not having Simon Cowell is perceived in the media as creating a bit of an issue but it’s nothing to get upset about as the numbers are holding up okay.
"If you are just looking at the overnights they're not representative as there is lots more viewing done on catch up.
"There are two separate views – you might say it's not performing well but you could say it's only marginally behind [last year]. If you look at the numbers it's holding up pretty well."
Follow Maisie McCabe on Twitter @MaisieMcCabe