The 2003-04 season saw the Millers sell their prolific striker Alan Lee to Cardiff City and they clearly struggled without the Irish international's goals, finishing a disappointing 17th. The season was one of inconsistency with the Millers always looking over their shoulders at the bottom end of the table, but they gained enough points to ensure safety in the penultimate game thanks to a fine 3-0 home win against Burnley that helped them to finish 3 points and five places clear of the drop. The highlights of the season were a 1-1 draw with Arsenal in the League Cup - with the Gunners winning 9-8 in the a penalty shootout and the first win of the season which came in the 3^rd game when one goal was enough to send West Ham United packing. The result was overshadowed with the news that the visitors were so dissatisfied by the facilities at Millmoor that they chose to get changed in a nearby hotel.

The 2004-05 season was worse still, and finally Rotherham were relegated. Ronnie Moore resigned during the relegation campaign with Mick Harford taking over, which saw Rotherham at the foot of the division for almost the whole season with just 5 wins from 46 league games. What's more, once they slipped into the bottom place after just five games, they remained there until the end.

Off the field activities took the attention more than anything on it with takeover talks concerning a newly formed group called 'Millers 05' who were interested in taking over from owner Ken Booth. This change of ownership was confirmed on 1^st Jan 2005, the £3m deficit was wrote off and the club was loaned £600,000 so they could get through the final 3 months of the season.

Relegation was confirmed with three games left to play and at the end of the season the team was 21 points adrift of safety.