Danny Ward capped his return from injury by giving the Millers the lead but goals from Anthony Dwyer and Alex Fisher earned the visitors maximum points.
It proved to be a disappointing afternoon for Paul Raynor who presided over a team which included a mixture of first team professionals and academy scholars.
The Millers missed a host of chances in the first half and could have had the game won by half time. On seven minutes Jerry Yates couldn’t control the ball when well placed to finish after Darnelle Bailey-King’s cross, and a minute later Ward headed wide from a Richard Brindley cross.
On 18 minutes Yates was again found in the area but he mis-kicked after good work by Ward. Conor Newton then came close with a fizzing strike from 20 yards which flew just past the far post.
Mansfield ‘keeper Sascha Studer then made two smart saves to deny Bradley Peace-McDonald and then Max Hallatt as the Millers turned the screw.
On 38 minutes the Millers took the lead when Ward neatly took the ball around the ‘keeper to score on his return to action.
The lead was short-lived as the visitors levelled just two minutes later when Dwyer stabbed the ball over the advancing Thompson.
In the second-half Thompson denied former Miller Mitch Rose with a smart save before Yates almost restored the Millers firing just wide after a neat turn.
On 68 minutes Studer saved low down from Newton whilst at the other end Fisher beat Thompson to the ball but his effort cannoned off the post.
With seven minutes remaining Mansfield grabbed the winner when Fisher rose highest in the area to nod beyond Thompson.
Rotherham: Thompson; Brindley, Peace-McDonald, Hallatt, Milsom, Bailey-King (Muskwe 80), Newton, Green (Fenton ht), Hammill, Yates, Ward (Warren ht).
Subs not used: Denton, Rose, Bailey.
Mansfield: Studer; Marsden, Danquah, Fitzpatrick, Brownlie, Green, Blake, Rose, Fisher, Dwyer, Spencer.
Subs: Hearn, Thomson, Bishop.