The Boks have lost five games in a row away from home this year under White and won only four out of 10 in a disappointing season that included home losses against France and the All Blacks.
Also of concern to the unions is White's apparent reluctance to assume responsibility for poor displays after he criticised the system following the record 32-15 defeat against Ireland on Saturday.
And away from the pitch, the unions are also upset at White's decision to withdraw 24 contracted Springboks - of whom only a handful made the tour squad - from the domestic Currie Cup competition.
But even as the storm clouds continued to gather around the South Africa camp, the coach remained defiant ahead of Saturday's Test against England
"I can't worry about those sort of things," White said.
"I have a job to do on the field. We have a new batch of players here with us and for our World Cup plans it's important that we increase our depth.
"The decision to rest key players for this tour was taken jointly by me, the board of SA Rugby and the review committee at a meeting earlier this season.
"Also, just before the Rustenburg Test I had a meeting with former coaches and key-role players (among them Nick Mallett, Carel du Plessis, Rudolf Straeuli, Harry Viljoen and Naas Botha) and it was decided we must do whatever we can to create opportunities for all our players.
"This tour was always going to be about creating those opportunities."