The lock followed fellow World Cup winners Wikus van Heerden, Danie Rossouw (neck injuries) and Gurthro Steenkamp (hand injury) in pulling out of the tour.
The Springboks are already without Percy Montgomery, Butch James, Ashwin Willemse and Gary Botha, who have all joined their European clubs for the forthcoming season.
Matfield met with coach Jake White and decided that he wanted to spend time with his family before joining Toulon in the new year.
It was also confirmed that Springbok captain John Smit will not play at Twickenham on December 1 after his club Clermont Auvergne refused him permission to join the national side for that fixture.
Matfield will be replaced by Free State lock Barend Pieterse, while Tiaan Liebenberg of Western Province comes into the side in the place of Smit.
Scrum-half Fourie du Preez will only join the side in Wales after getting married on Saturday and leaving on honeymoon.
