After Sarries were beaten 43-33 on Sunday by the Stormers, his team's director of rugby Eddie Jones slammed their opponents' scrum, saying he was 'amazed' to see a South African side that struggled in that area.
Visagie though was not that critical, but claimed it was time that South African rugby changed its policy and selected players plying their trade beyond the country's borders.
"The Stormers were unlucky to lose two tightheads," he told South African newspaper Die Burger.
"They also did not fare any worse than other teams against us.
"We've done worse things to other teams' scrums. I also know scrumming is something the Stormers want to address.
"South African rugby's scrumming has gone backwards because players are discarded too quickly.
"Only the Sharks are solid on their own ball."
The 35-year-old also made a serious case for Toulouse tight-head Daan Human to be selected for this year's British and Irish Lions tour after his outstanding form for the French club.
He continued: "The best South African loosehead is Daan Human. He is doing brilliantly at Toulouse and is one of the best-paid loosehead props in Europe.
"I definitely think he can play a role against the Lions and it does not have to be at the expense of any of the younger players. Daan scrums well and works hard on the field."
Visagie also had a few other thoughts on some of the Stormers players that turned out on Sunday, adding: "I rate Beast Mtawarira and actually think Brian Mujati should be a loosehead prop.
"Heinke van der Merwe scrums around you. It's something referees are going to come down on. The question is also how good the rest of his game is."