But with the score at 30-24, a penalty try was awarded to South Africa after an illegal tackle by Liam Williams as he desperately attempted to stop Cornal Hendricks from scoring.
Morne Steyn converted to give the Springboks a 31-30 victory and a 2-0 series win and deny Wales a historic first win in South Africa.
And Gatland, who was head coach for the British & Irish Lions tour of Australia last summer, believes that one or two moments cost his side dearly.
"It's probably the toughest defeat I've experienced in my whole coaching career," Gatland said. "We're absolutely gutted and we didn't really deserve that final result.
"They're critical moments when you have to be 100 per cent accurate and in one or two moments we weren't quite good enough.
"We said we were poor in the first half last week and got caught. Our line speed improved a lot. There was a big improvement on last week.
"We put South African under a lot of pressure and scored some good tries. I'm proud of the effort and performance but gutted about the result."