The Cheifs found themselves down 15-0 at the break - the first time the champions had been held scoreless since 2010 - but they fought back to lead 20-18 with only a few minutes remaining.
But a try by replacement John Ulugia sealed the win for the hosts and rocketed them from 10th up to seventh overall on the Super Rugby league table to keep alive their hopes of reaching the play-offs.
"The contact area was very good. Defence improved, but I think we can still be a bit more dominant," said Waratahs coach Michael Cheika.
"When we made breaks we generally capitalised with something. In several matches this year we have come back from lost causes, sometimes to win, sometimes not to win.
"We are starting to build some belief in ourselves, whereas maybe earlier in the season we were talking about lack of belief. Don't get me wrong, there is nothing major occurring - just a bit more belief in each other and respecting the commitment we need to make to each other."
Waratahs full back Israel Folau picked up the man of the match award for his sparkling performance, that included a try and several try-saving tackles. Brendan McKibbin opened the scoring for the home side before right wing Cam Crawford scored a debut try on 15 minutes.
Waratahs right winger Cam Crawford scored in the 15th minute in his first appearance for NSW. Folau scored in the 37th minute, after leaping over two Chiefs outside backs to take a cross-kick from Bernard Foley from a ruck inside the 22.
McKibbin's conversion made it 15-0 at the break and the Waratahs' were flying. But Wallaby lock Sitaleki Timani picked up a yellow card at the start of the second half and the momentum began to swing the Chiefs way.
Gareth Anscombe rounded off a break by Aaron Cruden to cross for a try which he also converted and then followed up a McKibbin penalty for the home side with two of his own to cut the gap to five points.
Prop Ben Tameifuna's try from close range then levelled the scores and Anscombe's conversion finally put the Chiefs ahead for the first time with eight minutes to go. But the Waratahs' mounted one final assault which ended with Ulugia piercing the Chiefs defensive line for the match winning try, while McKibbin converted.
But while the Waratahs were able to celerbate a notable victory, it came at a cost. Wallaby wing Drew Mitchell, who is heading to Toulon next season, pulled up after only six minutes with a torn hamstring that is likely to sideline him for some time.