It was only the Tricolours' fourth win since they first went to New Zealand in 1961 and they outscored the home side 3-2 in tries.
The All Blacks were missing a number of key players, including their injured skipper Richie McCaw, and never managed to get their noses in front in Dunedin.
Dupuy kicked two penalties and converted all three tries before leaving the field with minutes to go with a cut head. Having kicked an early penalty he then added the extra points to a try from his half-back partner Francois Trinh-Duc.
William Servat burrowed over from close range for the second try and Dupy's conversion made it 17-3 with half-an-hour gone. The All Blacks hit back with a second penalty from Steven Donald and then conjured up a try from nothing for No 8 Liam Messam to cut the gap to six points just before the break.
Two more Donald penalties brought the All Blacks back on level terms, but Dupuy hit the mark again to edge the French back in front. Then came the crucial moment when full back Maxime Medard intercepted a Luke McAlister pass and raced 60 metres to the posts.
Dupuy's conversion stretched the lead to 10 points and a late corner try from Ma'a Nonu wasn't enough to bridge the gap.
The two teams meet again next week in Wellington.