There was little to choose between the sides in the first half and it turned into a kicking duel between the fly-halves.
Otago's Nick Evans came out on top with three penalties for the home side compared to Sam Young's solo effort for Taranaki to give Otago a 9-3 lead at the break.
The second half belonged to Oliver though and he only had to wait nine minutes before he was driven over from a five-metre lineout for his first try.
Evans' conversion, added to an earlier fourth penalty, pushed Otago's lead out to 19-3 but Taranaki rallied and scrum-half Craig Fevre punished a handling error by Seilala Mapusua to dive over before Todd Feather's conversion got Taranaki back to within nine points.
Eight minutes from time Oliver scored his second try - a carbon copy of the first - to seal the win and give prop Carl Hoeft the perfect send-off in his last match for Otago.
He heads to French club Castre on Monday to join former All Blacks team-mate Kees Meeuws.
The result also moved Otago up to third in the table behind Canterbury and Auckland, who thumped hapless Northland 53-7.
Northland have not won a match so far this season and were never in the game as Auckland ran in eight tries to Northland's one.
Tony Koonwaiyou and Isaia Toeava grabbed two apiece with Isa Nacewa, Angus Macdonald, Tasesa Lavea and Saimone Taumoepeau also crossing the whitewash.
Despite a valiant forward effort Northland were simply outclassed in the backs with their sole try coming from hooker Tim Dow who threw a couple of dummy passes before diving over from a ruck near the try-line.
It was an error-ridden affair between Southland and Bay of Plenty in the day's other game with Southland coming out on top 19-13.
Southland held a 6-0 advantage until five minutes from the break when the Bay finally woke up and fullback Adrian Cashmore scored from the base of a ruck.
Mike Delaney added the conversion and a penalty to give his side a 10-6 half-time lead.
But the boot of Richard Apanui and a try to Watisoni Lotawa swung the match Southland's way in the second half with another Delaney penalty the only score the Bay could muster.