London Irish had staged a dramatic late comeback from 31-19 down to level the scores and force extra time, thanks to tries from Olivier Magne and Robbie Russell.
Barry Everitt, whose last-minute conversion had sailed wide, then put London Irish into the driving seat with a penalty eight minutes into extra time.
However Forrester, the Gloucester number eight who was overlooked by England for this summer's Test tour to Australia, emerged the hero.
After Anthony Allan's break, Forrester chased down his own kick forward and dived on the loose ball to score the decisive try a minute into the second period of extra time.
Irish, who finished fourth in the Premiership, have been transformed under the guidance of Brian Smith and captaincy of evergreen England centre Mike Catt.
From a stodgy side which scored the fewest tries in last year's Premiership, they have thrilled crowds with attacking rugby and the fastest back three in England.
Gloucester's game-plan, particularly in recent months, has been similarly bold after head coach Dean Ryan gave youth its head at Kingsholm.
Gloucester had started on the back foot and were penalised twice in quick succession, allowing Everitt to boot Irish into a 6-0 lead.
Delon Armitage almost picked off an interception and was speeding away towards the Gloucester try-line, with only lock Alex Brown in pursuit, when referee Nigel Whitehouse called play back for a deliberate knock-on.
And it proved a significant turning point. From the chance to move 13-0 ahead, London Irish suddenly found themselves 7-6 down as Mike Tindall chipped through for Mark Foster, who outpaced Topsy Ojo to touch down.
Ryan Lamb converted and then added a penalty to extend Gloucester's advantage after Kieron Roche was penalised for a late tackle on James Simpson-Daniel.
Lamb produced a wonderfully measured kick into the corner which forced Catt, under pressure from Simpson-Daniel, to concede a close-range lineout.
And Hazell peeled off the back of the lineout drive to burst over and score Gloucester's second try of the half.
But London Irish hit back immediately with Armitage climbing higher than Simpson-Daniel to claim Everitt's towering cross-kick at the second attempt and touch down.
Everitt landed a tough conversion in strong winds from wide on the left but Lamb, who had been holding his hamstring, responded with a penalty for Gloucester after an infringment at the lineout.
Gloucester held the edge at half-time and it was Irish who re-started the quicker. Flutey could not keep his feet after running onto a pass from Juan Leguizamon but the Exiles kept the pressure on and the ball came back for Everitt to slot a drop-goal.
Lamb looked to spark a counter-attack from his own line but paid the price by running straight into a thundering tackle from Leguizamon.
Lamb went off with a bloody mouth - he was replaced by Ludovic Mercier - and just two minutes later Leguizamon was dispatched to the sin-bin for a dangerous tackle on Richards.
The Exiles coped well with being a man down - Mercier and Everitt exchanged penalties - and they had looked the side most likely to score with Ojo and Sailosi Tagicakibau threatening out wide.
But Gloucester moved clear again with another penalty from Mercier before Simpson-Daniel applied what seemed the killer blow when he intercepted a wild pass from Exiles lock Bob Casey and ran over 50 metres to score under the posts.
But Irish would not be denied so easily. They piled forward, at times committing 13 bodies to drive Gloucester back and set the platform for Magne and then Russell to burrow over.
Everitt, who finished with 19 points, pulled his last-minute conversion wide and the game went to extra time.
His penalty then pushed the Exiles ahead - only for Forrester to snatch the trophy from Irish hands.