It began well enough for the hosts with Ryan Lamb kicking three early penalties, but Arnaud Mignardi and Pieter Van Niekerk scored tries for Agen.
Allen crossed for Gloucester but Lamb failed with the conversion and Agen were ahead 15-14 at the break.
Thomas Soucaze went over again for the visitors before Agen claimed their try-scoring bonus point through Conrad Stoltz.
Allen gave his side hope with his second try and Olivier Azam got Gloucester even closer - before fly-half Francois Gelez dropped the goal that seized victory for Agen.
Hooker Steve Thompson wrecked Border Reivers' hopes of pulling off a shock Heineken Cup triumph in a Franklins Gardens thriller that Northampton won 37-13.
The home side led 13-6 at the break thanks to a Mark Robinson try and the boot of Carlos Spencer, and went further ahead when Paul Tupai crossed.
Simon Danielli then raised the visitors' hopes of making a comeback before Thompson notched a brace to seal the deserved win.
Glen Jackson scored 22 points as Saracens recorded a hugely impressive bonus-point 37-20 victory over Narbonne in France.
Scrum-half Neil de Kock scored a first-half brace of tries to set Saracens on their way at Parc des Sports et de l'Amitie, although it needed a late penalty try and a Jackson interception to end Narbonne's challenge.
South African de Kock darted over from close range and then added a second try from 40 metres out as the English club showed grit to keep Narbonne out in the dying minutes.
Kiwi Jackson scored in injury time and, with a win over Glasgow under their belt last week, and a home match against Narbonne in the offing, Saracens look to have one foot in the European Challenge Cup play-offs.
Opposing front-rowers Franck Tournaire (Narbonne) and Fabien Ongara spent 10 minutes in the bin for fighting, with Saracens lock Tom Ryder also having a yellow card waved in front of him for a late tackle.
Narbonne outside-half Christopher Ruiz scored 15 points, but a penalty try in the 77th minute sealed Saracens' win, before Jackson earned a bonus point.