"Our dressing room is like a morgue at the moment," Gaffney said.
"There's elation at the fact that we have gone out there and done exactly what we wanted by coming away with five points.
"The attacking part of our game was very good and deserves to be rewarded but the defensive part is leaving a lot to be desired.
"It's not the structure, it's individual defence which is letting us down.
"The players have to be accountable and we'll work very hard on it. We seem to slacken off when we get a handy lead and haven't got the instinct of some of the more elite sides who can put teams totally away.
"We have to become much more ruthless and not concede soft tries. The last two were just abysmal."
Looking ahead to the trip to Biarritz, Gaffney added: "I know how hard it will be. It will be a forward battle and the game will be decided up front."
Glasgow coach Sean Lineen praised his side for rallying to secure a losing bonus point but he was as unhappy as Gaffney with his team's defending.
"There was a lot of rugby played but we had to work slightly harder for our tries than Saracens. We made it very easy for them," he said.
"We gave three of the four tries to Saracens on a plate. They played some good rugby and we had to hang on.
"Our defence has been fairly watertight in the last few games so to leak tries like that was disappointing.
"We made too many unforced errors and Saracens are one team who will make you pay for that but the boys showed tremendous courage to come back."