But Latham, one of the squad's most senior players, remains upbeat and insists the Wallabies are gearing up to hit England with a big performance at the weekend.
"The most disappointing thing about it all, I guess the guts of it, is that we know as a squad we are better than the way we are performing," said the 1999 World Cup winner.
"We are performing at training the best I have ever seen any Australia squad train.
"We have some of the most talented individuals in world rugby in one team and yet we seem to be narrowly losing.
"It is small things that are losing us Test matches and that probably adds to the frustration. But behind all that frustration there is a lot of confidence, there is a belief we are not that far off.
"The Aussie cricket coach, John Buchanan, said when we met in Sydney that everything can be nice and smooth and you think the whole squad is running as a well-oiled machine when you are winning, but when you get a few losses you start to find problems you never knew existed.
"It's good to have these moments in sport so we can iron out these problems we have in our game. We are making the small changes to ensure that when we are at game day we are almost at perfection.
"It could easily click into place on Saturday. We can go back to last week and see what could have been and gain confidence from that - but what could have been and what is are two different things.
"We could easily click and we know if that happens it could be a good feeling. Playing England over here has always been a huge challenge and an exciting challenge to be part of. It's been a while since we had a good win at Twickenham."
Latham, aged 30, almost moved to Munster two years ago when his international days looked numbered. But he decided to stay with Queensland, adapted his game and is now enjoying his role as one of the Wallabies' elder statesmen.
"It is an exciting time. We have 18-year-olds in the squad, they bring an enormous amount of enthusiasm and innocence and that is very infectious for a guy like myself," said Latham.
"We are a very confident squad and have very confident individuals. That is the main ingredient of this whole touring squad.
"It's not all doom and gloom when you are losing. It's individually very character-building. It's a good team environment to build yourself within."