Harlequins battled gamely against Leicester, the club where Richards spent 23 years as player and coach, and led 9-5 at half-time.
But Leicester, who had scored after just 39 seconds through Tom Varndell, edged ahead after the interval with a try from Scott Bemand while Harlequins flanker Will Skinner was in the sin-bin for a late tackle on Paul Burke.
"The turning point was the yellow card," said Richards. "It really was silly. They scored the try when Skins was off and it meant we were chasing the game and it cost us dear. That is life.
"Skins will learn from it and a lot of the guys will learn from playing in this arena and against this opposition.
"There are a lot of positives. It doesn't give us the points we want but if we take the same intensity and passion and work on the small things the win will come. We are not worried about it."
Harlequins were missing key players like Paul Volley, Andre Vos and Stuart Abbott but, apart from their return, things barely get any easier over the coming weeks.
EDF Energy Cup games against Llanelli and English champions Sale await Quins over the next fortnight before a return to Premiership duty at high-flying Bristol.
"No-one here is panicking. In the first four weeks we have played some of the best sides in the Premiership," said Richards.
"We have accepted that if it doesn't go our way, so be it but we are not going to panic. It will turn round.
"I don't think it will be a survival fight. I know you are saying it is four played, four lost. But we will prove you wrong at the end of the year."