The bonus-point victory sees Wasps move to the top of the Premiership, while Quins only have Worcester below them at the other end of the table.
"I'm not that concerned because I still have huge faith in my side," said Richards, whose team scored tries through Ugo Monye, Hal Luscombe and replacement Nick Easter.
"There are a lot of sides who have come here (the Causeway Stadium) and been beaten, and we're another one who has come here and been beaten quite comfortably."
Quins had three men sent to the sin-bin by referee Dave Pearson, and Richards admits his side's discipline needs to improve.
"The disappointing thing for me was how we were beaten," he said.
"Three yellow cards was not particularly good and they scored three or four tries in that period.
"It goes to show that discipline is of paramount importance."
Richards also revealed that former All Black fly-half Andrew Mehrtens will be out for six to eight weeks after undergoing a knee operation during the week.
His opposite number, Ian McGeechan, was adamant his Wasps team still have plenty of room for improvement despite being the only side in the league to still have a 100% record.
"We let a side massively off the hook and we should be relentless about the way we play and we didn't do that today," McGeechan said.
"We scored five tries comfortably and we could have doubled that.
"The only pleasing thing is that there's probably 50% more to come out of the team."
Wasps scored tries through Paul Sackey (2), Dan Leo, John Hart and prop Peter Bracken, while fly-half Dave Walder kicked 15 points in another assured display.