Ashton's position has been consistently undermined over the last six months despite him guiding England to second-placed finishes at the World Cup and in the RBS Six Nations.
Former World Cup-winning captain Martin Johnson is expected to be appointed team manager shortly, which could spell the end for Ashton.
Vickery told the Daily Mirror: "It doesn't matter what happens, I don't think certain people are ever going to be happy with Brian.
"Brian could go and win a Test series in the summer in New Zealand and something would still be wrong."
Vickery's position is also under threat, with Matt Stevens pushing to be first-choice prop, but the 32-year-old insists he still has plenty to offer England.
"I don't play rugby just to captain my country," said Vickery. "I want to be part of this team going forward in whatever role that is.
"I'm passionate about playing for my country and I feel I can still do it."