The back-row forward has been repeatedly penalised in the Wallabies' three Test matches this season, including receiving a yellow card for three consecutive ruck infringements in the first Bledisloe Cup game against New Zealand.
Elsom claims he is now in two minds when he goes into a breakdown, giving an example from the most recent Test match against the All Blacks, in Brisbane last weekend.
He had already been penalised twice and deliberately eased off at the breakdown rather than risk another costly trip to the sin-bin.
"There was a moment in the game when I'd already given away two penalties and I was tackling a bloke and the referee [Alain Rolland] was saying, 'hands away, six'," Elsom told The Australian.
"I wasn't on the deck but I didn't see [opposite number] Jerry Collins anywhere and I thought, 'maybe my night at the breakdown is over'.
"It's a bit early to be calling 'hands away' when no one's on the deck."
Elsom is worried about attracting a bad reputation, causing referees to penalise him whenever anything goes wrong at the breakdown. Â
"When you're getting pinged, you've got to really watch what you're doing," Elsom said.
"It's very difficult to know where the line is.
"You've got no option but to pull back from the breakdown from then on. Probably, all you can do is lay low until you stop getting attention from the referees.
"I guess, whether you're right or wrong, if you keep giving away penalties, you won't be out there."