Players will be put through a series of tests over the duration of the three-day camp which will also have a strong focus on orientation for the 47-man squad ahead of next year's World Cup.
"We set some standards for this camp at the end of the spring tour and the four players have been sent home for failing to meet those standards," Wallabies coach John Connolly said on Thursday.
"They will resume training with their Super 14 teams tomorrow to start fitness programs."
He denied the sanctions would adversely affect the squad and said the move sent a strong message to those players harbouring ambitions of being part of the final squad to make the journey to France in September.
"Players were told at the start of the camp that if they were unable to meet certain standards they would be removed from the camp," he explained.
"We have very limited time available to us for physical preparation before the World Cup and this camp was to give us a platform from which to build on for the rest of the year."
The remaining 43 players will complete the camp on Friday having completed core strength, running and strength tests over the first two days.