O'Neill, who has spent close to 14 years in two different spells in the role, was the main administrative figure the last time the Lions faced the Wallabies back in 2001.
He was due to continue past a second Lions tour in 2013 but an increase in his commitments outside of rugby has led to his early resignation.
"John's workload beyond rugby has recently grown significantly and unexpectedly," said ARU chairman Michael Hawker.
"John had previously indicated he was standing down when his contract ended next year, so the timing of his resignation is now logical for both him and the ARU.
"His commitment to the game over such a long period of time has been extraordinary and we know that will continue in the future.
"Under his leadership, a Governance review into the game has been initiated; planning for the 2013 Lions tour is well underway; the Wallabies have improved from fifth in the world when John returned to the game in 2007 to now second behind the All Blacks; participation levels are at an all-time high; the ARU has strengthened its financial position; we have increased influence at the International Rugby Board; Rugby has been further expanded with a fifth team in Melbourne; and the new Rugby Championship has been launched with the inclusion of Argentina, refreshing the Southern Hemisphere Test competition.
"John has been a great servant of Rugby in Australia over two decades and has achieved the key objectives of a five-year strategic plan since re-joining the ARU in 2007.
"John's resignation enables the Board to put in place new leadership to build on that platform and to drive our substantial agenda over the next five years, while enabling John to focus on his broader business interests."
O'Neill, who will continue to serve as an ARU delegate to the International Rugby Board, as Chairman of the IRB Regulations Committee, and as a Board member of Rugby World Cup Ltd, insists Australian Rugby is in a good position to push forward without his leadership.
He is convinced the building blocks are in place for the Wallabies, and for the Australian game in general, to progress to the next level.
"I am honoured during my second term at the ARU to have played a role in shaping the transformation of the game and I look forward to seeing Australian Rugby go from strength to strength," said O'Neill.
"Australian Rugby has a rich culture and a proud history, but we have also been working to rapidly modernise and evolve the game.
"I believe the key planks necessary to drive that development are in place, allowing my successor to build on the solid foundations Rugby enjoys and take it to the next level of success.
"There are other challenges for me to pursue outside the game now, although my involvement at an IRB and Rugby World Cup Ltd level will ensure I have a strong ongoing connection."
O'Neill's deputy Matt Carroll will become acting CEO from November 1 as the ARU begins a global search for a long-term replacement.
No timescale has been put forward to announce O'Neill's permanent successor but the ARU are seeking the best of the best ahead of a hugely exciting year of rugby.
"The ARU has a big agenda and there is much still to be done to strengthen the game," added Hawker.
"We will be seeking another world-class sports administrator who can deliver the ambitious plans we have for the future of Rugby in Australia."