The 34-year-old will compete with Darren Edwards, signed from London Irish, for the number nine jersey as the Tykes chase an immediate return to the Guinness Premiership.
Rauluni, currently playing for Fiji in the Pacific Five Nations, spent two years with Rotherham after a stint with the Queensland Reds.
"Jacob is a vastly experienced scrum-half who will bring a great deal to our squad. Along with Darren Edwards, we will now have strength in depth in a crucial position," said Tykes director of rugby Stuart Lancaster.
Leeds have also signed Mike Aspinall from Harrogate as their third-choice scrum-half for next season.