The Japanese, who are bidding to host the 2011 Rugby World Cup, looked to have taken the game by the scruff of the neck when tries from Goshi Tachikawa and Takashi Kikutani gave the home side an early 14-5 lead.
Spain came back however, with a drop goal from Cesar Sempere and a try from Pablo Feijoo reversing the scoreline in the Europeans' favour.
A second drop goal from Sempere further extended the visitors' lead to 18-14, before Japan collected their thoughts and pushed over another try from Takashi Akatsuka to restore their advantage at the interval.
The second half started evenly until Japan's superior kicking finally turned the match for the home side.
Veteran Keiji Hirose notched two penalties and a drop goal to open up a lead for Japan and, although David Mota grabbed another try for Spain, Nataniera Oto and Hirotoki Onozawa crossed the line late on to put some make-up on the final score.
It was only the second meeting between the two sides and marked a winning start for Japan under new coach Jean-Pierre Elissalde, who took over from Mitsutake Hagimoto in the summer.