The Australian became Scotland's first overseas coach when he accepted the challenge in December 2003.
However a poor run of results resulted in Williams being dismissed in April 2005.
Scotland won just three of 17 Tests under Williams, but he claims there was opposition to his leadership based on more than rugby grounds.
He told Irish magazine Emerald Rugby he came into the job unprepared for the reaction he believes met his appointment.
The former Leinster coach said: "The thing I didn't know was that Berti Vogts was a German coaching the football team, and I didn't realise the massive antipathy towards him.
"It was wrongly placed because it was based on racial grounds and unfortunately when he left, that got placed on me.
"It didn't really matter if I was any good or not - I wasn't a Scot. So that whole racial thing was pretty hard to accept.
"The game is becoming more of a soap opera and once you become the story, it stops you being able to coach.
"I became the story simply because I wasn't Scottish and we weren't winning.
"There was no public acknowledgement that there was a four-year plan and that there was a blood-letting phase."