Walsh was at an explosive Tour media conference later that year and Armstrong, when inevitably asked about the book, singled him out, as the journalist vividly recalls.
Amrstrong said: "Well as the esteemed author is here, I will answer this.
"And then he said 'Extraordinary allegations, no, extraordinary accusations must be followed by extraordinary proof.' Everybody thought that was a great one-liner."
Walsh had based his book on interviews with Betsy Andreu, the wife of cyclist Frankie Andreu, a former teammate of Armstrong, Emma O'Reilly, who had acted as personal masseuse for the now disgraced cyclist, and a former teammate from the 1990's, Steven Swart.
When details of the systematic doping carried out by Armstrong and his team finally emerged in a report by the United States Anti Doping Agency (USADA) in 2012, it vindicated the whistleblowers' stance -- as well as Walsh and his co-author Frenchman Pierre Ballester.
But back in 2004, with Armstrong at the peak of his fame, it was inevitable he would challenge such damaging revelations to protect his reputation.
The Sunday Times stood by their man and his story, but when libel action in the UK courts was commenced by the litigious Armstrong, they knew the outcome would probably be in the American's favor.
Walsh recognized the seriousness of the situation, but admitted his judgment became clouded.
"I'm there saying 'Well I don't care, I just want this stuff out there.'
"And I wasn't seeing reason to be honest. I would have been a bit of a nightmare from the legal department's point of view and they were right.
"It did cost the Sunday Times a million pounds, but the newspaper were tremendously supportive as was my sports editor," added Walsh, referring to the out-of-court settlement reached with Armstrong in 2006.
By then the Texan had retired from the sport for the first time, though the rumors would not go away.
Walsh believes that cycling's governing body -- the International Cycling Union -- bears a heavy responsibility for not cleaning up its own sport in the face of overwhelming evidence of doping, not just by Armstrong but other leading riders.
He is heavily critical of its chiefs past and present, Hein Vergruggen, who resigned in 2005 to be replaced by Pat McQuaid.
"They were the people whose ultimate responsibility it was to ensure that the riders riding clean were protected. They didn't do their job," said Walsh.
"McQuaid said he was very anti-doping, but he didn't want to find out the truth about Lance Armstrong. He wanted basically, to sweep it under the carpet, and in my opinion, his organization now cannot have any credibility as long as he's president."
Walsh's fellow Irishman McQuiad has a different perspective.
"Hindsight is an exact science and hindsight is 20-20 vision," McQuaid told CNN as part of the Changing Gear series. "Of course you would do things differently but that doesn't mean that I regret anything that I did.
"Many, many federations around the world told me that under no circumstances should I contemplate resigning," added McQuaid defiantly.
McQuaid is being challenged for the top job at the UCI by British Cycling's Brian Cookson, and Walsh, while not specifically backing any candidate, is convinced a change is urgently needed.
"I have been saying this since the whole controversy unfolded -- the people who were in charge during this fiasco, shouldn't still be there.
"If cycling could find a credible candidate within its own ranks to take over from Pat, it would immediately change the perception of the UCI and people would say 'You know what? Let's give this new guy a chance.
"And let him reassure us that anti-doping really is going to be the number one item on the agenda."