But Woodward is adamant that he will only consider a move to the governing body after the 2012 Olympics.
The RFU was plunged into turmoil earlier this year when chief executive Sir John Steele resigned from his post due to his handling of the recruitment of a performance director.
Woodward looked set to take the performance director position in March, but his interview was delayed and Steele and the RFU board suddenly changed the job description to take out any involvement in the running of the England Test team.
After reports that Woodward was not interested in such a weakened position, the RFU changed the job description back to its original specification - an embarrassing U-turn that ultimately cost Steele his job.
Despite this, Woodward, who is held in high regard at the RFU after he coached England to World Cup glory in 2003, decided he would no longer be considered for the job and insisted he would remain in his current role as director of elite performance at the British Olympic Association.
He has not ruled out taking on some role at the RFU afterwards though, and if they wait until August 2012 to appoint a new performance director, he could apply.
When asked whether the episode had put him off a return to HQ after the Olympics, he said: "No it hasn't put me off at all. They have gone through difficult times but there are some really good people there who I know well.
"They will not be proud of what has happened over the last few months but they will come through.
"I've always said that I would be here at the BOA until 2012.
"Even if the rugby job had come to fruition then for me it would have been something I might be interested in after 2012.
"There was never any doubt that any new job I'd start would come after 2012. I've spent five years getting ready for this (the Olympics) and I want to see that through. I'll re-evaluate where I am after that."
Woodward has been mentioned as a possible candidate to replace Steele, but he insists he would not be interested in taking the vacant chief executive's job.
"My expertise is what I do now - coaching and performance," he added.
"It's nice for people to say that but I will not be applying for the chief executive's role of the RFU."
Woodward, who currently works with coaches of all Britain's Olympic sports, admits he misses being part of rugby, but is happy to stay in his current role due to the attraction of being involved in a home Olympics.
"I'm really enjoying the job. It's one removed from coaching. There's not quite the buzz of the changing room but I can always go back to that," said Woodward, who was speaking at the Sports Journalists' Association lunch on Fleet Street today.
Woodward will take a two-week break from his job next month to fly to New Zealand to watch the man who captained his team to glory in 2003 - Martin Johnson - try to emulate him by winning the World Cup as coach.
Those hopes took a blow on Saturday when, just a week after beating Wales at Twickenham, England put in a poor performance to lose 19-9 against the same side.
Woodward hopes the defeat will serve as a wake-up call for the team, who he thinks have a good chance of winning the Web Ellis trophy.
"If you speak to any team they will all say that they do not want to be playing England in a World Cup," Woodward added.
"England have a good history in the competition now. It's a good team, a strong group of players.
"I think Saturday's game could just be a big wake-up call.
"It's a very simple game. They are not there to entertain, they are there to win the game. They will get through Saturday. It will set them back a little bit but no more than that."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.