Brown and three team-mates faced unsubstantiated allegations of sexual assault during England's ill-fated 2008 tour of New Zealand, an episode he now reflects on as the darkest time of his life.
All four England players denied any wrong-doing and no official complaint was ever made against them.
A Rugby Football Union investigation, conducted by disciplinary officer Judge Jeff Blackett, found there was "insufficient credible evidence" to support the allegation.
Brown was fined £1000 for staying out all night after England's first Test defeat in Auckland and for arriving late to a physiotherapy appointment.
"The off-field stuff and getting dropped (by England) straight afterwards was probably the most difficult time of my life," Brown said.
"There was a point where I thought 'is it worth it if I am going to get this sort of thing?'."
But Brown had won three England caps, he had played Test rugby against South Africa and New Zealand and he wanted to prove he was good enough for the highest level.
The 26-year-old set about improving his game and restoring his name, with the help of sprint coach Margot Wells and Harlequins director of rugby Conor O'Shea.
Brown has been one of the outstanding players in Harlequins' unbeaten start to the Aviva Premiership season and he scored twice in their famous victory in Toulouse just before Christmas.
"Something like that (experience in New Zealand) gives people the wrong impression of you. That is nobody else's fault but my own. I had to go away and get my head down," Brown said.
"I worked hard on every aspect of my game and I think that is starting to show on the pitch. I think I have shown another level to my game this season.
"It took Margot and Conor to bring that level out of me, which I am really grateful for.
"I think there is still another level to give - there has to be if I want to step up and do what Ben Foden has done and play well in an England shirt."
England's interim head coach Stuart Lancaster will name his Six Nations squad on January 11.
Brown has carried with him one memory to drive that improvement, the moment against South Africa on his international debut in 2007 when he learned just what Test rugby was about.
"My first Test in South Africa was a real wake-up call," Brown recalled.
"I was the young guy, coming up from Division One, playing my first season in the Premiership and I thought 'yeah, I'd like a bit of international rugby'.
"I hit a hard line off Andy Gomarsall and got greeted by Schalk Burger and their centre Jean de Villiers and got absolutely smashed.
"As I was peeled off the turf by the physio, there was a realisation of the step up.
"It was Andy who introduced me to Margot and that was probably the biggest decision of my playing career. She has helped me off the field, getting my head right, and on the field getting my running right.
"Conor coming into Quins has been a blessing for me and Mark Mapletoft has done brilliant work with the backs, helping us score tries and get our attacking game going.
"It's been brilliant season for myself and the team."
Harlequins, flying high after their European triumph in Toulouse, face Saracens in the Aviva Premiership on Tuesday - a battle between first and second in front of an anticipated 80,000 crowd at Twickenham.
Quins centre Jordan Turner-Hall and prop Joe Marler both return from injury while Maurie Fa'asavalu returns at blindside flanker.
Saracens start with Andy Saull at openside after Jacques Burger was ruled out for eight weeks with a knee injury. Peter Stringer makes his first start at scrum-half since joining on loan from Munster.
Elsewhere on Tuesday, Leicester have rung the changes for their trip to face Worcester with Scott Hamilton, Matt Smith, Billy Twelvetrees, Ed Slater and Ben Woods all coming into the side that kept the Tigers' European campaign alive with victory over Clermont Auvergne.
Fourth-placed Exeter travel to fifth-placed London Irish, who are smarting from their European defeat to Racing Metro.
"Exeter is a must-win game for us," said Shontayne Hape, who returns at inside centre for the Exiles.
"This could put us back into the top four. The whole Premiership is so tight so there is massive pressure on us to win this game."
Exeter have won their last five matches and recall Argentina wing Gonzalo Camacho among four changes for their Amlin Challenge Cup win against Newport Gwent Dragons.
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The Barbarians are one of the top invitational rugby sides with a long history and classic rugby jersey.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Lions get ready for their matches with this green training jersey. It's what the players wear. Get in the Gear!
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
The Lions are ready to get their Australia tour underway. They arrive in Perth on Monday.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.