Lindsay-Hague touched down in the 84th minute to help Quins, who had trailed by three points heading into the final three minutes of regulation time, claim the victory.
Harlequins went into the match boasting a 100% record at the top of the Aviva Premiership but were without their former All Black fly-half Nick Evans, who has been rested for two weeks after playing a major role in the team's flying start.
Rory Clegg has been handed the number 10 jersey in his place and it was him who engineered the opening score for Ben Urdapilleta in the seventh minute.
Clegg's direct break into the heart of the visiting defence was ended by a strong tackle, but a good off-load gave possession to centre Urdapilleta, who dummied to switch the ball left to Sam Smith only to step inside once the defence was committed before crossing the line.
Clegg added the conversion, and his determination to take the game to Irish saw the visitors on the back foot and defending waves of high-speed attacks.
However, it was Irish next on the scoreboard when their most incisive attack led to Harlequins offending at the breakdown and Tom Homer's towering penalty was on target.
Having absorbed the home side's exciting opening barrages, Irish entered the game's second quarter looking increasingly confident with ball in hand, with ex-England caps Paul Hodgson, Nick Kennedy and Topsy Ojo showing evidence of their quality and experience.
Irish could have reduced to deficit to one point in the 28th minute, but Homer's penalty from 50 yards just faded wide.
But the fullback was back on target five minutes later, thumping home a penalty awarded for offside 30 yards out to make it 7-6.
Within three minutes, Clegg restored the gap with a well-struck goal as Quins, fielding a less experienced line-up, made their first serious visit to the Irish half for 15 minutes.
That score lifted the hosts and they squeezed another penalty out of Irish on the stroke of half-time, but Clegg's powerful kick crashed back off the left-hand post.
Quins began the second half as they had started the match, driving forward in confident style.
Irish conceded a penalty at the breakdown during their defensive efforts and Clegg's kick sailed straight between the posts to put his side 13-6 ahead.
A marvellous break by Urdapiletta then ripped open the Irish defence and left three would-be tacklers clutching fresh air. But, with the line beckoning, Quins contrived to waste a genuine try-scoring opportunity with poor passing that led to the ball going into touch.
Irish made the most of that escape by breaking out for their first serious attack of the half on 53 minutes and won a penalty on half-way for not releasing. Homer kicked a fine goal to reduce the deficit.
Then Quins pulled down a driving maul in their 22, gifting Homer his easiest kick so far, and he chopped the lead to a single point at 13-12.
But captain Will Skinner roused his young side to lift their heads and led the charge that forced Irish to offend at a ruck, the confident Clegg firing home his fourth goal from five attempts.
Irish stormed back again and, after coming close to a try in the 67th minute, it was the turn of the hosts to handle a siege as their line was pounded.
Richard Thorpe spun off a scrum five and crossed the line only to be held up. But the visitors were not to be denied and David Sisi was driven over on 73 minutes.
Homer converted that to put Irish 19-16 ahead with time running out, but Clegg hauled Quins level with a penalty on 77 minutes before replacement Lindsay-Hague's late converted try sealed the win.
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The Barbarians are one of the top invitational rugby sides with a long history and classic rugby jersey.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Lions get ready for their matches with this green training jersey. It's what the players wear. Get in the Gear!
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
The Lions are ready to get their Australia tour underway. They arrive in Perth on Monday.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.