The hosts had led 26-13 with eight minutes to go but a Dan Murphy score and a penalty try put the Aviva Premiership side ahead by a point with time running out on a rain-sodden encounter.
But the Ospreys responded emphatically, driving number eight Allen over the line against some tentative defence to seal an excellent win.
Wing Tom Grabham also crossed for the Ospreys, with fly-half Matthew Morgan kicking four penalties and two drop goals, while replacement Sam Davies, son of Gloucester boss Nigel, slotted five points.
Prop Yann Thomas got the first of the visitors' three tries with young fly-half Billy Burns, brother of regular Cherry and Whites number 10 Freddie, kicking eight points before suffering a shoulder injury.
Tandy was delighted by the character shown by his side, who were led magnificently by captain Sam Lewis and well-prompted by fly-half Morgan and scrum-half Rhys Webb.
The performance underlined the talent the regions are churning out at the end of a week that has seen further hand-wringing over the flow of Welsh talent out of the principality, with Jamie Roberts set to be the next high-profile player to ply his trade elsewhere.
Tandy said: "It was more nailbiting than I would have liked, if we had not come away with the spoils I would have been disappointed.
"But it was testament to the maturity the guys showed to try and build for a penalty first and then to go for a try, that was the icing on the cake. They thoroughly deserved the win.
"We had lots of young guys out there and all in all it was a real positive night for Ospreys rugby.
"I probably got a bit carried away at the end because I know what that meant to the group.
"But we had some guys tonight making their debuts or only their first or second starts and it is something we can feel really positive about."
Gloucester director of rugby Nigel Davies, meanwhile, had few qualms with the result, but was frustrated by how the Premiership team had let the game slip away from them at the death.
He said: "We just didn't manage the game and the conditions well enough. The Ospreys handled that far better.
"We put a lot of pressure on ourselves for a large amount of the time. They had a better kicking game than us, we had no length and no accuracy with that.
"That was the deciding factor, we need to play the game in the right areas and we just couldn't do that.
"It was fantastic to see the way the boys fought back. But it was sloppy in the end to leak that try, having worked so hard to get back in front.
"We probably didn't deserve the lead but because of our resilience that's what comes with it.
"And we should have nailed it and finished it off right there.
"But If I'm honest it's a fair result, I don't think we could have expected much more."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
$29.99
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.