Northampton made up for their heavy defeat at Franklin's Gardens with the victory and the result keeps Saints above them in the mix with two more pool games remaining while Ulster, whose 13-game winning run came to an end in front of their home fans, are no longer in control of the group - even though they still managed a losing bonus point.
The visitors scored all their points in the opening 40 minutes with a try from number eight Gerrit-Jan van Velze along with a conversion and penalty from Stephen Myler.
Northampton deserved their win despite Ulster's frantic efforts to snatch the game, with Ruan Pienaar's last-gasp penalty to win it sailing wide.
Ulster's points all came from the boot of Paddy Jackson.
The home side looked shaky from the start and after Phil Dowson stole a lineout on the home side's 22, Saints - clearly up for this contest after last week's embarrassing defeat - pummelled the Ulster line.
The result was an opening penalty attempt for Stephen Myler right in front of the posts, which he slotted after six minutes.
The visitors scored next after spending much of the intervening period in Ulster territory and when Jackson failed to gather a Saints up and under, they snaffled possession and moved the ball wide into the right wing channel where number eight van Velze smashed over.
Myler kicked the touchline conversion and the Saints were up 10-0 up after only 20 minutes.
The home side needed to respond and nearly got away when Nick Williams intercepted an attempted chip over the top and had to make do with a 30th-minute Jackson penalty from a rare incursion into Saints territory to open their account.
Seven minutes later, with Ulster now on the front foot thanks to some surging runs from Nick Williams and Jared Payne, Jackson nailed a second successful penalty to cut Saints' lead to four points - which is how it ended at half-time.
The home side then got the start they needed with Jackson's 42nd minute penalty to put them one point behind
Saints came roaring back and Samu Manoa was penalised for a double-movement before Tom May fumbled a difficult pick-up with the line at his mercy, then Ken Pisi could not reach an over-cooked cross-kick with again a try looking on.
Ulster were riding their luck as the game approached the hour mark but somehow they got themselves into the Saints 22 for the first time in the half. Williams drove over the line from close range only for the TMO to rule against giving the score.
There was more drama with less than 10 minutes to go Myler's drop goal attempt was charged down by Ruan Pienaar but neither Darren Cave nor substitute Craig Gilroy could break free.
With less than five minutes to go, substitute Paul Marshall's break off the blindside of a scrum near Ulster's line made huge ground before Tommy Bowe was tackled and injured - but the Saints were penalised at the breakdown and Pienaar had a kick to win the game after waiting for several minutes before Bowe was carried off on a stretcher.
Fortunately for the Saints, Pienaar's kick was wide and they held on to take the four points and fully turn the tables on their opponents.
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.