The newly-returned Ruan Pienaar got down under the posts to earn a dramatic bonus try after Ulster had broken out of their 22 in injury time with the last move of the game.
Ulster's other tries came from Paul Marshall (2) and Andrew Trimble, with Paddy Jackson making a vital contribution of 21 points from the boot on a pulsating night for new Ulster coach Mark Anscombe's side.
It was not all plain sailing for Ulster, though, who lost Stephen Ferris prior to the game with a back injury.
Castres made a brighter start and Romain Teulet slotted a fourth-minute penalty after Ulster were penalised for dragging down a maul.
But Ulster responded three minutes later as Paddy Jackson slotted a straightforward penalty after Castres had strayed offside.
Ulster carved out a try in the left corner from Trimble one minute later.
A Castres high kick went loose and Jared Payne made ground rather easily to set Trimble up for the try.
Jackson then landed a superb conversion from the touchline and Ulster were in front, soon opening up a 10-point lead when Jackson added a second penalty.
Paul Marshall's chip and chase then led to Ulster's second try in the 21st minute, the scrum-half dotting down near the posts to give Jackson an easy conversion.
Castres needed to hit back and Pierre Bernard finally launched a cross-kick left for Marc Andreu to score.
Teulet added the extras but Ulster immediately hit back with Jackson's third penalty after Castres were pinged for not releasing.
Jackson's fifth kick from five attempts was the final score of the half and Ulster led 23-10 at the break.
It was not long before they had their third try, Iain Henderson's surge allowing Paul Marshall to dart over with a blindside break close to the line.
The decision went upstairs to the TMO before Jackson missed his first kick of the night when trying to convert.
He atoned for his miss three minutes later with a long-range penalty that took Ulster to 31-10.
The visitors hit back once more and, after pummelling the Ulster line, winger Marcel Garvey slid over.
Referee Andrew Small went to the TMO before the try was given and Teulet added the conversion.
Ulster's cause was hindered, however, when Tom Court was shown a yellow card for hauling down the Castres maul three minutes before the hour mark.
But Ulster still increased their lead through another Jackson penalty before Pienaar made his first appearance since the Four Nations Championship when he came on for Paul Marshall.
The home side survived Tom Court's binning but struggled to put any more points on the board, even with Castres prop Yannick Forestier being sent to the bin in the 75th minute.
But in the game's final play, Ulster broke out of their 22 when Johann Muller won a ball at the breakdown.
Luke Marshall's break set up Pienaar who showed a lovely swerve and pace to get through the last line of Castres' defence for the bonus try, which Jackson converted.
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
$29.99
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.