Exeter went into the break 19-3 ahead through a try from Gonzalo Camacho which Gareth Steenson converted, and the Irishman followed that up with three penalties with Andy Goode replying with a late penalty.
The former England fly-half added two more after the break before the game petered out until the final 10 minutes when the Chiefs scored tries from Haydn Thomas and Chris Budgen.
"It was four very valuable points but I can't help but feel that we left a point out there, not least because large parts of our game held up well and should have given us a foundation to push on a little earlier than we did," said Baxter.
"The enthusiasm of the players to defend was far better than last week [at Bath], our toughness on the game line in the first half was better than last week and our maul defence was fantastic.
"That [the maul] is probably Worcester's biggest weapon and they got no change there at all and our scrum was pretty dominant.
"Those kind of things should have given us the platform to push on but in the second half we lost the momentum for 20 minutes.
"Once we got that first score in the second half, we really could cut loose but I am a little disappointed that it took so long for that score to come.
"If we had controlled the momentum a little better and pushed on five minutes that fourth try would have come.
"You have to give Worcester credit, they are a good dogged side and they are not just going to give up on the game.
"They kept coming at us in the second half, they got some reward for their pressure and it made an awkward game for us."
The defeat means that Worcester have not won on the road in the Premiership since New Year's Day against Wasps.
And Warriors head coach Richard Hill accepted this was probably his side's worse performance of the season.
He said: "It was difficult to put your finger on exactly what was going wrong but it was across the board with a lot of inaccuracies. We had a very poor start, which puts you on the back foot.
"You can't come back here after being 19-0 down at one stage against Exeter. That is what we have to address because that is the fourth game in the league where we have conceded a try in the first five minutes and it knocks the stuffing out of you.
"Exeter, as usual, were pretty accurate because their skills are good. They very rarely make any mistakes and they put pressure on you because of that.
"But we were making too many errors, drop passes, lost the breakdown battle, couldn't get our drive going.
"When we did win the lineout, Exeter defended it really well, so we couldn't find a way to get our game going.
"Exeter were in really good body positions and they defended our drives and we are disappointed because we do enough work on the maul."
Gilbert has released a new line of rugby cleats. The Gilbert Virtuo 8S is part of the exciting new product. Check it out.
The entire All Blacks apparel line has been updated for 2013/14. Check out the New Zealand All Blacks polo.
The Nike Tiempo is a solid rugby cleat and one of few styles still made from full-grain natural leather.
The Gilbert Blitz 8S rugby cleat is a great cleat at a great price of $69.99. Get a new pair of cleats today.
A cool looking all black rugby cleat with the high performance adidas is known for. Get in the Gear!
Wear the crest of the British and Irish Lions on your t-shirt. A great look for the summer.
The All Blacks Performance t-shirt is black with hints of blue from the training jersey. Very Cool.
The New Zealand All Blacks training jersey for 2013/14. Get in the Gear!
The USA Rugby Pro Alternate rugby jersey is perfect for any fan of the Eagles. Get yours to wear during the summer Test matches.
The NEW All Blacks 2013/14 jersey has arrived at World Rugby Shop. Dare to wear the colors of the All Blacks.