High-flying Villa set themselves up for an away tie at Reading on March 7, but they made hard work of beating the battling Championship side.
Agbonlahor notched his 14th goal of the season in the 42nd minute only for Darren Ambrose to cancel out this lead with a 72nd minute penalty.
But Carew settled the game with two successful penalties in the 81st and 89th minutes.
Showing four changes from the side which hammered Burnley 5-2 in their previous game, Villa dominated long spells without being able to overwhelm the struggling Championship side until the closing stages.
Although without their skipper Stiliyan Petrov, who scored the late equaliser at Selhurst Park to earn the replay, Villa dominated the opening exchanges with his deputy Fabian Delph setting up most of the attacks.
But it wasn't until the 16th minute that Villa broke through the Palace defence.
Agbonlahor rounded Matt Lawrence only to see Julian Speroni make an easy save from the striker's weak shot.
Speroni was under fire and did well to keep out a high cross from Ashley Young and also a powerful drive from Carew as Villa strived to take the lead.
Palace had a let-off in the 30th minute when James Milner opened up the defence with a delightful throughball.
Stephen Warnock raced through unchallenged. He only had Speroni to beat but chose to square the ball into the path only to over-hit his pass to the relief of Palace.
But the London side could not survive indefinitely and three minutes before the break Villa deservedly took the lead when Agbonlahor scored with a glancing back-header from Ashley Young's left wing disputed corner.
Surprisingly Villa were forced back onto the defensive at the start of the second period with Ambrose testing Brad Guzan with a raking drive.
Villa continued to make hard work of progressing with Agbonlahor again failing in a one-on-one situation against Speroni.
The goalkeeper continued to have an inspired night with another fine save from Ashley Young to keep alive Palace's hopes of an equaliser, which arrived in the 72nd minute.
Alan Lee was brought down by Warnock as he was charging through and referee Martin Atkinson had no hesitation in awarding a penalty which Ambrose rammed home past Guzan.
But seven minutes later, Villa settled the game with the second penalty of the game.
Carew was brought down by Lawrence and picked himself up to fire his spot kick past Speroni.
In the 88th minute, Carew was again brought down by Lawrence and once again Villa's lanky striker converted his spot-kick.