The Premier League side looked to be in for a comfortable afternoon when Milner fired them in front early on, but the Gills hit back strongly and Simeon Jackson claimed a superb equaliser just before the hour mark.
Hopes of an upset were dashed when Ashley Young went down in the penalty area, with Milner converting the spot-kick 11 minutes from time.
The home side were forced into a reshuffle with Garry Richards and Barry Fuller suspended and Albert Jarrett out with an ankle injury. Manager Mark Stimson's lack of options was shown with his substitutes, six of whom were teenagers.
The pre-match optimism around a packed Priestfield lasted precisely 14 minutes, when Villa took the lead with a flash of Premier League class.
Adam Miller was dispossessed by Milner, who swapped passes with Stiliyan Petrov to carve open the Gills defence, before firing first-time past Simon Royce.
The hosts wobbled briefly, but falling behind appeared to help them settle. Dennis Oli shot into the sidenetting and Jackson almost capitalised on a sloppy back pass, but Zat Knight came up with a perfectly-timed tackle.
The home side continued to enjoy the best of the game after the interval and levelled on 57 minutes. Jackson, who gave Knight problems throughout, spun his marker before lashing home his 12th goal of the season.
The goal stirred Villa, who set up camp inside their opponents' half. Gills defended well enough until 12 minutes from time, when Young fell under a challenge from Miller.
To the home fans' disbelief the referee pointed at the spot and Milner coolly converted to send Villa through.