Villa twice looked set for victory after a John Pantsil own goal and Kyle Walker's first ever league goal put them in the ascendancy.
But Andy Johnson and then Dempsey struck back for Mark Hughes's side.
Villa dominated the first half with winger Stewart Downing causing plenty of problems and linking up well with full-back Walker down the right flank.
Newcomer Jean Makoun also impressed in the middle of the park and Villa deservedly led through the unfortunate Pantsil. He had also scored an own goal in the corresponding fixture last season.
But it was a different story in the second half as Villa lost momentum and the visitors grew in confidence with skipper David Murphy impressive in the middle of the park.
Source: PA
Source: PA