An own goal from Abdoulaye Meite gave Villa an early lead before Peter Odemwingie levelled with his 14th goal of the campaign.
Albion's Paul Scharner was sent off after 62 minutes for two bookable offences, but Mulumbu netted the late winner.
West Brom have now taken 16 points from nine matches under head coach Roy Hodgson, but they fell behind after four minutes as Gabriel Agbonlahor held the ball up inside the Baggies box before picking out Stewart Downing and Meite deflected his low cross past Scott Carson.
Scharner became the first player to be booked for a challenge on Nigel Reo-Coker but Albion levelled when James Morrison's deep free-kick was headed back across goal by Jonas Olsson and Scharner miskicked before Odemwingie lashed the ball into the roof of the net.
Two minutes later, they were reduced to 10 men when Scharner hauled down Stiliyan Petrov and was sent off. Then Mulumbu grabbed the winner after 84 minutes when he ran onto a pass and evaded Luke Young before a challenge saw the ball ricochet off Mulumbu's standing foot and over Brad Friedel.
Source: PA
Source: PA