Martin, who now works as a media pundit, made the claims in the build-up to Saturday`s clash between the two clubs at Upton Park.
But Downing believes Villa can capitalise on the Hammers "being down in the dumps" after their defeat at Bolton last weekend.
Downing told Press Association Sport: "He (Martin) is not in our changing room. He doesn't know anything. I think team spirit is the biggest thing here. We have stuck together, have a great set of lads.
"Team spirit is a big thing at every football club and fighting and tackling and making blocks and filling in for each other. That is all part of being a team. If it wasn't there, we'd be in trouble.
"That is easy for him (Martin) to say. Who did he play for, West Ham? I think he should look at West Ham. I'd tell you if the team spirit wasn't there."
Source: PA
Source: PA