I go with eric on this one.
I have both an xd and and M&P (both in 4"), and I think your issue is caused by the reason I prefer the M&P. The xd's ambi mag release is recessed in the girp, and the M&P's mag release is larger, is flush with the grip and proud of the trigger guard.
This is a good reason to go to the gun shop, try out some new holsters, and add one more used holster to your box of "tried it but ..."
I honestly believe that the bulk of holsters sold in this nation are actually in gunroom boxes because they didn't work in the trial and error process of getting set-up.