Slightly tangentially, it might be the gun. I took a pistol course where we shot every students' gun. The idea was that all guns are more similar than you'd think. And, the guns in the course did vary from the squeeze-cocker HK P7, to Sigs, Glocks, 1911s and more. And yes, there were similar to shoot, and I wouldn't mind owning and shooting any one of those. However, I bought a Beretta 92 once, and HATED it. Like M25 mentions above, it just shot weird to me and seemed to have more recoil than all other 9mm I'd shot. I sold it a few months later. I saw someone post saying that the XDm has a high bore axis. Does that, plus the lightweight frame equal higher muzzle flip? Would that make it worse to shoot? Is the slide on the 92 lighter than normal?
I may have to try what is mentioned: and shoot some qualifiers with a 9mm vs a 45acp. I recently bought a .45acp that's the same model as a 9mm I've owned and enjoyed for years. I wonder if the 9mm would outshoot, both in time and/or accuracy, the 45acp? How many cycles would I have to shoot to get an answer that removes most randomness?