Friction, i.e., direct contact, with the leather (or kydex) is what causes the holster wear.
Your "mechanical transfer" doesn't occur until long after the holster wear has already occurred/begun. Your theory has a great deal of chemical interaction in it. It's that "paste" interacting with the finish with the gun, especially when the gun is in the holster, don't you think?