Last evening I pulled out the "holster box." I have a Fobus paddle holster in there that is used for demonstration purposes only. I like the holster itself, but I hate the paddle - it digs into my rolls.
I found that I can make marks in the holster material with a little effort, but that same process with more effort does not affect my XD or M&P. With that I would say that if you don't allow dirt or other abrasives to build up in the holster it will not harm mar the gun. However, remember that if you let crud build up in leather it will mar any material there as well.