I also say use a retention holster.
I was talking to one holster maker about doing a holster for me and he started laffing at me when I said I wanted him to make it for me with a thumb break on it. He told me that that was an insult to him and would not make it for me, of course I didn't use him.
I allways look at it this way also as to why you need a retention holster: you go to a public restroom and sit down on the thrown, now take a look at where your gun is and how easy it is to be grabed away from you, that is how many pick-pockets grab wallets and stuff from you also.
Ya it sounds funny and you say "that will never happen to me" but it is at a time when you are most vorable and not in an easy spot to help yourself.
I also have trouble keeping a gun in place when I am carrying IWB. I'm 6'3", 310 Lb's and have had 3 back surjurys. Trust me I am allways readjusting my pants so it is much easyer for the gun to "come out of the holster" without some kind of a retention holster.