Since I've owned 3 XDs, I invested early in the B&J Machine Sight Tool. Happened to catch it on-sale at MWUSA. I've broken even so far over paying a gun smith.
For XDs you need both the basic tool and the hold-down clamp. Those suckers are in there tight. You will never shoot them loose. A sight might break but it isn't going to drift.