I carry and handle every gun as if it was going to fire if I touch the trigger. I mainly carry Springfield XDs now because I like them overall and I like that I can handle it without touching the grip safety. I am much more confident with an XD than with my Glocks because of that. I also intentionally bought a IWB that I cannot reholster while on my body so I would have to remove it, place it on the weapon, and return the gun and holster to my belt. I think once my gun is drawn I will chance any situation where reholstering is a must. Depending on the holster you use, this may be an option to get around any concern of ND while reholstering. Again, I would not carry a gun that was not ready to fire when I needed it. The noise of chambering a round is enough to get a CHL holder shot and destroying any advantage of surprise. It also takes away the opportunity to pull and fire in a moment the BG gets distracted.
Main thing is, pay attention when handling the firearm. If it cannot be done with all your attention then don't do it.
That is my opinion.