You lightweights shouldn't have a problem. As a matter of fact, I've come to think that we've got it better than the beanpoles.
I am 6' and 335, but when I started carrying a 5" 1911 IWB I was about 360. I've also carried my Kahr K-40s and revolvers from J-frame to N-frame without any discomfort.
It all comes down to a very good holster and belt, which won't be cheap. Still, you should be able to equip yourself nicely for around $150-$175, maybe a little more since extra length belts are fairly expensive.
My personal preference, going back a decade, are those made by Tucker Gunleather.