I had a wedding to go to, so I thought I'd make wearing a suit an exercise in concealment. I carried 3 guns that day - a Hi-Power in its normal OWB holster, a PA-63 in a 5.11 holster shirt, and a Ruger LCP in a pocket holster. Ah yes, and extra mags for all the guns - two for the PA-63. I decided not to wear the normal 2-mag belt carrier and instead pocket-carried (on my weak side) the mags for the BHP/LCP. PA-63 mags were in the strong-side pocket on the holster shirt. No one noticed. Next time I want to do 4.
For your case, obviously an OWB holster is not feasible for office carry. The 5.11 holster shirt is quite suited to the task, though. The gun is carried vertically, so that fits your need there. Since it would be worn underneath your shirt, there's no cover garment to worry about, and taking off your jacket would be a non-issue. It has worked well for me on many occasions. While I generally opt not to carry my Hi-Power and go with something else when a belt holster is not an option, I have carried the Hi-Power in the holster shirt and it did work.
You never stated what kind of gun you have, so if you clarify there you might get some more specific suggestions.