I agree. The shoulders are the first priority for fit and you look great as far as I can see.
I have several AllSaints jackets that all fit properly all over, but almost feel like a wetsuit when zipped all the way up. Thing is, they always look better open, or 1/3 zipped, anyway so I don't sweat it.
If you want to layer for warmth, get some trim cut merino or cashmere sweaters.