I've found that most AllSaints leathers will fit tight across my stomach or chest.
I always look first at the shoulders and how well it looks across the upper back. Then I make sure that it isn't ridiculously tight when I zip it up.
I'm 6'1 with a 40" chest. Half of my AllSaints leathers are medium and half are small.
As for layering, most of these jackets are designed to look very trim, especially in the arms, so usually a t-shirt is best, or a light knit. Anything...