Could I fit into a XS hoodie from hollister?

First off I bought a L hoodie from hollister not to long ago and it fits way to huge that it makes me look fat. I ALWAYS and ONLY get XS shirts from them though. including their tank tops, graphic tees, tube shirts, ect. Sometimes I'll get a S but that's rare.. lol I'm 5'6 and weigh 115-123. I'm deff. not fat. I have a flat stomach, I'm just worried about my arms being really tight and the jacket hiking up my back. /: so what do you think? honest-