Unfortunately, everything is all about space and square footage. Assuming you've maxed out your space, how can you upgrade your other buildings (main hall, supplies, barracks, etc) if there is no way to increase your population ? you will need to sell off some . Example:
Lets say you have 10 city exp available for residences (2500 square units) you can build
31 x lvl19 2x4 = 17187 total population.
20 x lvl21 3x4 = 17708 total population.
Thats a gain of exactly 520 pop. that wont even get me my next silk upgrade.
Recall when res when from lvl15 (3x3) lvl16(2x4) they got smaller.
Mind you, you can always spend $300-$400 and buy snail culture building.
Ash