Yes, there is a population per square loss at level 20, but that is reversed at level 21. Upgrade to level 21 immediately and you should be able to balance things out.
I did confirm that while I could have supported my manufacturies, armories, warehouses, and MH with the space available for housing in the fairy chapter (if I had ever built that out entirely... sigh, expansions), the same is not true for the Orc chapter. Using Elvenarchitect, I was able to confirm that the available space - with expansions - for all upgrades but no other additions to my infrastructure cannot be supported by the available space remaining for houses and cultural buildings. So I believe the incremental gain in population per square - and the gain in culture per square - isn't enough for the upgrades offered.
I will need to have Mountain Halls and Abyss wonders, as well as another cultural wonder to add to my Monastery, in order to have the population and culture necessary to support the infrastructure I have and could support without those under Fairies.
It will be interesting to see what they do for the next chapter, and if this trend will continue.