Because this topic came up several times and it intriqued me I have started a new city. Won't ever be anything large and won't stay long, but I wanted to see what happens when you don't build the barracks.
First, chap 1 is going to take quite a bit longer. In chap 1, you don't have a lot of resources, the questline is an important source of much needed coins and supplies. However, once you hit the quest to build the barracks you are stuck as it cannot be declined, so you will have to limp along until you get past the advanced scouts for chap 2, which will render all chap 1 quests optional. And you'll have to skip several and lose those rewards, some of which were deliberately designed to help you speed along the first chapter(s).
Second, in chap 2 you'll run into the same issue at some point where certain troops are required and upgrading the barracks to a certain level. The training ground with additional such quests can at least be built and sold off.
With the low yield (low amounts of relics and low lvl factories) for T2 goods catering provinces will be a pain too.
In chap 3 same thing again with quests to train certain units and thus getting stuck on the questline and missing rewards that would help you speed along.
Best if you want to try this is to find a good FS that could help you along with some 1-star trades and thus donate some goods to give you a boost, but that goes for any city. The missed rewards nobody can donate to you.
All in all I found it is possible, but painfully slow to get to chap 4 without a barracks, but it can be done. After that things slowly get easier as you can place quite some extra factories when not having the barracks (nor armories, which you can build and sell off for quests).
As for the barracks not being able to be sold because of being a 'core building' I find that no longer true.
Yes, it WAS true because when elvenar was started a new player did not get a whole bunch of troops to start with, there was no training grounds, nor a mercenary camp, nor were there troop-instants or any kind of buildings that created troops except the barracks. Playing without any kind of troops at all was all but impossible. That just is no longer the case; there are multiple other ways to get troops without having a barracks, it is not even the only building where troops can be trained, which means it no longer is a real core building.
This really should be considered by Inno, and it should be made possible to sell it off. If someone later wants to rebuild it, spending a lot of time and resources doing so, there is nothing against that.