You don't have to build Armories (there are quests which want those but I believe all of them can be skipped), but if you intend to battle you should definitely have some. The bigger your squad size the more armories you need to create units. If you don't want to fight, then better save the space and use it for more goods buildings (you'll need the goods).
Yes, battles become harder and harder and the way they work is weird (for example the damage does not decrease with each defeated unit from a squad, there's only a very small percentage decrease even if you defeat most units in a squad). But that information is made available upfront, it's even pointed out on their Wiki page and that would be the definition of intended behavior (something working as it's designed to). We may not like it and each of us can have our opinion on how it should be improved, but it's clearly intended behavior.