Because mercs are COOL. That aside...
On mercs in EuropeMercs didn't become a thing because of war or peace or surplus in trained warriors. Mercs became a thing because of 1) fragmentation of society, 2) task-specialisation, due to expanding economies, and 3) the invention of money. In short, societies moved from groups of farmers defending themselves part-time from the predations of their neighbours, to groups of farmers led by soldiers using money to hire professional soldiers to help defend them from the predations of their neighbours.
Mercs died out when societies grew large enough that they could raise their own armies, which in the economic sense means they could essentially pay people to be loyal. Thus, condition 1) was invalidated.
On mercs in BattletechMechs and perhaps more importantly
Mechwarriors are rare. Mercs represent that small % of these resources that prioritise money over national loyalty. The big Merc outfits don't really count, they're like Boeing or British Aerospace or Elbit - nominally private companies, but generally loyal to that one nation. Its these small guys who make up the bulk of the trade.
The world of Battletech is more fragmented than ours mainly because of the interstellar barrier. Hence there is at times a relatively sudden and unexpected local need for extra military might, and that is where mercenaries thrive. Don’t look at them as soldiers, look at them as bank security guards in a world where bank robberies are a lot more prevalent and the police a lot slower to respond.
Hiring mercs is like hiring temporary staff versus permanent staff: they are cheaper and more liquid than State troops; you hire them when your business anticipates a spike in activity, and you fire them when activity levels go back down. All overheads included they cost you less than full-time employees, and being of a temporary nature you can push them around while your full-time employees need to be handled with kid gloves, especially in BT where a significant proportion of them are your cousins and nephews and nieces etc. Temp staff come to you equipped and pre-trained; they're selling themselves as a wholly-inclusive, self-contained, full-stack unit. That takes pressure off your supply and admin network. Lastly, the really desperate ones are expendable - use 'em, abuse 'em, pay any survivors their due and then eject them.
That last factor is a big attraction to people to become mercenaries. They are basically taking on huge amounts of risk in order to hopefully make the big payoff - kind of like the private investment market. And just like said market, you'd see 80% of the private guys lose their shirts to the institutions and the 20% who made the right call or got lucky. So yes, the industry makes little macro-economic sense – but it can make a lot of
micro-economic sense.
You're not entirely wrong though - we did see the results of peak saturation in the merc trade in the late 3060s: the 1st major event of the Jihad, the "bursting" of the merc "bubble" on Outreach and subsequently the rest of the IS. So yes, there was a point when the industry became too big to be supported, people with guns couldn't get hired, and with a little motivation and capital injection from the Blakists to accelerate matters, those people decided to turn those guns on the authorities.
If you think you have a use for them, heck, if you just want to pay to have them as a deterrent against your neighbor getting uppity, the salary & maintenance is minimal.
In a world where every house is starved for mechs, I don't think there is ever a "drop them from the payroll consideration".
You might drop them for Loyalty, or because you have 0 interest in going on the offensive, or because you want to use those funds to raise house troops.
But I doubt many houses are out there just dismissing Mercs because they don't "want" to have an extra few battalions on call.
Remember, the DC tried to Black Mail the Dragoons into staying because they did NOT want to give up access to that many trained mechwarriors in 3025.
The FedSuns is HUGE, Hanse was only too happy to have those troops to help defend his boarders.
As far as affording goes. The Militia & Regular Army figures in BT, even at their largest, are nothing close to the kind of density that the USA or Russia have. I don't think tax dollar cost is the issue.
WW2 soldier, looking at today's army: "How can the US not have 10% of the population in the army?!"
Nam soldier: "How can the US not have 2% of the population in the army?!"
11th-century militiaman: "Doth not every man, villein, freedman and knight alike owe serjeanty and knight-service unto his liege-lord?!"
As technological advances extend life expectancies, make life more comfortable and make each individual soldier more lethal, people become less willing to risk their lives - especially if the invaders can minimise their practical impact on daily affairs - and smaller armed forces become more capable of controlling larger populations. And money is always, always an issue, because the national budgets reflect more-or-less the priorities of its people; money spent on the military is money not spent on making life more comfortable, 1 way or another.