They weren’t buying them and no they shouldn’t
You can't buy what's not available.
If the portion of people who use non-mechs and will use non-mechs in the future if they haven't yet were as large as you seem to think it is we would be seeing more support for non-mechs.
Also while yes there is enough demand for novels to be viable right now I am unconvinced that is a good indicator for how much of the actual player base actually cares about fluff. At least when it comes time to put minins on mapsheets. Again I feel it pretty safe to say if as many players cared about fluff the way you seem to think we'd be seeing a lot more support for certain product types that we aren't seeing right now.
Not including certain unit types in future TROs is not effectively retconning them out of existence. Until players use certain unit types at all without house rules they still exist.
I grant there's always been more interest in mechs. That's why there's only a couple non mech TROs and those were done a long time ago. Those since have been mechs with other units.
Fluff makes the universe. Without the fluff we might as well be playing chess.
Their not being in a TRO doesn't make them retconned out of existance. Their being not included in the next rule book retcons them out of existance. Being in a TRO didn't make the Snowmobile Combat Mod Legal. Being in a rule book made it legal. It being absent from the next book retconned it out of existence. That's what makes me worry about the future. If TPTB are abandoning, not just unit types but key units for an era, like the Elemental for Clan Invasion, will those other units be in the next rule book?
I think one thing that the grognards like you aren't thinking about is that new players aren't going to be all that interested in 3025 as they're going to want the newest stuff.
As a new player I was interested in the universe as a whole. I wanted the newest stuff as soon as it came out but I also wanted to know what happened in the past. The more that was published the richer the universe became. As a grognard I would like that to continue. I understand that some will only want the newest but there are some who will want to explore the entire universe. Only they can't do that if the older books aren't available. It's like getting someone started on Star Wars Episode VIII and telling them that they can't see the previous seven episodes.
I doubt that will happen. There is some core stuff that doesn’t ever really change much in BattleTech between the different editions. I imagine we will still have hexes, walk/run/jump will still be +1/+2/+3 to-hit, kicking will still be easy, etc. will infantry and vehicle rules stay the same? Will all the tech as of 3150 still be included? Will walking into water still carry the potential for a PSR that could result in a fall, breach, and instant kill? Those might be more iffy.
Also, most of the revisions haven’t invalidated recordsheets either, which is always helpful. At least, they have remained close enough to be playable. The layout and details on recordsheets have changed a bit over the years as tech was added and construction rules changed, but not so much as to invalidate them completely, generally speaking. I doubt any rules refresh after a time jump would invalidate all the existing recordsheets...but I guess they could.
Anyway, I’m just guessing about all this. I could be way off base and Total Warfare could be around for another decade or more. It just seems that, time-wise we are at least due for a “Total Warfare: Revised” if not a whole a whole new base rulebook, and the end of the Dark Age would be a convenient time to do something like that. I’m just reading tea leaves over here, so don’t worry too much about anything I’m saying just yet!
It does feel like a new rule book should have come out a while ago. I hope that the layout is done better though. All the page flipping and being in multiple books gets annoying. I also hope that all the currant units are included.
As for the "bloat" I can see the setting having a more unified tech level, with the currant level of tech being "retro". Nothing is removed. Nothing invalidated. Just rendered obsolete in the way Primitive Tech gave way to Standard. Like in the Third League PDF. The older equipment/units (Currant tech/units) still work. They're just not as good. That's what I hope for any way.