Remember, the Great Houses were deploying forces and fighting too. It wasn't purely the Star League. In fact a majority of the fighting forces were House units. So the Star League has to save face, the Great Houses also have to save face. They weren't just subservient; they were partners in that venture. To varying degrees, they supported it and wanted it to happen. They were vocal and public in that support. So a defeat doesn't just shame the League it shames them too.
I think some version of a periphery victory doesn't bring down the Star League. I think the Great Houses and the Star League, together are largely forced to save face and justify what happened, and preserve the support of their respective populations in the face of horrendous military losses (particularly losses among the Great House troops deployed).
Some might distance themselves from the Star League to do that (that doesn't mean leave the Star League, just get critical of how the war was waged while being a SL member), but I think most actually just latch onto the idea that the periphery powers are SOO strong, that the Inner Sphere needs to recognize them as a legitimate threat and so it's important that we band together in the face of this existential threat.
Remember, one of the key events prior to the Reunification War was some fighting between the Federated Suns and the Taurians... that the Taurians won. The Federated Suns reaction to that was to decide the Taurians were a great threat and needed to be fought. Think of it like a repeat of that on a bigger scale and involving the SL and all the Great Houses.
So most likely the Star League stays intact, with a lot of finger pointing and blame thrown around in every royal court. But the Star League and the Great Houses are able to channel a lot of that blame on the enemy. The Star League pulls back, it writes the story to tell the tale of the Star League standing up to the strong periphery powers who killed so many of the brave Star League's soldiers. It builds, it militarizes even more, and eventually there's the Second Reunification War. Or some long-standing status quo where the periphery nations remain separate from the Star League, locked in a perpetual cold war for generations.
It's also important to remember that the Great Houses were kept in check from fighting each other a lot by the Star League. Any one Great House pulls out of the League, the gloves are off. If the Capellans back out of League membership for example, then Camerons get to look at the Davions and the Mariks and give them the nod to attack, with plenty of SLDF support. Batter the Capellans down a few pegs until they are brought back onto the League. Star League membership represents a kind of check-and-balance. The first Great House to bail on that, loses that protection and just begins to look like a tempting target to the rest.