That's not entirely accurate. Only the Rim Worlds garrison was ever completely withdrawn, in 2755 as part of the "Birthday Proclamations" of Richard Cameron. The Taurian Territorial State was long-recognized as the most fractious and disloyal of the Periphery nations, and had a garrison no-smaller than any of the others as a result.
The Taurians being so resentful of the Star League was central to Stefan Amaris' master plan. He created analogous levels of discontent (and created a pan-Periphery Secret Army to even the playing field across) the Periphery realms so that when the Taurians began the Uprising, the other nations would easily follow suit into a mass revolt and draw off the SLDF from the Hegemony. Without New Vandenburg and Fort Gorki, Amaris would not have been able to have enough context to move the Rim Worlds Army in to "augment" the depleted Hegemony defenses.
When I mentioned infrastructure, I was referring to the orbital/space facilities that serviced the 31-ship WarShip fleet thought to be in existence in 2765. We know that during the Uprising, SLDF Navy sunk all but one WarShips belonging to the Territorial States, the sole survivor being a Concordat ship. My initial post was to suggest the SLDF Navy, as part of what seems to have been a campaign to hunt down these ships with a vengeance, likely ranged far enough ahead of the ground forces reoccupying the Concordat to the wreck the service and construction centers for their main prey.