My two bits: The Fox as designed is a pretty convincing assault transport. In particular, a Fox can jump into a hot jump point, divest dropships, take a few hits, and then do a preprogrammed jump out. If you ask yourself "what is the best system for doing this", the answer may not be very different from a Fox. If you go smaller the cost overhead of a warship eats you alive. And of course larger is ever more expensive with a serious 'to many eggs in one basket' issue.
The problem with the Fox, in my mind, is the choice of available dropships. Instead of investing in modifications to the Fox (which can of course use some improvement), investing in dropship designs seems advisable. I'm personally fond of 100 kiloton carrier dropships mounting something in the range of a half-Thera ASF complement with a 7/11 movement profile costing about 2B. There are limits to large scale dropships---you really don't want them landing on a world where they are a huge sitting duck. But, with some effort, you can use large scale transports in highly survivable ways to create an extremely capable invasion force off a Fox's 5 complements. Loaded up with 5x 100kiloton transports, the Fox is an adaptable (since you can choose the dropship loadout depending on mission) and formidable jump transport capable of supporting an entire invasion on its own. Thus, investing in large scale dropships using an engine slightly smaller than the Fox maximizes the value of the existing investment.
In terms of alternative warships, I'd consider a large 5/8 raider warship with no dropships to be an interesting complement. That would enable interesting and different strategies from the Fox-assault approach discussed above. However, you don't need many of these in a navy. Thus, my overall proposal would be a mixture of these two.
1. The only real issue I've ever had w/ the Fox itself is minor nitpicks about weapons choices & maybe having "too much" SI.
I'd have probably limited the SI on my "starter" ship to 80-95 range. Armor type being IFA isn't ideal when FC seemed to be the Norm/Most Common of the 4 naval armor types even if it's not the "best". But no one can say the Fox isn't well armored for it's size, its a brick of a Corvette, hell, its a brick as a Destroyer.
No, my problems have been little things... the 10* NAC10s isn't a good use of tonnage on such a small frame, less NACs over all &/or using some slightly bigger guns.
For similar firepower & layout I'd use 8* NAC20s & save a few tons for more overall firepower.
Ideally for a ship that size I'd use 6* NAC20's in only 2 bays (Either Fore/Aft Sides) and then use NLs in the other "quarter/side".
The 2 different NLs is also annoying, logistically just go w/ NL45's since this is a small ship.
Barracudas are nice but AR10's would have been better for sheer versatility.
Finally the MGs.... in the same arcs as AMS but didn't get slotted in as added AMS, so ... very... ODD.
No issues at all w/ the Quad LPLs, its solid Anti-Fighter defense there.
Overall, the only thing the Fox needs is a "Reskin" to change some weapons around.
Well, and a few more shuttle bays but then again, if your devoting 2 of your Collars to a Vengeance + Mammoth as part of your taskforce, then, you've got some spare shuttles there.
*****
2. I'm not sure I see a Behemoth sized anything as super realistic since any DS that sized is fluffed as being really rare.
Also the Thera was known to bankrupt the ASF fleets of the FWL being difficult to field that many on a single platform & stripping fighters from other formations.
So even having them at 1/2 a Thera but on a more common DS model isn't realistic for any faction's production ability of ASF.
Besides the Vengeance is already pretty rare, I'd work on getting more of them in production & bringing back the Titan to production.
You can always group up a few if you need an Aero-Brigade but otherwise a Regiment is plenty for most anything & can be split up into different locations.
*****
3. After years looking at 2/3 & 5/8+ ships, none of them impresses me. 3/4 & 4/6 are truly the sweet spots for Warships.
There are already "Real Life" limitations on what a WS Crew should be able to handle even if there is no "Game Rule" for High-G body stress.
The 5/8 should only see use in short bursts in combat &, IMO, isn't a big enough edge over 4/6 to warrant the loss of tonnage for Guns, SI, &, Cargo.
The Avalon & Mjolnir at 4/6 seem to have "Speed" covered from what I see.