TL;DR: 3025 era: JR7 Jenner. If you really need more JumpJets, then the Spider.
I'll talk about the 3030MUX, et al, experience here, as it seems to still hold the title of implementing the strictest adherence to the books while playing in realtime.
The first rule for a Scout, or Recon 'Mech was, if you were trading shots with the enemy, You Were Doing It Wrong. This was a pretty damned serious rule, and noobs who violated it ran a very concrete risk of losing their rides and getting reassigned to the Urbie.
The most basic quality of a 'Mech that you wanted was speed. "Speed Is Life" was the official Scout/Recon motto. Probably the most salient reason for this was that, in realtime, with imperfect information, there is simply no, or little, opportunity to calculate optimum paths, ranges and solutions. First, you had to be a pretty good pilot to drive something moving at 129kph and not 'cliff', which meant crashing into a hexside you couldn't climb. For a light 'Mech, this could, and did, result in a legged, or destroyed 'Mech. If you were out of radio range, and especially if you were in the off-hours, you could literally be stuck out there for hours before anybody even realized you were gone. Part of the morning agenda was seeing if any of your noob recon-scouts did exactly that, and got found by an enemy strike pack or wound up at the bottom of a lake or river. And if it was a valuable munch 'Mech, your ass was grass. Even if you wound up running right through an enemy formation, accidentally, with enough speed and luck, you could survive it.
The other reason for this was that, because of visibility, possibly having some ECM, and speed mods, with enough speed, you could become practically unhittable while yet remaining within sensor range of the target, while the enemy would be forced to tolerate the observation, or break out of their chosen defensive spot and try and chase you away. Of course, this could also result in the pursuers getting ambushed, so it could really put them in a jam. There were plenty of times where a lone scout determined the outcome of an entire two or three hour op, simply by making the enemy main body in a timely fashion without getting killed.
As pilots proved their skill at piloting, scouting the right locations, and relying the right info they could be moved up to more 'valuable', generally meaning faster first, then better armored, Recon/Scout 'Mechs, some of which might even be decent skirmishers. Especially with the later units, '67-'85, you could have a 'Mech with enough intrinsic speed to do scouting missions, and armor and guns to engage in a little light combat without getting one-shot disabled/killed.
'Time to study' and 'respawn', video and tabletop, still pretty much sort of corral you into a different set of tactics than a straightforward application of by-the-book BattleTech rules in realtime.
For the 5/8 & 6/9 'Mechs, I think there is definitely a role. but they might be more like a 'command' 'Mech for a scout element, or a backup to scrape enemy pursuit 'Mechs off a Scout's tail, etc, which are part of the Recon mission, you might say, in support of the Scout role, and vice-versa.