The Light 'Mech is built for the same reason 20th Century navies built a lot more Destroyers and Light Cruisers than Heavy Cruisers or Battleships. If a Destroyer is adequate for 90% of the tasks, why spend the money for a Battleship to do the same job? When that other 10% happens, the Destroyer is fast enough to run away until a real Battleship can get there to deal with the problem.
A Light 'Mech has the versatility to perform in circumstances where a vehicle is at a disadvantage, or totally incapable of operating. When a vehicle can perform the job, you send a vehicle; when it can't, you've often got a Light 'Mech that can. When the Light 'Mech isn't enough, you sent a bigger 'Mech, and it then boils down to who sends the most and/or the biggest 'Mechs to the scene first.
Unlike in the typical "battlefield-only" game, there are peripheral functions behind the lines that are not represented on the table, such as logistics, command, and support personnel and equipment. Light 'Mechs would often be employed in hit-and-run raiding against those supporting assets, rather than trying to stand toe-to-toe against heavier combat units on and around the battle line. Since those assets are not represented on the board, there's a lot less of any importance for a light 'Mech to do. When the map is the size of a postage stamp, there's simply not enough room to make speed and maneuverability viable alternatives to armor and firepower.
The problem is not with the Light 'Mech, it's with the limited representation of battle as a purely local and static situation.