OmniMechs were always advertised as the greatest thing since sliced bread solely because of their modular, freely-customizable loadout. In canon descriptions, that's portrayed as a massive advantage because of the easy customization (which doesn't happen much in any canonical material) and ease of repair and maintenance (which occurs off-board and has little if any boardgame effect).
Out-of-character, I think the concept was a logical extension of what had proven to be a major selling point for BattleTech: People just love the construction rules. The ability to build their own 'Mechs or customize canonical designs contributed greatly to the game's success.
As has been pointed out, there were problems with the concept however that apparently were underestimated. Record sheets and BV calculations have been mentioned; their modularity made OmniMechs inherently unsuitable for RATs and for tournament play (for people with a record sheet fetish anyways - I've seen people argue about the order of critical slots in a given section as if it mattered). It probably also dawned on FASA that they couldn't really sell new TROs if there was a series of customizable chassis that could be, well, customized. No need for more 'Mechs. Of course, we're still seeing more Omni chassis for some reason.
Thus, standard configurations were born.
And these were then retro-explained with easier logistics and ease of repair/maintenance. It makes a mockery of the concept, but I'd chalk it up as FASAnomics.
Another IC explanation I came up with is that the Clans forgot how to fight effectively over time. When Omnis were introduced, they probably were a superior weapon because Clan warriors still had some common sense. Two centuries later, when the Clan Invasion occurred, the Clans had degenerated into, well, Clans as we know ("honor before reason" and a penchant for fighting left-handed just to show off). They wouldn't use the potential of OmniMechs anymore on the battlefield, but despite not actually using Omni capability still regarded them as superior and fielded them as a matter of prestige (much like they have, but refuse to use, artillery and WarShips).
As for why the Inner Sphere would want to copy the concept and then also fail to use its potential, I don't know. Perhaps they wanted home-grown platforms to mount salvaged Clan pods on. Or perhaps they just created proof-of-concept designs. These same guys built "upgrades" with OS missile launchers, after all. There may be some IC reason that doesn't translate into game rules.