They should be AOEs
Sure, in an atmosphere, underwater, or underground, where there's a lot of matter to turn their energy into a shockwave. In a vacuum, you've only got a few dozen or hundred kilograms of bomb mass to work with (unless you get close to the target and can heat the target sufficiently). That makes nukes pretty toothless in space, but for their x-ray and neutron radiation.
The thing is, BT spacecraft aren't bothered by the big electrical surges from the x-ray pulse of a nuke, and they have near-magical radiation shielding. The only thing left is brute damage, which nukes do poorly in a vacuum. They're much more effective when they can superheat millions of tons of atmosphere, creating an explosion to smash objects within a few kilometers. Meanwhile, the scale of space combat (18km/hex) means that anything less than several gigatons isn't going to have an area of effect.
if the nukes were that toothless in comparison how they work in irl.
No. BattleTech's nuclear weapons were modeled very carefully on real life nuclear weapons, down to borrowing Cold War equations for primary and secondary effect radii and their vacuum behavior.