Several points here. A mercenary unit isn't necessarily a feudal organization. The unit CO is the boss, he doesn't hold the unit as a vassal of anyone. Therefore, if he doesn't want to pass it to a blood heir, he doesn't have to. Many units do, many units don't. Likewise the mercenaries in the unit aren't vassals of the CO. If he appoints an incompetent successor, they can leave. As such, there's good incentive not to pass the unit to an inexperienced or incompetent heir, at least not immediately.
I'm not equating mechs with land exactly, but there is
usually a strong link, especially at lower levels; the noble is granted land in return for service, usually as a mechwarrior. If the noble can no longer provided the required service
or cannot effectively rule their fief, they're likely in danger of losing said fief. The two go hand in hand. The idea of simply giving the mech to the best pilot would be as likely to get the family stripped of lands & title for incompetent rule as giving the mech to an incompetent pilot would get lands & title stripped for not fulfilling military obligations. Also, I doubt you'd find historical cases, or many parents who'd wish to strip their children of their inheritance. Besides, was every member of the extended family trained from birth to possibly inherit the fief? The best mechwarrior might be a completely incompetent Lord. I don't think it's realistic at all to think a noble could just hire someone to rule their fief and not need to do so themselves. Again, I don't know of any historical or canon precedent for that. I doubt their noble superior is going to accept "The guy I hired to do my job screwed it up." That's a good way to get stripped of your title. I could perhaps see the mech being loaned out to an aunt/uncle or cousin or niece/nephew, etc if the noble ruler was incapable of piloting for some reason, but not the designated mechwarrior being given the title as well. That's completely foreign to any depiction of feudalism or neofeudalism I've seen.
Is there any canon precedent for property being held communally by an extended family? That's how I'm understanding the argument here, that whoever in the whole family is the best warrior inherits the mech and the fief as a whole each generation. Is that what you're saying? I can't think of any example of that happening.
I think that when it comes to succession disputes, what's good for the goose is good for the gander. Sure, the House Lord wants an established succesion so as to prevent fratricidal conflict, but
so would lower-ranking nobles. The danger of fratricide doesn't disappear because the inheritance is a resource-poor continent on some Outback world instead of a whole realm. In fact, if a family had such a conflict it'd be a pretty good reason to strip them of lands & title and give it to a more deserving ruler. As it says in Handbook House Davion:
A noble leader has the right to remove from power nobles subordinate to them, subject to the approval of his or her direct superior, bestowing that noble title upon the successor. In cases where an entire noble family is implicated in wrongdoing or shown to be incapable of governing their landhold, that family may be stripped of its title