Y'know, as a pre-Tomcat pure fleet-defense interceptor there's just not a lot it can really do by comparison to the Double Ugly. It's faster than the F-4, slightly, about 50mph according to Wikipedia. Combat range is equal to the Phantom as well, and while it can climb faster than the Phantom it surrenders a good ten thousand feet of altitude. I think the other reason that it never went beyond a "hmm, what if" stage?
Imagine trying to LAND a 104 on a carrier. That would be a hot screaming nightmare, doubly so if the blown flaps failed (which was not uncommon). Coming in 30 knots faster (140 vs 170 approach speed) means you have a LOT less time to react to things, and on such a small target...yeah, no. Takeoff speed for the F-104 was also much, much higher than the air-grabbing Phantom; the typical liftoff for a Starfighter was around 190 knots - with an M61A1, 725 shells, and four Sidewinders. The Phantom did it with 4.5 times the amount of ordnance...