Unless you are going to be using either a typically translucent colour (yellows, oranges), or are all about painting in very thin layers, the primer colour will have very little impact. You've seen the effect of grey versus white primer on yellow, and I wouldn't recommend priming in black for a white or very light paint scheme, but beyond that it's not a deal-breaker. The primer is meant to give you a bonding layer between the bare metal and further paint layers. A thin coat is all you need, and you really shouldn't be relying on it to fill every crevice and completely obscure every trace of metal. Usually doing that means you've over-done the primer (which probably isn't a huge deal in itself unless you've REALLY gone overboard). Save the base coat for the actual base coat.
A couple examples of what I mean by the primer colour not mattering much:
1) an Avalanche that's mostly white. Although I did use a white primer, my base coat is more of a medium grey to establish the deepest shades. The white is then brought back up with highlight layers.
2) a Mad Cat mkII E that's primarily near-black. That started with a light grey primer. Even if I had gone with black, I'd still need to lay down a black base coat to ensure good coverage, and it would make getting the yellow-orange sections to pop that much harder.
