I have primed miniatures in sub-freezing temperatures, but when I do, I bring the miniatures inside to dry right after spraying on a coat of primer. I also make sure that the spray can is at room temperature before I head outside to spray.
I find that high humidity is usually a bigger problem for me than low temperatures.
(I'm not saying that this is good practice, but it has worked for me.)