After reading up on YUV (or YCrCb) a little bit more, I realized that in order to use it properly all components need to be the same bit depth (ideally), and you need all three components to do the conversion properly. Meaning Y, U and V. So making Y a float and UV unsigned chars would save me a bunch of memory during star generation, but then creates a huge headache for colourization.
So by making them all floats I'm not saving much, and by making them uchars, I'm losing my HDR. And because I'm NOT too worried about obscene optimizations I figure there wasn't much in terms of savings between 2 uchars, and 3 uchars. Thus I settled on keeping my stars HDR using a float, and keeping my colours RGB using uchars.
Once I'm ready to write to file then I can do all the fancy conversions :)