This is to be expected
Floating point color is not something you should be using unless you have floating point data sources. High Dynamic Range (HDR) images are a advanced imaging concept, right up there with the fact that human color perception is relative.
The short answer is that this will not work out for you.
The long answer is that HDR color can never be shown directly, it has to be conditioned before display. Otherwise the color values above 1.0 could not be displayed. As such the data in a HDR image is considered to be linear, anything other would make no sense. The net result is that your image is going trough a HDR conditioner AND a linear to RGB transformation (what depends on your color profile most likely sRGB*). In theory you should be able to reverse these transforms with a profile to profile change to get the color you want.**
However the conditioner is meant to be relative, just like a camera. This in turn makes the problem a bit hard to solve. But then, there is really nothing to solve, other than eliminating your mental pedantry. Your supposed to evaluate the color relation by eye NOT by numbers.
So the HDR mode is meant for data measurements and the conditioner is meant for making it look good. The integer color values shown are ONLY as a hue guide that's it.
* No theres no such thing as non color corrected HDR
** Doing this is certainly a good exercise, I've done it once. Unfortunately explaining how to do it would take up far too much time. Lets just say that IF you need to be this pedantic then you do not want to use 32 bit mode because it works on the concept of relative color and conditioning which is not what you operate on.
