I'm not an Affinity user but I think I can still answer this, because the same thing happens in Photoshop.
It's because the black that existed in the RGB document was not converted to 100%C 100%M 100%Y 100%K, when you converted it into CMYK colour mode, but some other mix of inks, and so the Invert filter produced an unexpected result.
It's basically inverting the CMYK colours (subtracting each percentage from 100%). So for example, a colour like 70%C 70%M 70%Y 90%K, which although it looks like black, will actually invert to 30%C 30%M 30%Y 10%K. Obviously this isn't white.
To fix it, assuming all the pixels are only black or transparent, and that you have already converted to CMYK mode, you could select the layer that has the pixels on it, then engage the Transparency Lock/Alpha lock in the layers panel (I assume Affinity has this function). Choose a foreground colour which is 100%C 100%M 100%Y 100%K and fill the layer with it. This black will now invert to a pure white (i.e. 0% ink) in CMYK mode.
The example below shows a similar problem in Photoshop. This image is in CMYK mode, the squiggle on the left is not 100%CMYK because it was converted from an RGB image, but the squiggle on the right is 100%CMYK. When I enable the Invert layer effect, the difference is obvious.