With full credit to @Wolff for these images. As was discussed, grey is 100% relative to lighting, surrounding colors, perception, and the method you are measuring. Take these images for example. The main image actually has no "red" in it at all. If you take an eye dropper and measure any area that looks red, they are all actually shades of grey.
if you take a snip of the parrots wing and take it out of context, it looks like browns and greys.
If you put that exact same snip back over the parrot, it looks red again in context with the blue/green.
But that being said, LCH or LAB would be your best bet to measure for neutral grey.
@Luciano's recommendation of this article: Color Theory: Is there a measurement of "colorfulness"? Has some good information for what you are trying to do.