The title of the article 'Tauba Auerbach’s RGB Colorspace Atlas Depicts Every Color Imaginable' is misleading.
As you already said RGB is intended for screen display and not print. It is - as far as I am aware - impossible to faithfully reproduce all RGB colors using offset printing. Even if it was possible to reproduce all RGB colors, no RGB color space contains 'Every Color Imaginable'. Another thing to note is that RGB and CMYK are not color spaces, there is no such thing as a single RGB color space. RGB and CMYK are color models which are effectively meaningless without an associated color space, which defines the gamut (effectively the available colors). Adobe RGB and sRGB are examples of color spaces.
It is Art.
The book is a piece of Art, created by an artist. There isn't a lot of information on the artists website or in the Colossal article, but I'm assuming faithful reproduction of color values wasn't a priority - it is a piece of art, not a reference book.
How was it done?
Without any solid information all we can do is guess, but I'm assuming the images were created in RGB and at some point converted to CMYK. The conversion could have been done early on and attention paid to getting the colors as close as possible to the RGB values or it may have only been converted at the printing stage - which would mean some of the colors being printed are very far from what they should be.
The pages are printed using an offset printing process. This related question talks about ways that can be done that get closer to representing colors in an RGB color space. The cover and page edges we're airbrushed (i'm assuming) by hand.
Digital offset print on paper, case bound book, airbrushed cloth cover and page edges
Either way - the book isn't an accurate representation of any RGB color space - it's just a cool piece of art.