It's not really about legibility. The text can be understood whether it's justified or not. The issue, as I see it, is about readability - how easy it is to read the text, how much effort must the viewer expend in order to ingest the copy.
The overall problem with full justification is accessibility. People with cognitive disabilities, or older audiences, can have considerable trouble reading justified text.
You mention "rivers of white" in the question. So, I know you're aware. However, merely because you don't perceive this problem does not mean others will not perceive a problem.
While these are geared towards web viewing, the overall theory holds strong for printed text as well. Fully justified text is simply harder to read for some.
Many readers can have difficulty tracking the next line in justified text. And, as with anyone, if tracking the next line is too difficult there's a probability reading will stop unless its mandated for some reason.
In addition, my opinion is that blocks of fully justified text are much less interesting and subsequently less intriguing in terms of visuals.
Same text, utilizing the same space.... but I find the right image much more visually interesting.
I feel this comes down to form v function. Which is more important...
form - readability is sacrificed in favor of the the visual uniformity created by justified text, or...
function - better/easier readability for the viewer
Humans see words as shapes. In the grand scheme of things this can be expanded somewhat to paragraphs. One can often recognize a paragraph they've read by the shape of the paragraph itself. This is admittedly a bit of conjecture, but looking at the image above, it would be easier to determine where you left off reading in the right image merely by the shapes.
My preference for body copy is to use ragged right text whenever possible and avoid full justification for all the reasons above. At times that means one needs to adjust word/letter spacing slightly or insert a couple key soft returns. I attempt to keep the ragged right within 3 to 5 or 7 characters if at all possible, and no hyphenation.
In my personal experience, I've never experienced anyone complaining that a block of body copy is set ragged right. However, I've often heard how difficult text is to read and ultimately determined it was the full justification. I avoid full justification as much as possible and really only use it when it is specifically requested by a client.
I see full justification similar to using too little or too much leading within a paragraph. There's a threshold that once you pass readability will drop considerably due to next line tracking difficulties for the reader. Too long of a line length can also cause the same visual tracking difficulty. In my opinion, this same difficulty is inherently built into full justification.
I do think full justification can be a necessary evil at times. In situations where there are hundreds of pages of text full justification can be a quick format to use to help avoid widows and orphans. Using ragged right text with a lot of pages can necessitate more manual "massaging" of text. I can see how full justification may appear more presentable given a tight deadline or desire to not visually ensure all pages have good line/paragraph breaks. I would rather put in this work myself. But there's no judgement if others don't.