As shown in the image, "the" easily fits on the line above but regardless of what I do, InDesign pushes it down to the next line.
InDesign is using a muti-line composer (called Adobe Paragraph Composer) by default. It does not actually try to even your line it tries to even as many lines as possible within the paragraph. So it has detected that the other lines are better off with the 'the' pushed downwards. It creates a less ragged result.
If you want to use a simpler heuristic of one line at a time you can instruct it to do so by choosing the Adobe Single-line Composer option in the justification section of your paragraph style.
The other answers are correct that this is down to the work of the Adobe Paragraph Composer which is balancing your whole paragraph as opposed to each line individually.
If you do want your "the" to stay on the previous line (without changing to the Single-line Composer) just replace the preceding space with a non-breaking space (Type → Insert White Space → Nonbreaking Space or cmd+alt+X). Although InDesign generally does a good job at this so I would leave it without careful consideration.
As far as I know, InDesign is using the justifying algorithm of LaTeX (which is a very good one!). This algorithm is built to create an optimal justifaction for the complete page. So it could be that in a special line in your left justified text one word could be shifted, but the result of this shift would produce bad typography for the whole page.
If you use an algorithm looking only for a "good" justifying for one line you will get as a result a bad typography for the whole page as known from the usage of Word.
Usually I would say InDesign does a good job with the inbuild justifying algorithm. Better than you or I can do (supposed we both are not learned typographers ...)
If you work with Windows press Control+Alt+X between the two words to eliminate the forced line break.