For myself I can say: I spent many years working as a graphic designer, both as a freelancer and working for agencies... during that time I was also working as a technical illustrator and technical writer; later I went back to school yet again and moved into what I had always really wanted to do but thought I couldn't break into: architecture.
I've now spent a bit over a decade in the architectural and BIM field, and love it; I still do work in graphic design, technical illustration and writing, I do a fair amount of architectural illustration and rendering, and am now learning new skills in Unity game development to further my ability to handle real-time rendering Arch-Viz work.
Graphic design, as already said in another reply, is a discipline, an art of sorts, and a method of creating graphic languages: it's also a communicative field and a creative field.
It's not for the most part a fine art, albeit true that many graphic designers are also artists in their own right, and many well-known artists have pursued a "day-job" of graphic design to pay the bills.
That said, coming as I do from the intersection space of architecture, graphic design, technical illustration, 3D modeling, rendering and animation, having no degree in most of the fields in which I've practised successfully for years, my answer is: the best thing about graphic design is that it's portfolio (results) based.
If on top of great results, you can demonstrate a good, client-focussed process, you can easily overcome most objections. Conversely, if you now burn 3 more years of Uni and yet have no appreciable skills, talent or portfolio at the end, you'll have sharply limited success.
I'd say start pursuing design work now, do it as a sideline, and see if it's really your cup of tea.