The reason people generally have passion for anything is that they have a motivator to do it. It doesn't matter if it's graphic design, car repair, physics, obscure math, or Legos, they have some sort of motivation to do what they do. Some may not even know what their motivation truly is, they "just need to do it".
It seems as if you don't have motivation in your current line of work. That's perfectly fine. Most people hate their jobs, so you're quite a few steps ahead of most people. This alone isn't a great reason to stay where you are, but something to think about if you decide to leave (there's always a "worse" option).
Many people never find the motivator they need to find passion in anything, however some of these people may simply never recognized or capitalized on their passions. It happens, but only if you let it. If you take the time to search for your passion, you might be able to find something that you really would be rather doing. It might not be in design, and that's fine! It could be something related to design, just not in the specific version you are in right now.
My mother is an artist. For years, she was a digital graphic artist. Right now, she's into water color pencil. She has also done sign painting, stained glass, oils, acrylics, portraits, still life, sculpting, knitting, quilting, and so many other things that I don't know them all. The thing I'm trying to get at is that your passion might change over time, and it sounds as if it might already have.
You got into graphic design, presumably, because you really liked it. You had a passion for doing it for the sake of doing it. You are now a skilled designer, and your passion for learning your craft is lessened, since you aren't learning something new all the time. Notice that I changed how I spoke about your passion there in that last sentence? I moved your passion from design to learning about design. I'd suggest taking more classes after work, any classes you can find that even remotely hold your interest (and are affordable). These could be at a local college, museum, maker space, a gallery, or even just a friend.
See if you can rekindle the passion you started out with, not by quitting your job, but by expanding what you do. This may lead to a new job, sure, but make sure you are leaving your job for a good reason, rather than an "eh" excuse. Granted, some of those "eh" excuses can lead to greater things, but they usually have at least some sort of plan behind them.
Quitting your job without know where you want your path to lead you is going to be problematic, and finding your "perfect" job will be nearly impossible if you don't know what it looks or feels like.