This answer is using Illustrator CS6 screenshots and animations because... to date.. I find that the best version of Illustrator for a few reasons. The panel operations have been the same for a few decades. It's unlikely you'll find the basic panel operations different than what may detailed here.
The first thing to do with the Color Panel is to Show Options
if they aren't already displayed. This way you can see color sliders.

You'll see sliders for whichever color mode the document is currently in, RGB or CMYK....

The Color Ramp at the bottom of the panel is there as a general target area. It is not meant to be a precise color selector. You can increase the ramp size by increasing the panel size.

Again, this is just to generally target a color area - blue, red, yellow, etc. not pick a specific color. You can hold the mouse down as you drag over the ramp and the color will change in a fluid manner.
(The polarized display of the color ramp in the animations here is due to the gif format (for animation) and dithering. The ramp is always smooth within the application, other than the Web Safe
ramp. I'm uncertain if anyone actually uses "web safe" colors in today's world.)
Once you pick a general area of color from the ramp, you can then use the sliders above the ramp to further target the color you're after.

If you depress the Command or Ctrl key then click a slider and drag, you'll find all the sliders will move in unison.
For CMYK, A slider will only move in unison if the stop has a value other than 0 or 100%. CMYK slider stops "stick" at either 0 or 100% - see the K slider stop in the animation below and how it remains at 0 while other stops move. For RGB documents, slider stops will "stick" at 255, but not "stick" at 0.

This can be helpful to refine a color without overly altering it's hue.
The most useful part of the panel for many are the HSB sliders - Hue, Saturation, Brightness.

This is where you can tweak specific aspects of a color similar to the Color Picker options. You can increase/decrease Saturation to add/remove color vibrancy. Increase/Decrease Brightness to lighten/darken a color. Or alter the hue slider to get a different color range.

(Note: The Command/Ctrl shortcut trick for slider unison will not work on the HSB sliders.)
The annoying aspect of the Color Panel, at least up to CC2019, has been the inability of the panel to stick to HSB. The panel has a tendency to auto-switch back to sliders based upon the document profile (CMYK/RGB). This forces the user to constantly reset the panel to HSB sliders. Users have repeatedly asked for a fix for this, but as of CC2019 there is none.
With all this any color you want is achievable from the Color Panel. There's really little or no need for the OS-driven Color Picker in Illustrator. In fact, the Color Picker is merely there mostly so users see it when they expect to see it - a way of making Photoshop, Indesign, Corel, Quark, Xara, Freehand, etc. users more comfortable.
There's nothing wrong with choosing color from the Color Picker. However, often the issue can be the double-click to open the Picker, especially on things like Gradient Color Stops. If you prefer to use the Picker, there's no harm in it. But you may need to adjust your inherent desire to double-click to open the Picker and instead click an object (or color stop) once, then click/dblclick the color square in the Toolbar or Color Panel to bring up the Picker.
I'll be frank, I've used Illustrator since it's inception (v1.0) and I have never opened the Color Picker in Illustrator while working. Not once. I've only ever opened it to take a screenshot of it like I did for this answer. But that's merely my workflow and I'm not by any means advocating that anyone need work how I work. I simply learned Illustrator and familiarized myself with the Panel long before the Picker was ever part of Illustrator. However, when I hear or see users expressing "pain points" with the Color Picker, my reaction is often to question why they are even bothering with the (unnecessary) Picker.
(For clarity, use of the term "Picker" or "Color Picker" above refers to THIS window.)