Below is Illustrator's "Recolor Artwork" window. I've chosen to recolour (yes, I'm British) using one of my swatch groups. Notice that the first source colour is replaced with the first swatch group colour which is near black and which I don't particularly want. I want to manually choose another swatch but can't seem to figure out how...
I've tried every combination I can find in the dialog and the most immediate solution I've found is to enter the swatches numbers in the sliders. There's a "randomise" swatch order button so I would think there must also be a way to choose them manually.