Very interesting question! But also very impossible to answer I'm afraid. I'm looking forward to see other answers. In the meantime, here are some problems to consider:
Separating a given color into to solid inks is not a trivial mathematical task. There might be infinite solutions, or for some colors there might be none.
Your main problem is that you need empirical data. You say this problem resembles CMYK separation. Yes, but only on the surface. In CMYK separation you have 4 known inks which each can have different raster percentages. You want to find four solid inks which together will create the given color. So you have infinite colors to choose from. And you don't even know how many inks to use to create a given color.
The only reason we can convert an RGB color to CMYK is because we have color profiles.
RGB profiles are made by agreeing on a standard (I don't know much details about this), setting up a screen to follow this standard, displaying different RGB values on the screen and measuring the resulting colors in Lab.
CMYK profiles are made by agreeing on standard (paper type, inks, amount of ink to apply, dot gain etc.), setting up a printing device to follow this standard, printing physical samples of different mixes of CMYK and measuring the resulting colors in Lab.
All the measured data is made into a sort of lookup table which enables us to convert from a color we display on screen to a color we print on paper. For a given RGB color our applications can look up in the RGB profile which objective color it has. Then the application can look up in the CMYK profile which percentages of CMYK to mix to obtain this color when printed using a certain standard.
If the display we view the RGB color on is properly calibrated according to the RGB standard and the printing device we print with is calibrated according to the CMYK standard, the colors we get on print should look similar to the color we see on screen.
When it comes to other inks besides CMYK, we simply don't have those measurements. Even if we narrow down the possible inks from infinite to only using the 1114 Pantone colors, we still have the same problem. We know (more or less) which Lab color a printed Pantone color has on coated and uncoated paper, but we don't know how they look when they are overprinted.
Adobes's applications can give us an overprint preview which, in most cases, is pretty precise, but that preview is based on some hidden proprietary math, not measurements. It's not accurate and even if we don't care about that we can't use it in our own code because we don't know the algorithm.
The 1114 Pantone colors can be combined in many ways. There are 63824579126 4-color combinations, 229791464 3-color combinations, 619941 2-color combinations and the 1114 single inks. If you had access to a giant table of all the resulting colors of these combinations then you could make a script which trough trial and error found the best set of 4 colors to reproduce a given set of colors.
To print and measure so many combinations seems to be impossible, so you need to dig deeper into color science to find some mathematical way of approximating overprints. Maybe other can point you in the right direction. It's beyond my understanding and beyond the color theory designers normally use in their work.