Basically I want to mix halftoned spot colors in the way you do CMYK, but with any combination of spot colors of my choice.
Photoshop has an almost-good-enough way of doing this with image>mode>indexed color, where you can specify a limited set of colors and each pixel can only take on one of those colors. You can separate the colors easily from there with the magic wand tool, but the pixel dithering effect obviously prevents you from being able to make good halftones with those separations.
sample image with indexed colors: white(ish), green, blue, black
I'm under the impression that some specialty software for t-shirt screenprinters does this, but I'm hoping theres a way to do it with adobe apps.