This is essentially what the color management engine does. What you do is you convert your color to linear, and then you formulate a the primaries as a matrix. You may want to formulate your color space using 4 properties [Red, Green, Blue and White point], or by 3 properties [Red, Green, Blue]
How it is done
The matrix formulation to 4 properties is described in length by Ryan Juckett although he describes how to convert a linearized sRGB to LAB conversion. It suffices to substitute lab primaries with your own. I am leaving this derivation for a different forum since it is mostly out of the scope of this site, Photoshop can do this (see below). Most graphic designers aren't so fluent with Linear algebra. Remember to convert to whatever non-linearity your lights expect.
The formulation with 3 properties is quite similar. The first step is to linearize (again, since linear algebra does not work with nonlinear values) except you just plunk your primaries into your matrix (as row or columns, your choice). invert it and use that matrix for conversion. Convert to nonlinear again.
Doing this in Photoshop
Now as I previously mentioned this is what color profiles do. Photoshop has a rudimentary way to define custom color profiles. Anything more complex you'd need to make the profile yourself as described above in the math section.
This is what you do:
- Choose Edit → Color Settings...
- In this dialog enable More Options, if it is enabled your fine and it reads Fewer Options
- In the Working Spaces drop down choose Custom RGB...
A new dialog opens,
Image 1: Example of conversion dialog, insert your values.
in this dialog write your:
Note this does not help you to see what things look like with different primaries!! (for one a lot of data was lost capturing and second your still looking with your primaries) You can only get channels that show intensities at these primaries. But if you want a better emulation, turn the channels to spot colors. But this emulates how humans see with human primaries
But this wont work for simulating different organs
Real environment data is not like the one you see in images. What we call color is only a sensation. If you want to simulate a different sensing then you need to scan the world differently than you do now.
For one color capture in cameras assume you have a sensing organ like a human. Now a different creature might have different primaries but also different distributions of those primaries in spectra. See red does not exist in reality, it exist in your brain, there are many different distributions of light that give you a red sensation. Some give you the same sensation event though they are different (called metamers).
Image 2: Human color response for each color sensor
Shifting the elements around and their distribution changes what goes where and how its sensed.
If you really want to emulate this what you need is a spectroscopic camera, or get a spectroscopic image and then integrate that data by your own response curves. This is much more accurate, although how to show or how to process things that is another thing.
You can find hyperspectral images here If you want to do more serious work on this.
Whew, thats it. Good luck