Is it possible to subtract two layers to calculate transparency?

I have an image with two layers: Backround and its copy + some modifications. Is it somehow possible to subtract colors of the backround layer, so that all pixels that are different will be extracted with transparency and extraction result combined with bg layer would look the same, or almost the same as the original merged layer?

Input: Possible result: part of the layer is opaque, part is transparent. Result is identical to the original merged layer. UPDATE

I wanted to use this to extract difference between original and modified layers so in practice both Background and Modified layers wont have one color. Here's a more real life example. I have a background and its copy with some brush strokes and Smudge Tool strokes. Here's the result I'm having now. I use Difference to spot changed pixels and then load it as selection, eliminate half-transparency from this selection and extract fully opaque pixels. This is a result I'd like to get: some pixels have transparency because Difference wasn't 100% white. I realize this may be not possible, but no harm in trying I guess. • Sep 3 '18 at 21:12
• @Joonas I updated my question, sorry it wasn't clear at first. Sep 4 '18 at 11:29
• This is not straight forward but if you mapped the image to black and white and used this method, you could extract layer masks and just refill with the original colors.
– curious
Dec 11 '19 at 4:01

If you want to reblend in Photoshop the merged layer and the original background to recreate the original foreground layer and use normal 8 bit colors, the answer is "It's not possible"

But 16 bit colors have the needed headroom. The following image has a foreground which is only one color and varying opacity. The image mode is 16 bit RGB. Probably you know the normal merging equation:

M=xF+(1-x)B where

M=rgb value in the merged layer, x=the opacity of the foreground color in scale 0...1, F=rgb value of the foreground layer, B=rgb value of the background layer

This formula is calculated for every pixel and and separately for r, g and b.

The opacities of the foreground pixels can be calculated if rgb numbers are known in every pixel in the original foreground, background and merged layer.

x=(M-B)/(F-B)

M-B and F-B can be achieved ok with blending mode difference. The sign of x will be finally right, because it must be positive.

So, we prepare the needed layers for subtractions. Below all there are spare originals. The differences are peformed by merging the background copies down. The resulted layers are divided. The result should be the original foreground color opacity: We can test it by placing it to the layer mask: Quite close, altough not exact. Merging the recreated merged and making a difference with the original merged we can find with the color picker that the biggest difference has brightness 4% The questioner wanted to know is this still solvable, if a)the foreground color was unknown b)the foreground color was unknown and varies along the image

The merging equation gives in a single pixel actually three equations, but four unknowns (opacity x and foreground R, G and B). If we assume R,G and B to be the same in the whole foreground and write the same equation in another pixel where the background color is the same, but the merged color is different and not the same as the background, we get another three equations, but only one new unknown - the opacity in the new pixel. That must have exact solution.

Only 2 pixels is needed to determine the unknown R,G and B. But that cannot be done with Photoshop's layer blending modes, because they work per pixel. You need to solve the equations or have an iterative searching algorithm.

Case b) is more complex. If there's only smallish number of different RGB combinations used in the foreground, we have much more equations than unknown variables. A clever algorithm can guess the right places to take pixel samples like we could see it and calculate the colors iteratively.

Unfortunately I am not a programmer, even less a math programmer, so I cannot show a program which solves b).

• There used to be a plugin for Photoshop called AlphaWorks that might be able to do this, but I don't know if it still works since it needs a 32bit version of Photoshop, and GIMP has its Color to Alpha filter, which can do this easily. But nice answer nevertheless. Sep 4 '18 at 10:24
• Thank you! If I understand correctly, in your 'FG color copy' you used the same blue color as in the FG layer, but I wonder if the color is unknown? And if it's not one color? I realized that my question wasn't 100% about what I'm trying to make, I updated it. Sep 4 '18 at 11:29
• Yes the foreground can be inferred if you can extract a good enough alpha. I have explained the process here Dec 15 '18 at 22:15

With solid color background you could use "background eraser tool" Discontignous, tolerance 70-100% + some foreground color protection. It is similar like "color to alpha" in Gimp. But you need bitmap to alpha and I don't know this in Photoshop.

Or you could modify this: http://cmtt.github.io/color-to-alpha/

• Hi Jaroslav, I updated my question: actually I don't need to transform one specific color to alpha, I'm trying to understand if it's possible to subtract two layers, if both of them aren't simple colors. Thanks for the link though, it's also useful Sep 4 '18 at 15:03
• I think you can modify it to read one pixel and apply it on area 1x1 px in image and repeat it for all background pixmap. Anyway it should have ugly performace so with if you will rewrite it more then it could be fast. Sep 4 '18 at 16:07