If you want to reblend in Photoshop the merged layer and the original background to recreate the original foreground layer and use normal 8 bit colors, the answer is "It's not possible"
But 16 bit colors have the needed headroom. The following image has a foreground which is only one color and varying opacity. The image mode is 16 bit RGB.
Probably you know the normal merging equation:
M=rgb value in the merged layer, x=the opacity of the foreground color in scale 0...1, F=rgb value of the foreground layer, B=rgb value of the background layer
This formula is calculated for every pixel and and separately for r, g and b.
The opacities of the foreground pixels can be calculated if rgb numbers are known in every pixel in the original foreground, background and merged layer.
M-B and F-B can be achieved ok with blending mode difference. The sign of x will be finally right, because it must be positive.
So, we prepare the needed layers for subtractions. Below all there are spare originals.
The differences are peformed by merging the background copies down. The resulted layers are divided. The result should be the original foreground color opacity:
We can test it by placing it to the layer mask:
Quite close, altough not exact. Merging the recreated merged and making a difference with the original merged we can find with the color picker that the biggest difference has brightness 4%
Add due the extended question:
The questioner wanted to know is this still solvable, if a)the foreground color was unknown b)the foreground color was unknown and varies along the image
The merging equation gives in a single pixel actually three equations, but four unknowns (opacity x and foreground R, G and B). If we assume R,G and B to be the same in the whole foreground and write the same equation in another pixel where the background color is the same, but the merged color is different and not the same as the background, we get another three equations, but only one new unknown - the opacity in the new pixel. That must have exact solution.
Only 2 pixels is needed to determine the unknown R,G and B. But that cannot be done with Photoshop's layer blending modes, because they work per pixel. You need to solve the equations or have an iterative searching algorithm.
Case b) is more complex. If there's only smallish number of different RGB combinations used in the foreground, we have much more equations than unknown variables. A clever algorithm can guess the right places to take pixel samples like we could see it and calculate the colors iteratively.
Unfortunately I am not a programmer, even less a math programmer, so I cannot show a program which solves b).