So two basic things to get out of the way first:
- the color computations can be thought of as being done on each
single channel in turn.
- the numbers used are not {0-255} but rather
{0.0-1.0). So rgb(128,128,128) is rgb(.5, .5, .5) (or 128/256)
ACEkin is correct: overlay is a screen & multiply.
If layer B is <.5, multiply otherwise screen. (using your layer names; B is bottom layer)
A straight multiply is B*A=NewPx. Overlay doubles the result so that middle grey is preserved.
Multiply part:
2(B x A) = NewPx
for B=.49, A=.49, NewPx = .48 (grey on grey is nearly unchanged)
for B=.4, A=.2, NewPx = .32 (dark grey on medium grey result in darker)
for B=.01, A=.01, NewPx = .00 (black on black results in black)
Multiply makes things darker, since multiplication on any two fractional parts will always result in a smaller value, and in terms of color definitions that means "darker."
The Screen portion is doing a multiply on the distance between the color and white, which should darken it, but the result is subtracted from white which (I think) reduces the impact (?)
1-2(1-B)(1-A) = NewPx
for B=.5, A=.5, NewPx = .5
for B=.6, A=.8, NewPx = .68 (light grey on medium grey results in lighter)
for B=.99, A=.99, NewPx = .99 (rounded, but slightly closer to white)