Why does a layer in overlay mode not show over black or white areas?

Question

Layer A is above Layer B. Layer A is on overlay mode and is filled with white, grey and pink colors. Layer B is split in 3 sections: white, grey and black. Why do the contents of Layer A not show over the white and black sections of Layer B? Could someone explain how do the color calculations work in overlay mode? I'm using Photoshop CC.

Result:

What I'm actually trying to accomplish is darkening the background while bringing out the subject of the following image. On the left is the original and on the right the result after applying the overlay layer:

Answer 1

The overlay mode is (kind of) a combination of multiply and screen. The middle gray has no change, any value darker is darkened and lighter values are lightened proportionate to their deviation from the middle gray tone (128,128,128) If you multiply white, the result is white; if you multiply black the result is black. That probably explains the behavior you see. The same goes for screen mode effect. In other words, you cannot make white any brighter and black any darker. The middle gray is not affected.

Is this a rhetorical question or is there an end result you are trying to get?

UPDATE

I now understand. Overlay blend mode is not the tool to use since it will lighten the lighter areas, opposite of what you want. Darkening white will result in strange grays. I think your best bet under these circumstances is to create an artificial look with a vignette. Here are the steps and adjustments:

1. Create an elliptical marquee selection as shown in the image, covering an area just above her head down to the mug
2. Right click on the selection, choose "Feather" and adjust the feather depending on the image size, as a starting point use about 1/20 of the long pixel dimension. You will not see much change after you click OK here.
3. Press Shift-Ctrl-I to invert the selection so that the outside is selected, see the image below
4. Add a Levels adjustment layer and move the sliders as shown in the screen capture, note the slider on the Output Levels bar, pull it to the right. You want something slightly artificial but not too over done. This will make the light areas darker and darken the darker areas as well
5. If you do not want the bottom part to darken, select the Brush tool, make sure its hardness is low (right click on the image with the brush to see the setting), pick up the black color, and paint the bottom part of the layer mask as shown in the screen capture.
6. The resulting image will look like what I added at the end. This should pull her more forward but if you get carried away to darken the background further it will look like you are peeking through a keyhole unless that is the effect you want.

Addendum:

By the way, if the only thing you want to do is to lower the value of the whites behind her, do all the steps except in the Levels adjustment do not move the center triangle below the histogram to the right. Just move the output levels triangle on the right towards the left. Remember, this could lead to some unnatural looking highlights, blown or not.

Answer 2

So two basic things to get out of the way first:

1. the color computations can be thought of as being done on each single channel in turn.
2. the numbers used are not {0-255} but rather {0.0-1.0). So rgb(128,128,128) is rgb(.5, .5, .5) (or 128/256)

ACEkin is correct: overlay is a screen & multiply.

If layer B is <.5, multiply otherwise screen. (using your layer names; B is bottom layer)

A straight multiply is B*A=NewPx. Overlay doubles the result so that middle grey is preserved.

Multiply part:

2(B x A) = NewPx
for B=.49, A=.49, NewPx = .48 (grey on grey is nearly unchanged)
for B=.4, A=.2, NewPx = .32 (dark grey on medium grey result in darker)
for B=.01, A=.01, NewPx = .00 (black on black results in black)

Multiply makes things darker, since multiplication on any two fractional parts will always result in a smaller value, and in terms of color definitions that means "darker."

The Screen portion is doing a multiply on the distance between the color and white, which should darken it, but the result is subtracted from white which (I think) reduces the impact (?)

1-2(1-B)(1-A) = NewPx
for B=.5, A=.5, NewPx = .5
for B=.6, A=.8, NewPx = .68 (light grey on medium grey results in lighter)
for B=.99, A=.99, NewPx = .99 (rounded, but slightly closer to white)