Using a Smart Object as a mask for another layer?

Question

Let's say I create a complex smart object containing layers, filters, effects, shapelayers, etc.

Then, in the main comp I want to use that smart object to mask another layer/smart object with.

However, I do not want to see the masking smart object itself.

Is there a configuration, such as blend mode, or similar, that lets me do this, without just adding a layer mask and copying over the visible parts?

Basically, I want to let the smart object (the mask) be fully editable, and make it automatically apply as a mask to another layer/smart object, with no manual steps other than editing the smart object.

Note, I am specifically not asking how to make a "clipping mask", by Alt-clicking on the separator line between two layers. I know about this function, but if the layer above (the one being clipped) is transparent, parts of the layer below (the one masking) will be visible. That's what I want to avoid.

Basically, to reproduce an example comp, create two layers by using the rectangular marquee tool, and creating two solid color layers, of different colors:

+==============+
I              I
I              I
I      +------------+
I      |            |
I      |            |
I      |            |
+======|            |
       |            |
       +------------+

Then I want to mask the topmost layer so that only that layer is visible, but only the parts masked by the bottom layer, without the bottom layer being visible, like this:

       +-------+
       |       I
       |       I
       |       I
       +=======+

With a straight clipping mask, I would still see the underlying layer.

Answer 1

What you're looking for is a clipping mask. That's done by placing your complex SO layer under the one you want to mask. From there you have three choices:

• Target the layer you want to mask and choose Create Clipping Mask from the Layers Panel flyout menu.

• Use the keyboard shortcut Ctl+Alt+G (Cmd+Opt+G on Mac).

• Hold down Alt/Option, hover on the line between the two layers until you see the cursor change and click once.

To remove the non-overlapping parts of the clipping layer, open the Blending Options dialog and uncheck Blend Clipped Layers as Group, then drag the white end of the top Blend If slider to the left until the bits you don't want disappear. You may also need to turn the fill down to 0.

Answer 2

DO use a clipping mask > Double-click on the layer that should be invisible > untick the second tickbox (that says something like 'blend clipping mask as group' (I'm manually translating this from my own laguage because I have no English version installed)).

EDIT: I forgot to mention that after these steps, you might need to set the Fill transparency to 0%.