How to use opaque Smart Object as transparency mask?

Question

I have two opaque Smart Objects. Object A is black-and-white and I want to use it as a transparency mask for object B.

In other words, for the following layers:

IMAGE REMOVED - Imgur replaced the original image with a different one (hosted at the same URL!). I have removed it to not confuse the readers.

How can I get a Red X on a transparent background without rasterizing object A? Please note, that layer B is a smart object and can be fancy (i.e. not just red). I want to be able to edit both of these smart objects and see the effects instantly.

Answer 1

This is not possible.

Smart objects can only be used as clipping masks (Option/Alt-clicking between layers). Smart objects can not be used as any other form of mask. Masks must consist of pixels or paths and nothing more complex than that.

However.. if some raster content is okay.....

Assumptions:

• The X layer Smart Object can not be edited in any way.
• raster masks are okay

• Image is RGB

  1. Move the red layer above the X.
  2. Set blend mode of red layer to "Lighten"
  3. Highlight the X layer
  4. Layer > Adjustment Layer > Invert..
  5. Highlight the X layer and the Adjustment layer
  6. Right/control-click on highlighted layers > Convert to Smart Object
  7. Option/alt click between the two layers you now have.
  8. Highlight the X layer
  9. Command/Ctrl-click the Blue Channel
  10. Layer > Layer Mask > Hide Selection

Final image:

Zoomed to show edges:

Steps 3 through 6 could be replaced by simply editing the X smart object so it's inverted.

Conversely, you could simply highlight the X layer, Command/Ctrl-click the Blue Channel. Inverse the selection and apply a mask to the red layer. Then turn off the X Smart Object visibility. This provides a much easier file to work with.

I'm not clear as to why you need to use the smart object to mask anything. If you need a red X why not simply create a red x?

Answer 2

As Scott stated, this seems to be impossible (or, was, in CS5). The fastest way I know (which involves rasterization though) is:

• Select Layer A (the one with an X).
• Ctrl+A, Ctrl+C, Q, Ctrl+V, Q (copy the contents of Layer A, switch to quick mask mode, paste, switch back).
• Apply current selection as a mask to Layer B and hide Layer A.

These steps have to be repeated every time Layer A is changed. Compared to Scott's answer, this will work with any kind of layers (not just plain red).