3

I would want to enlarge a mask be it either a pixel based or vector one by adding a "bleeding" effect so to increase the masks's area. Adding actual "Outer glow" FX doesn't change the masking area, but only adds FX itself if put on the adjustment layer and if put on a group - cancels the adjustments, that are inside the group. Transforming the mask is not option since I may have several masking zones on a layer and the size increase has just a different result, not identical to an "Outer Glow". Selecting the mask and then modifying the selection takes too much time and has no live preview.

I'm coming from Affinity Photo, where the "Outer Glow" can actually be used on a layer used as a mask (Affinity's masks can be their own independent layers). So I wanted to find an identical way of doing so in Photoshop.

Adding feathering(via Gaussian blur) and increasing the area(via Outer Glow) in Affinity Photo of a mask layer.Adding feathering(via Gaussian blur) and increasing the area(via Outer Glow) in Affinity Photo to a mask layer

Improve this question
3
  • You cant actually do this in photoshop without a extra plugin. I mean plugins can manipulate alpha but the vanilla setup is really short on those. Most probably the reason for this oversight is that it would cannibalize on sales of after effects.
    – joojaa
    Jan 18, 2021 at 7:46
  • @joojaa Thank you, there are answers in this thread, explaining a workaround without plugins. But do you know such a plugin you mention about?
    – Gražvydas Žuklys
    Jan 19, 2021 at 22:18
  • Yes but its not feather.
    – joojaa
    Jan 20, 2021 at 5:05

3 Answers 3

Reset to default
4

An option could be to use a smart object with Maximum or Minimum filter as a shallow Knockout layer in a group. This way the filter expands/contracts the transparent area which drives transparency of the group it's in. And since it's on a smart object, it's non-destructive. Add a gaussian blur on top for blurry mask.

enter image description here

Update: How this setup works.

In the Blending Options dialog the smart object in the group has Fill Opacity set to 0 and Knockout set to Shallow meaning that the effect will be limited to the group. Opaque pixels of the smart object layer will "stop" the pixels of the layers beneath, transparent pixels will let the layers beneath bleed. Maximum filter is then applied the smart object and it influences transparent pixels of the SO shrinking/expanding the masking effect.

enter image description here

Improve this answer
4
  • Thank you. I've come across a workaround using Maximum/Minimum filters on the actual mask itself. Though, your way seems more edit friendly. Unfortunately, I can't understand your layer system. There seems to be a pixel layer, filled with black and a cutout (via erase), which was turned into a smart object and then used as a mask (independent layer inside a group, actually). It has 0% fill and has blending changed. I can't get the mask to work, the adjustment is global.
    – Gražvydas Žuklys
    Jan 17, 2021 at 20:36
  • 1
    @KostasMargiavičius I updated my answer, I hope it's more clear now
    – Sergey Kritskiy
    Jan 18, 2021 at 9:30
  • It is clear now, thank you. Your answer is the most accurate answer to my question, though, keeping in mind photoshop's way of doing it in some cases it is probably more practical to use the destructive way.
    – Gražvydas Žuklys
    Jan 19, 2021 at 22:06
  • 1
    I'd probably go with destructive on this but I really like knockout layers, they are underdogs of blending :D
    – Sergey Kritskiy
    Jan 19, 2021 at 22:19
4

In Photoshop all mods of the layer mask are destructive, but it's possible to make easily a spare and restore it if the modification went bust. The next method works with pixel layer masks.

So, here we have only solid green in the image layer and the layer mask makes 3 separate areas visible:

enter image description here

The mask occurs in the Layers panel, it's in the Masks panel and it's also a channel in the Channels panel.

You can make a spare by duplicating the mask channel:

enter image description here

The spare is only a dummy channel but you can restore the original layer mask with it if needed. It's a good idea to rename it for ex. SPARE Mask

Layer mask can be "Refined". You can feather (=blur) and move the mask edge to expand it. Contrast affects the BW contrast of the mask:

enter image description here

You can access the refinement dialog in the Masks panel or in the Layers panel by right clicking the mask icon.

You see the result as you adjust. Checkerboard is replaced by white during the edit.

Here's the effect of the mask edit:

enter image description here

If the original mask must be restored, simply Undo. If undo is no more available, delete the wrong layer mask. Right click it in the Layers panel and select option Delete.

Ctrl+Click the spare channel icon in the Channels panel to create a selection. Then insert a new layer mask. It's based on the selection.

Nothing prevents you to save also a copy of the edited mask as a dummy channel and restore it if needed later.

Other maybe useful things to know:

  • you can have an empty layer with a layer mask
  • you can move a layer mask to another layer in the layers panel
  • painting manually to layer mask is possible, only select the mask icon in the layers panel.You'll see the effect, not the mask.
  • you can get a layer mask to onscreen for edits and cut&paste by clicking the mask icon in the layers panel and holding Alt key at the same time, no need to go via the channels panel.
Improve this answer
1
  • Thank you. I some cases it is actually easier to just use the technique you write about, the destructive one and by saving masks in the channel panel. Although, if you're also interested in a strictly non-desctructive method, there is an answer in this thread.
    – Gražvydas Žuklys
    Jan 19, 2021 at 22:11
2

You can not apply layer styles to masks.

As far as I know there is no way to alter a layer mask in Photoshop (raster or vector) in any non-destructive manner. Masks won't accept styles or smart filters or anything which would allow non-destructive editing of the mask itself.

To simulate an outer glow on the mask you could....

Create a separate layer containing the same base shape as the Mask.

Set the Fill Opacity to 0%.

Then apply an Inner Glow style to this new layer.
(Inner because it's no longer the reverse as a mask would be)

enter image description here

Improve this answer
4
  • Sorry, maybe my title isn't precise enough. I don't want the actual FX. I need to enlarge the mask non-desctructively. I've mentioned "Outer Glow" because it's a trick from Affinity Photo to add bleeding to the mask (not feathering), which doesn't work in Photoshop.
    – Gražvydas Žuklys
    Jan 17, 2021 at 2:27
  • 1
    @KostasMargiavičius As far as I know there is no way to alter a layer mask in Photoshop (raster or vector) in any non-destructive manner. Masks won't accept styles or smart filters or anything which would allow non-destructive editing of the mask itself. You may need to examine other construction methods, such as clipping masks perhaps.
    – Scott
    Jan 17, 2021 at 2:54
  • Thank you. You may check the other answers if you're interested in this topic. One answer covers using actual masks by saving them and modifying them destructively and one answer actually mentions a non-destructive technique, although its non that smooth as in Affinity, but good to know you can do it in Photoshop.
    – Gražvydas Žuklys
    Jan 19, 2021 at 22:14
  • Well, it wasn't a leap to save the current mask and make a new one. But I thought the goal was to not do that.
    – Scott
    Jan 20, 2021 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.