How to make a Effect act as transparency in Photoshop?

Question

I know I can create a layer mask, but this method only accepts a bitmapped mask as far as i know, than the result is not perfect. I tried using this method: https://www.youtube.com/watch?v=xGBNHY5rAxg

What I'm trying to achieve is:

Layer 1 - Regular bitmapped (Raster type) image (it will be even better if it could be a vector masked layer, but it's ok if it can't be).

Layer 2 - A vector masked layer (with fill 0%) with a stroke or outer glow, whatever. The effect of this layer must fade out (make transparent) the pixels of Layer 2, BUT WITHOUT THE NEED OF "RASTER TYPE" IT.

Idea example:

Then, if I change the color of background...

What I'm really trying to do is a PNG Icon, that I made with Shapes (vector masked layers) of a envelope, where the lines are made with strokes. So, the stroke may turn the layer below it transparent. I don't wanna use the bitmapped mask because I lose the vector benefit (I can't change the size without doing the whole process again) and I lose quality (because the final result is not the same. The transparency created by the bitmapped mask is not perfectly equivalent of the image used).

Maybe this give some idea:

As you can see, the icon is really tiny, and I thought pixel by pixel, then there is why I care so much about the perfect equivalence of transparency (in this icon I used white stroke to make the envelope lines. I wanna the white become transparent when I export as .png, but as I said, don't wanna use a bitmapped mask (create a layer mask, raster type my image, and then use it as a mask of the image itself) because I care too much about quality).

Thank you in advance! (for read so far hahaa)

Answer 1

I ran into this exact same issue a while ago, also while drawing a bunch of small icons. Turns out you can do some pretty neat stuff with the "Blend if… This Layer" slider in the Layer Style panel:

Dragging the right slider all the way to the left basically tells Photoshop, "keep everything I draw black opaque, and make everything white transparent".

The important part is holding the Alt/Opt key when you click the slider to split it up into two values. This creates a smooth transition for all the in-between grays.

So you put your icon's shape layers into a group, apply these blending options to the group, and then wrap it in another group to add any color overlays, drop shadows, etc that you need.

I posted a few examples and a more detailed how-to on Dribbble.

Here's how it looks for your test file (PSD download):

Answer 2

Being Photoshop, there’s probably quite a few approaches to this problem, but only one I can think of that maintains full vector edited and scaling.

There’s a few things going on here.

1. Square is just a shape layer for the square. Nothing tricky there.

2. The Blurry Circle group has a circle as a vector mask. The vector mask is set to subtract and also has mask feathering of 6px.

3. Circle is just a shape layer for the filled portion of the circle.

4. The Icon group is just there to neaten things up.

A color overlay layer style can be applied to the Icon group to color the entire icon.

Changes to the background colour will be shown through the icon.

A word of warning: This entire document is constructed using vector shapes (yay!). It also uses mask feathering (yay!). Mask feathering doesn’t scale when the document (boo!). However, there are scripts that can scale mask feathering (yay!).

If you’d like to take a look at the PSD I’ve created, it’s available here: https://www.dropbox.com/s/4stmws0piigh1g8/mask-feathering.psd

Answer 3

First, I will answer the question about the envelope, as I think this is what you really want to know.

Your logic is a little bit off here.

If you want to retain transparency, ie. not use white lines for the envelope, you should instead create a rectangle, then on the same layer change the shape tool to 'line', then change the Path Operations to Subtract, and subtract the 2 lines from your rectangle.

You can add multiple shapes to the same layer by holding down Shift as you click and drag to draw your shape - adding to the shape, but in your case as you want to subtract from the shape, so you hold down Alt.

EDIT - another technique to achieve the same thing:

I think you are using an old version of Photoshop, so do this: Group the black rectangle of your envelope and add the 2 (or 4) lines to a new layer. Then drag this vector mask onto the group. This will achieve the opposite of what you want, so in the group mask select all the lines, and change Path Operations to Subtract. You could actually separate all the shapes into new layers this way - by putting each shape onto the mask of a nested group.

Now for the vector 'glow' transparency.

When your circle shape layer is selected, open the Properties palette and try messing with Feather and Density. This is how you can blur the edges of a shape, however I'm not sure if it's possible to keep a hard line on the inside of the circle and blur the outside like a shadow/glow.

Hope this helps.