How is this masking effect achieved in photoshop?
How to make it in Photoshop without using any flashy tricks:
Start from a photo of a white shirt. Remove the background using clipping path or other method which leave perfect solid edge. Magic wand and any color based selecting methods are usually useless for this purpose when the target is a photo:
The shirt without a background is one layer named Original.
Add a new empty layer "White". Make a selection in layer Original. With magic wand select the emptiness around the shirt and paint into layer "White" solid white color full of the selection. Use a big solid brush. Do not use the paint bucket.
Insert a new empty layer "Black". Invert the selection (reselect the emptiness, if needed, do not try to select the shirt) and paint solid black full of the selection into layer "Black":
Goto layer Original, select all and copy the layer content to the clipboard.
Goto layer Black and insert a layer mask.
Goto the layer mask (=click the mask icon holding Alt at the same time). Paste in place to the mask the original shirt from the clipboard (= Ctrl+Shift+V).
Invert the layer mask (Image > Adjustments > Invert). Now you should have onscreen:
That's the layer mask. If you clck the layer content icon and close layer Original, you see the wanted result:
Now you can remove layer Original, merge layers Black and White and export the image as PNG.
If you put it over any image with blending mode = Normal, the image is shown in the hole and shaded:
NOTE: It's useful to save the multilayer image version, too. You can apply curves to the layer mask of layer Black to find the proper shading strength.
The next step to higher realism would be to use the original shirt image as a displacement map for the inserted "Any image". That would make the wrinkles more plausible. But that's beyond the scope of this answer.
You can see exactly how it's done by looking at it. The "White Area" outside the shirt is white (100% opacity), the shirt area is transparent (to show the design underneath) and there are some shadows to overlay to give it the feeling of being a material.
To apply the mask on the web, they have layered the PNG over your artwork using SVG. There is almost definitely another step in their particular process as the mask they have in that png alone doesn't seem to shadow correctly if it is just applied as a mask. However it certainly can work with a solid background colour as your outside mask and basic shading (and highlights) within the masked area.
I've attached an example.