How to get rid of annoying jagged line of t-shirt mockup

Question

As I was trying to make a t-shirt mockup for my own project, I ran into this problem. First off, I made a selection of the t-shirt and put it in a seperate layer, making the model to have an empty png in the middle.

Then I used alpha channel to extract the shadow of the shirt and put it on top to make it look realistic.

To change the colour of the shirt I created an empty layer called "change colour" inbetween layer 2 and 3. I made a layer mask for the "layer 2" and put that layer mask to "change colour" layer and paint it with any colour (in this case it's light pink)

Now with everything done, here is my result

So I'm pretty much happy with how the shirt looks like but if you look closely, it has an annoying jagged line and edges around which makes it looks very unnatural and unrealistic. I've tried everything suggested online from using the pen tool to make a perfect selection to trying to smoothen the edges but none of them seemed to work. What do you guys think may be the problem here and what should I do to improve it? It would be highly appreciated if anyone could help me with this one out using the same technique that I did.

Here's the original image if anyone would like to demonstrate

Thanks a ton!

Answer 1

You have too many layers with cut outs and masks. Stacking up multiple cut out edges that butt-up against each other will cause edge conflation artefacts.

Try this instead:

1. Mask out the t-shirt layer only. This will mean you only have to deal with making and perfecting a single layer mask. You may need to make the layer mask manually using the pen tool, or use the select and mask functionality and edge refinement tools, and then paint on the layer mask to manually edit it (which is how I made the example below). Note that the mask is inverted, to cut away the t-shirt, to leave a hole.

2. Put the shadow layer under that - but use the alpha trick to extract the shadows of the entire image. Don't try to cut the t-shirt out of this layer. Those areas outside the t-shirt will be hidden by the layer above anyway.

Edit requested by OP on how to make this shadow layer, in more detail:

• Select all and copy the image using Ctrl+A then Ctrl+C
• In the Channels panel add a new Alpha Channel, and paste the image into the Alpha Channel using Ctrl+V
• Ctrl+click on the Alpha Channel thumbnail to load it as a selection
• Select the RGB channel (this will deactivate the Alpha Channel)
• Back in the layers panel, create a new transparent layer under the image, and do Shift+F5 (or Edit > Fill), and choose black as the fill colour
• Do Ctrl+D to deselect all

3. Finally add a fill to a layer at the bottom.

Example

Click on image above to view at 100% zoom.

Answer 2

You have destroyed anti-aliasing. In low res images anti-aliasing is essential for apparently non-jaggy images. If you have non-antialiased sharp border in a high resolution image the jaggines cannot be seen in low res displays. The border gets blurred without asking, the jagginess appears only when zoomed in.

In the original image there's at the edge of the shirt a gradient mix between the environment and the white shirt. You have changed a great part of that mix to pink. There should be a mix between pink and the original environment.

If you do not have a high resolution image for the job try properly anti-aliased selection to make the pink area. The anti-aliasing occurs automatically right if you can draw a path selection just in the middle of the anti-alias zone. Feathering a magic wand selection doesn't make it right (tested). With magic wand there's always selected too much or too little.

But you can use the quick selection tool. It's very clever item. You can use it in + or - mode in turns and it seems to learn what you want. Using it in the blue channel gives a good result fastest due the best contrast as you have already noticed. The selection is automatically well anti-aliased.

So, select the shirt in the blue channel with small size quick selection tool and create a new layer. Fill the selection in the new layer with quite dark color and let the layer have blending mode Hard Light. With the color you see clearly the faults in the selection:

There's sharp cusps which aren't selected, so the filled selection doesn't give color to them. You should walk around the selection before any filling and try to fix the selection by using the quick selection tool with small, even 1 px brush size in turns in + and - mode (+Shift, -Alt). I guess it becomes good except sharp cusps.

A piece in the edge of the neck opening is intentionally left bad. There's selected too much. You see something like a bruise in the middle of the edge of the neck opening.

Paint with a small soft brush color to the areas which were left out of the selection. Erase color in areas which seem to be unwanted extras in the selection.

After few strokes with a paintbrush and eraser the new colorization layer is perfect (or as good as resolution this low allows)

I guess you want to have a solid masked colorization layer for the fill color because that can have also patterns. In addition you want to insert an adjustment layer for ex. to increase the contrast of the details in the shirt:

You can use the manually fixed blue layer to make a selection which can be used to create perfect layer masks. Simply Ctrl+Click the layer icon of the previous filled layer in the layers panel. Then create layer masks for the adjustment layer and the colorization layer. I filled a layer with pink for colorization and let that layer have blending mode multiply. The curves layer is used to lift up all wrinkles of the shirt. It must be used carefully because it lifts also up the jagginess.