How do I apply anti-aliasing to the edges of a layer?

Question

How do I apply anti-aliasing to the edges of a layer?

I have a layer that has rough edges. I want to apply anti-aliasing to it so it blends well against the background. What is the best way to do that in CS5?

Answer 1

Blurring will not give you a true anti-aliasing effect, and in some cases the blur will look even worse than the aliasing. If you want to do a really good job, keep reading…

Part 1: Create an Anti-Aliased Selection

Use one of the following methods. The first is hardest but gives you more control and can produce a much better result.

1. Pen Tool – Select the Pen Tool and begin creating a path around the perimeter of the aliased shape. I will not explain how to use the Pen Tool. Check out the second option if you don't know how. Otherwise, draw the path, then go to the Paths panel, right-click on the path you created and choose ‘Make Selection...’ In the pop-up that appears, make sure that feathering is set to 0 and the Anti-Aliased checkbox is checked. Then click OK. A quicker way to make this selection if you are sure anti-aliasing is checked, is to Ctrl-Left-Click the path in the Paths panel.

2. Magnetic Lasso Tool – Switch to the Magnetic Lasso Tool by holding the Lasso Tool button until all variants of the tool appear, and then select the Magnetic Lasso Tool. In the options bar, make sure Anti-alias is turned on and set the Radius, Contrast, and Frequency to 30px, 1%, and 100 respectively (play with different values if those don’t work for you). This tool works in a quirky way, you have to click the tool near the edge of the shape you are tracing, but don’t hold the mouse button down. Just move the mouse to trace the shape with the cursor as best you can, and then click on the beginning anchor-point to close the selection loop.

3. Refine the Edge - This is the least accurate method, but can still produce a semi-decent anti-aliased selection quickly. Go to the Layers panel and Ctrl-Left-Click (Command-Left Click on Mac OS X) on the icon of the layer to be selected. This will create an aliased selection around your shape. Then choose Select > Refine Edge... via the main menu. In the Refine Edge dialog box, set Contrast to 40% and Smooth to 50 and set everything else to 0. You can also play with the Smooth and Contrast settings if those ones don't work for you.

Part 2: Contract, Invert, and Trim

Now that you have an anti-aliased selection in the same shape as the path, go to Select > Modify > Contract… via the main menu and enter 1 px in the dialog that comes up. Press OK in the dialog, and then click Ctrl+Shift+I on Windows or Cmd+Shift+I on Mac to invert the selection. Now press the Delete button on your keyboard to trim the aliased edge.

This should produce a true, anti-aliased outline of your object. It will be one pixel thinner than before, but in many cases that will be just fine.

Answer 2

If the image or layer has been clipped already, you can use a magic wand to select the empty area around the clipped item, invert the selection, contract the selection by 2 or 3 pixels, create a clipping mask, deselect, click on the clipping mask so the mask itself is highlighted in the layers palette, apply a slight blur to the mask.

Answer 3

I had an irregular border a few pixels wide of very dark color in between a bright foreground image and a light background gradient, due to the foreground image having been lifted from a dark background. The image was 4000x3000. I expanded the image to about 200% and color-selected the dark pixels. (I used the eyedropper to take the color of the dark pixels and then used it in the select-by-color option.) I made sure I had not inadvertently selected any pixels in the foreground image by performing a subtractive deselection on the area of the foreground. I then replaced the dark pixels with the local background color. Where the edge was rough, I used the smudge tool to blend the foreground and background a little. (At normal viewing size, this smudging is not visible.) This is unfortunately time-consuming and might not fit everyone's problem.