Take the 2-minute tour ×
Graphic Design Stack Exchange is a question and answer site for Graphic Design professionals, students, and enthusiasts. It's 100% free, no registration required.

How do I apply anti-aliasing to the edges of a layer?

I have a layer that has rough edges. I want to apply anti-aliasing to it so it blends well against the background. What is the best way to do that in CS5?

share|improve this question
    
What kind of image are we talking about that's inside the layer? –  Joonas Nov 11 '11 at 6:33
    
It's an irregularly shaped photo of a real-life object. Rather, the layer has been trimmed to the edges of the object but it doesn't blend against the background and looks horrible. –  Nick Strupat Nov 11 '11 at 16:19

4 Answers 4

up vote 8 down vote accepted

Blurring will not give you a true anti-aliasing effect, and in some cases the blur will look even worse than the aliasing. If you want to do a really good job, keep reading…

Part 1: Create an Anti-Aliased Selection

Use one of the following methods. The first is hardest but gives you more control and can produce a much better result.

  1. Pen Tool – Select the Pen Tool and begin creating a path around the perimeter of the aliased shape. I will not explain how to use the Pen Tool. Check out the second option if you don't know how. Otherwise, draw the path, then go to the Paths panel, right-click on the path you created and choose ‘Make Selection...’ In the pop-up that appears, make sure that feathering is set to 0 and the Anti-Aliased checkbox is checked. Then click OK. A quicker way to make this selection if you are sure anti-aliasing is checked, is to Ctrl-Left-Click the path in the Paths panel.

  2. Magnetic Lasso Tool – Switch to the Magnetic Lasso Tool by holding the Lasso Tool button until all variants of the tool appear, and then select the Magnetic Lasso Tool. In the options bar, make sure Anti-alias is turned on and set the Radius, Contrast, and Frequency to 30px, 1%, and 100 respectively (play with different values if those don’t work for you). This tool works in a quirky way, you have to click the tool near the edge of the shape you are tracing, but don’t hold the mouse button down. Just move the mouse to trace the shape with the cursor as best you can, and then click on the beginning anchor-point to close the selection loop.

  3. Refine the Edge - This is the least accurate method, but can still produce a semi-decent anti-aliased selection quickly. Go to the Layers panel and Ctrl-Left-Click (Command-Left Click on Mac OS X) on the icon of the layer to be selected. This will create an aliased selection around your shape. Then choose Select > Refine Edge... via the main menu. In the Refine Edge dialog box, set Contrast to 40% and Smooth to 50 and set everything else to 0. You can also play with the Smooth and Contrast settings if those ones don't work for you.

Part 2: Contract, Invert, and Trim

Now that you have an anti-aliased selection in the same shape as the path, go to Select > Modify > Contract… via the main menu and enter 1 px in the dialog that comes up. Press OK in the dialog, and then click Ctrl+Shift+I on Windows or Cmd+Shift+I on Mac to invert the selection. Now press the Delete button on your keyboard to trim the aliased edge.

This should produce a true, anti-aliased outline of your object. It will be one pixel thinner than before, but in many cases that will be just fine.

share|improve this answer
    
Thank you! Here's a tip I discovered that should save you time in the future. To select all the non-transparent pixels of a layer, hold down CTRL (Windows) or Command (Mac OS X) while clicking the thumbnail of the layer in the Layers window. –  Nick Strupat Nov 13 '11 at 0:24
    
Actually... with that selection I can't select to antialias checkbox. Any ideas? Or do I have to select it manually using the pen tool or magnetic lasso tool? –  Nick Strupat Nov 13 '11 at 0:26
    
I do not think there is such an option. The Ctrl-Click selection method you are describing mimics precisely the alpha channel of the layer you are selecting. If the alpha channel is not anti-aliased (and in your case it's not) then the selection will not be anti-aliased either. I am afraid you will have to use either Pen Tool or Magnetic Lasso Tool to get an anti-aliased selection. –  Alexei Nov 13 '11 at 0:44
    
Well, you got me thinking, and I discovered a third method that might be a bit quicker. Let me know how it goes :) –  Alexei Nov 13 '11 at 1:08

If the image or layer has been clipped already, you can use a magic wand to select the empty area around the clipped item, invert the selection, contract the selection by 2 or 3 pixels, create a clipping mask, deselect, click on the clipping mask so the mask itself is highlighted in the layers palette, apply a slight blur to the mask.

share|improve this answer

I had an irregular border a few pixels wide of very dark color in between a bright foreground image and a light background gradient, due to the foreground image having been lifted from a dark background. The image was 4000x3000. I expanded the image to about 200% and color-selected the dark pixels. (I used the eyedropper to take the color of the dark pixels and then used it in the select-by-color option.) I made sure I had not inadvertently selected any pixels in the foreground image by performing a subtractive deselection on the area of the foreground. I then replaced the dark pixels with the local background color. Where the edge was rough, I used the smudge tool to blend the foreground and background a little. (At normal viewing size, this smudging is not visible.) This is unfortunately time-consuming and might not fit everyone's problem.

share|improve this answer

I do this pretty much the same as Alexei answered but I use the pen tool to make my anti-aliased selection then I do all editing inside the selection ensuring you don't draw over the selection boundaries.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.