How to apply anti-aliasing to a layer with a transparent background?

Question

I have one layer with some black pixels in the middle and a transparent background.

If I add an opaque white background under it, merge both, and then click "Antialias" from the "Filters" menu, it correctly generates gray pixels where appropriate. However, I intend to use this image on top of a dynamic background, so I can't use anti-alias against a specific color, but if I don't add the background opaque layer and just use the Antialias filter, nothing happens. My expectation was that semi-transparent pixels would be generated, so that it would become anti-aliased with different backgrounds.

Is there a way to achieve this or am I trying to do something wrong?

Answer 1

I still haven't tested this, but I think I figured out a workaround. In troubleshooting another issue, I came to know the option "Color to Alpha", which seems exactly what's needed here. I think I can get Antialiasing like usual against one specific color and then use Color to Alpha to turn that color into different degrees of transparency. This would work fine on simple layers with very few colors, and it would probably be problematic in more busy patterns, but it fits my own needs at least.

Answer 2

I've been getting around this by box selecting around the item I want to antialias with a 5 pixel border. Then I fill the transparent area with a color that is different from the item I am modifying, but set the fill opacity to 1. Keep the current box select and do the antialias filter. Then use the fuzzy select tool (with threshold set to 0) and select the fill area, which should select everything up to the new antialias part. Then delete your selection.

For me, I had solid color backgrounds. So I would color pick the background color, then adjust the saturation or value by 1 in the color selector, then fill the transparent area with that color. This forces the antialias to use a transparent color of almost the same value when it does the calculations, so it's like you are antialiasing with full transparency (almost).

I noticed this works well with black or white (opacity 1) as the fill color, but it does slightly adjust the antialias to be lighter or darker than my item color. Not a big deal, but the effect is cleaner if you use a fill color closer to the border color of your item.

Similarly, if you use a red fill color around a non-red color (blue/green/etc) the antialias will use the red color when it applies the antialias. Which is usually undesirable.

Answer 3

If you really want to avoid even slight changes to the colors, you can use the 'Color to alpha' method from Smig's answer on two copies of the image, using different color channels for each, then recombine them into one image.

Precise steps:

1. Make two copies of the image.
2. Remove blue from the first copy (use colors->levels and set the output of the blue channel to 0).
3. Remove red and green from the second copy (use colors->levels and set the output of the red and green channels to 0).
4. Using blue (0, 0, 255), fill the first copy's background, apply the anti-alias and use color-to-alpha to remove the background.
5. Using yellow (255, 255, 0), fill the second copy's background, apply the anti-alias and use color-to-alpha to remove the background.
6. Put the resulting layers into the same image.
7. Set the mode of the top layer to "lighten only".
8. Merge the two layers.