I created a loader animation icon using the shape tool (ellipse). But when I export my animation, the circle has jagged edges and the quality isn't good.
What are good settings to prevent these edges and have a high-quality
.gif as a result?
There's really no such thing as a high-quality gif. The format was developed in the early 1990s and is limited to 256 colours, one of which may be fully transparent.
Best you can do is optimize it for the current background colour, and choose fills & blends that do not involve a lot of intermediate shades.
If you want a basic spinner / throbber icon and your target is modern browsers you can get much better results with css animation and/or svg.
Gifs only support 1bit transparency (meaning that a pixel is either fully transparent or not at all), so depending your case you might use dithering option (old-fashioned) or match the matte color with your background color. You can find a former and complete answer in this thread: How can I remove the white pixels around the edges when exporting a transparent GIF?
Or, you can explore new and modern possibilities with SVG icons and CSS3 animation properties to obtain clean and scalable results. Take a look to this site to explore this option: http://codyhouse.co/gem/animate-svg-icons-with-css-and-snap/
Where are you using this GIF? On a website?
If that's the case I'd recommend using CSS3 animation as your primary loader, as these will provide the smoothest results. Here are some examples.Note that these are not images but instead are shapes rendered and transformed by the browser.
An animated file back-up isn't a bad idea though as older browsers (such as IE9 and below) don't support CSS3 animations.
Unfortunately as mentioned, GIFs aren't really all that great, so your fallback will be worse quality than native CSS3 solutions.