What are best practices when increasing/decreasing tracking in a paragraph to make it look better, whether the paragraph is justified or not? How do you decide when it's overdone?
Not certain there are definitive values one can use here. Each typeface is different.
For example, I like ITC Stone Informal quite a bit for body copy. However, I will often apply a -5 tracking value when using it so it's slightly tighter than the default tracking.
On the other hand a font such as Ropa Soft Pro almost always requires either double spaces between words or it needs a value of +100 for kerning on spaces, but no tracking adjustments, when you use any weight above "medium". The word spacing is just too small in the font. The creator just made glyphs bolder, but failed to adjust word spacing for bolder glyphs. So, the heavier the face, the closer words are. (Ropa is only feasible for display/headlines to me.)
I dislike justifying paragraphs and avoid it in favor of ragged right, no hyphenation, whenever possible. I'll allow hyphenation in some circumstances. If I must justify text, I massage, coerce, or otherwise play with it to avoid large spaces, too much hyphenation, or condensed sentences as much as possible. But again justified type, to me, is a last resort, and really only when the client is specifically requesting it.
If there's one thing I constantly do when designing a piece with body copy, it's increase leading beyond the typical 120% of most applications. I prefer 140%-150% for leading. That's about the only adjustment I can state I make with every piece I do. On the other hand, some display typefaces work best with 60% leading rather than 120%. So, it's still kind of relative.
On the whole, I will almost never increase tracking unless it's extreme and intentional for design.
Beyond Metis's excellent answer, I can only add: you'll know it when you see it.
Tracking adjustments are meant to help your eye flow along the paragraph without stumbling over individual words, letters, or gaps. Some fonts are designed with wide kerning (spaces between letters in the font), so tightening by -5 snugs everything together. Some need a little air to breathe because the letters are practically in one another's pockets.
It's overdone when you glance over a block of text and immediately know something is wrong. If it's a painful dense black blob, or a wispy gray cloud, start back at zero and try again. There's no right answer because every font and every piece you design are different. What needs tracking at 10 pt. may be perfectly legible without it at 18. It's an aesthetic judgment call.
Sometime ago, working for a multinational publishing company for medicine and science books, the layout people had a very strict rules regarding text. The objective was not so much in terms of aesthetics or design, but rather the content which should fit in the number of pages stipulated per chapter and not exceed one line.
- Optimal leading= 120% (font size + 20%), can be more, never less
- Optimal column length= 1.5 alphabet = 27 x 1.5 = ≈40 CPL (characters per line)
- Maximum tracking allowed= +25%
- Minimum tracking allowed= -25%
- Avoid (or forbidden) fonts horizontal or vertical distortions, instead use the family options as condensed or expanded.
And strict means strict: I have seen how the company canceled an external designer's contract because he reduced the line spacing of a paragraph to less than 120% to fit a diagram on the same page.
There was also another big set of strict rules, but referring to syntax.