Since your question is quite broad, I'll answer broadly.
Grid-based design is all about rhythm and harmony. Rhythm and harmony both derive from proportion. In graphic design, architecture, music, cinematography or any art form you care to think of, these dominate.
Classical Greek architecture is my favorite visual example. -- If you dig into the design of, say, the Parthenon, one of the most beautiful buildings ever created, you find that the dimensions and spacing of EVERYTHING has an exact mathematical relationship to everything else in the structure, all derived (as far as one can tell) from the length of the baseline, which is itself related to the height of the hill it stands on.
In music, rhythm keeps the band and the audience in a basic agreement about what happens when. You can break the rhythm for effect, but if it's uneven or erratic you'll see nobody dancing and pretty soon nobody listening, either. Ask any band.
An letter-sized page takes its proportions from an octagon (actually from half of an octagon and the circle that encloses it):
You can take any of the intersects in this (diagonals between points, horizontals between points, the "margin" at the top, etc.) and work them into your grid. You can divide the page by even fractions (1:1, 1:5) or use more complex, but still natural, proportions (1:1.618 -- the Golden Section, 1:1.414 -- sq root of 2, the Fibonacci sequence) to divide your page arithmetically (same distance between grid lines) or exponentially (same multiple of a basic distance). Any of these provide visual rhythm -- things appear where the eye expects them to appear. The connection between the designer and the audience is maintained by rhythm, just as it is in music.
This applies to your white space as much as it does to your main design elements or text block. Harmony derives from proportion and rhythm.
BUT... keep in mind that this kind of design, strictly adhered to, can be so harmonious, so "complete" that it can be much too peaceful and at rest unless you do something to add motion or tension to it. (Think of a beautifully centered wedding invitation, perfectly spaced -- lovely, but utterly motionless. Or a perfect circle in the exact center of the page.)
This tells you when it's appropriate to break the grid: when you want something to stand out, to smack the viewer in the face, you give it a tilt, make it disproportionately large, make it tiny compared to the grid element that it sits in. Contrast by size, position or color; things that are off-balance or out of kilter; anything deliberately and clearly out of place will at least make its presence felt. A single twenty-something with a bright pink mohawk sitting in a row of concert-goers in tuxedos has impact because he's out of place (but, in this case, still "on the grid").
Grids and their use are a whole study unto themselves, but these are some of the basic principles behind them. I hope that's useful for you.