(BTW - nuns also made illuminated manuscripts...)
Very interesting question. I have studied old manuscripts for years, and there are a few things to keep in mind; either as explanations or as interesting anomalies.
I think a general history of illuminated manuscripts might also be interesting, but that would be a different question.
Vellum and parchment was/is crazy expensive. The process of preparing it takes ages, and a high level of craftsmanship is required (yes I have attempted to create it, with miserable results). So often you can find amazing frontispieces in these documents, but the further in you go, the "simpler" the designs get. Or at least the initials and the text are very space-economical. Large blocks of text, hardly any spaces, often no punctuations, short or non-existent ascenders or descenders.
- They had rulers, curves and compasses
- They did preliminary sketches on other surfaces. (Coal, chalk, wax
- The main composition where sketched out in thin hairlines with a silver solutions, minute scratches or tiny pin-pricks in the material. As there was not really any option to erase, this had to be pretty precise and/or got covered by the ink.
- They had some freedom in "filling in" from the main structure of the design.
- Mistakes could be covered either with darker colours or with leaf
- Working on vellum and parchment, the illuminator and the scribe might
well design around individual faults in the material! So the
material to some extent dictates the design.
- As time progresses: late middle ages, paper comes along and lines gets steadier, thinner, airier, more symmetrical and more elegant. This also a product of the engineering of tools getting more precise.
- There are geometry in the designs. Sometimes hidden, but they
balance the compositions.
- Some manuscripts illuminations are "off kilter" - but bear in mind
that due to chemical reactions, the colours we see in illuminated
manuscripts today, are not necessarily the ones the monk/nun/artist painted. Vellum yellows, colours fade.
- It was usually not the same person who did the text as the one who did
the illumination. Those were specialised job description: scribe vs illuminator :)
These days, we tend to look at these things with awe and wonder (with good reason), and may not observe minute details. To me, there is life in slight wobblyness in any work, and these documents certainly have that. For anyone that has tried to write with a loose-nib pen, there is an inherent wobblyness and a variation of thickness in the strokes. Not to mention writing on vellum! This adds to the vibrancy of the work (they of course used goose feathers. Same rule applies). There is a reason why many cartoons have small changes in the stroke with.
...and there were tricks of the trade. An interesting thing is what the Irish monks often did, and your example of the Book of Kells demonstrates this: they used red dots along the stems of letters. This does two things: it hides some overly wobbly line and softens the hard edge; and it brings a vibrancy to the page. It makes it "pop" and in a real sense actually vibrate. Pointillists later used this technique to good effect. Lindisfarne gospel, gospel of Matthew:
As for compositions; there was room for creativity (as long as it was all done in a frame of mind to "the glory of god"). On closer inspection, they use upper- and lowercase as they see fit. Particularly the Irish manuscripts - both in initials and the text itself - have a liberal blend seemingly willy-nilly.
There is also the fact that at least the earliest manuscripts were not intended to be read by any but the higher religious classes.
A practical guide to create your own illuminated letters, with a smattering of history: The art of illuminated letters, by Timothy Noad.
Dawned on me that I did not touch on a part of your question; about the design process, apart from that bit on rulers and silver solutions. The trouble of touching that, is that it is so entwined with other arts, architecture, literature, fashion and discoveries (both continents, sciences, nature). Often, these things were made to celebrate the "glory of god", but just as often, and maybe more, to celebrate the wealth of the person who commissioned the work. A good deal of these works - particularly various Books of Hours and psalms - were mainly a tool to flaunt your wealth under a thin veil of religiosity.