A good textbook is about content. Without something worthwhile to say, the design fades away. Content is King. So let us look at what kind of content you will need.
Most subject, need or at least greatly benefit from pictures to clarify things. The right picture at the right place can do wonders for your design. This said you can not just use any old pictures, you must go out of your way to make the picture that fits in the overall feel of your textbook.
Images need to be uniform in style, this speaks quality. Once the style is set you should not deviate much form the overall look and feel. This is unfortunately possibly expensive. As this would mean each image has to be carefully selected, or better yet individually illustrated for your project. So it might mean drawing everything from scratch.
Images need to be used consistently. If every chapter begins with an image then you are imprinting the readers a visual mnemonic. This can greatly enhance learning. The wrong image can also destroy your message, see this TED talk about the solar system for a concrete example.
So its important not too choose a image on coolness and style only. It also needs to be correct from a learning point of view. So preferably your illustrator should have a good grasp of the subject matter. For history books maybe it means pictures of people and places in the pages of the text.
2. White Space
Don't cram everything on one page, spreading the text may enhance learning as again now students have a page mnemonic. So work on fitting your material onto the pages well. You should also limit the number of concepts per page.
Its easy to think that students would read your book entirely, but they dont. They will look for external clues on what is important or fun to learn. Walls of text may not be the most inviting or cost effective way of presenting things. Off course if the book is just text then they dont have choice but then you need a narrative to capture the mind.
People have very limited memory, in fact humans have contextual memory. We tend to forget things once we leave the context, see "Walking through doorways causes forgetting: Further explorations". So you need to fit your explanations on the page or spread so that students can minimize paging between data. This means using the white space cleverly to keep the mind focused.
This also means you can not put data, in separate places. For example: Images need to be where they make most sense not in a full color leaflet in middle of the book. Its cheaper but detracts form good design.
3. Don't Use Too Many Gimmicks
One book is a one trick horse. Even if you can do millions of different designs does not mean you need those designs in one book. Sure this is a bit boring from a designer perspective. At the end of the day you want the book to be known for its one thing it does well, instead of the millions it does badly.
Keep it simple, dont use too many different typography schemes, pick two fonts make few styles for headers design your type boxes and artwork. But dont over style or decorate your stuff. Don't use millions of colors, use a few, and use them to maximum effect. In the end the book is meant to be read, not apprised for artwork. Concentrate on the structure of your book.
4. Spend Time on Getting it Right
Don't just dump the book out. Spend time on designing whats in the book, and how its presented. Its a lot of work, in fact half of the work is getting the feel and idea right.
Design up front, you need to have a overall idea what the book will look like when you make the content. Otherwise you wont hit the space requirements. Sometimes it means leaving out things you have slaved over. Don't fall in love with the content, be pragmatic. If it doesn't really have to be there, it doesn't have to be there. Even if its your most beloved creation of all time; put it where it belongs.
Remember the audience, you might need the academic boost. But your students might not need to be seen as academically credible at all times. So think about who your impressing first. If you think the book is for professors then some amount of spartan design might be good for you. But if its for undergraduates they might have different aesthetic sense than your older colleagues.
5. E-Book may Need a Separate Design
Because PDFs flow badly in electronic devices. Its easy to choose same design but in reality that does not make the design great. So the electronic version may need a totally different approach. The medium has different possibilities after all.
I should draw some illustrations for this post. But I dont really have time, so see my design already suffers. Can not be helped.