You're probably looking for a "glyphic" sans-serif, one with visible stroke contrast and with swellings at the terminals.
I hugely recommend Heliotrope by Matthew Butterick. It's well-spaced and proportioned for body text, and has a fair price and a common-sense license.
Canela from Commercial Type is more expensive, but the main styles come free with a Mac if you use one. It comes with a lot of optical sizes, weights and features. Ayer from the same designer, Miguel Reyes, is a more expressive heading face. I'm starting to see this everywhere.
You might also be interested in a humanist sans-serif (serif-style proportions, but a more monoline structure without serifs) like Ideal Sans.
Or in Berthold Wolpe's Pegasus typeface, which is definitely a text face but has a kind of chiselled, metallic feel.
The most famous design along these lines is Albertus, also by Wolpe, but it's a display typeface. It wouldn't work in a textbook, the contrast is far too high and the color wrong, but might be interesting as a companion for headings. It feels a bit arty for a maths textbook, though. There's a freeware digitization of the main weights.
Goudy Sans by Frederic Goudy is a typeface that looks great if you take care to use the right weight. The Bold and Black Italic weights are really cheerful and full of warmth. You'd have to have a heart of stone to hate them.
The regular weight, I don't think so much of. The design concept seems stuck in the Art Nouveau era with that condensed "a" and tilted "e". But when you can license weights one by one it's not a problem.