Ok, several issues here, let me address one by one.
I scan my illustrations to jpeg
Do not use JPG for saving the scans for commercial work. Use a lossless format from start. TIF would be a good option. JPG can be a good format for someone that really knows how to handle it, but in this case, we need to reduce the problems on every part of the workflow.
The proof always looks great but the printed book illustrations are awful.
What are you calling a proof?
If the Proof is a calibrated plotter print from a professional commercial printer and does not match the final product in a reasonable fashion... The problem is the provider.
A proof is NOT your homemade print, nor a print on the big printer on the office or a printer store.
It is a print calibrated to a specific color profile that will be used on a specific type of paper with a specific printing process like an offset lithographic print. It can also be a single direct print of a digital printer, before printing all the rest of the series.
I have tried calibrating my screen
Have you succeed or just tried? A monitor is either calibrated or not. It can be calibrated to some reasonable extent and that can be ok... to some extent, but I do not understand the idea of "trying to calibrate".
The ideal way to do it is using a device specifically made to do so, X-Rite Color Munky or DataColor Spider are good options.
and calling the printer
That is the first thing to do... Before ruining all the printed books.
Are you using the right paper, mainly a good quality coated? Did you see previous prints of the same provider? Did you discuss the expected quality? **Did you compare a real proof to the same finished product?
** Did you asked for the color profile they are using for the specific paper they are using?
as well as InDesign
...?? Did you call Adobe? They solve Installation Issues and stuff, not how to properly use the software.
Can anyone help with this issue? Specific settings would be helpful!
Yes. Someone that is involved in the project. Someone who knows the answers of the questions I wrote.
Stop trying to do all by yourself and ask for PROFESIONAL help, that is if you really value your project.
ID converts RGB to CMYK thus causing the problem
Ouch. No. This is the problem:
ID will convert the files from RGB to CMYK if you tell it to do it on the export.
There are even more questions here.
Do you need to convert it? Are you embedding the color profiles? Is the color profile right? The right resolution? Do you have effects, filters or transparencies?
There is a chance you do not even need to convert the color mode at all. Depending on the project you probably should send the files as RGB. Again, you need someone professional involved on your project.
Related
links there. They may be helpful.