In order to set up a cross-reference, InDesign has to ensure that it has the target page correctly identified. To do that, it has to know exactly what the page count of the book is. If the intervening chapter files are closed, InDesign can't be sure their page counts haven't changed, so it has to open them in sequence, check their page counts and update them if necessary, then close them. All that disk overhead adds up fast if you're not working from a solid state drive.
Keeping all documents and the book file open while working on long documents is a best practice, especially when you start adding cross-references or other internal hyperlinks. It allows InDesign to work at full speed without unnecessary disk i/o. It also allows the program to track changes in page count and page numbering so they update along with any cross-references and internal links in real time, makes synching styles much faster, and is an easier and more reliable way to work.
If your system is too short of memory to work efficiently with multiple documents open, add some RAM. It will pay for itself many times over in saved time, not to mention helping your hard drives last longer.