Your description looks very similar to this reported Inkscape bug:
"I noticed also that after navigating in the image through the scroll bars or wheel, the reactivity of Inkscape drops down drastically, selecting a single object (even very simple) with F1 can take several seconds, this when the whole image is a bit “complicated” with several thousands of points. Only zooming in (or out) and back brings the things in a normal state of reactivity again."
The bug doesn't appear to have been fixed yet, although you might want to try older versions of Inkscape and/or the latest development builds to see if it might be version-dependent. Some comments on the bug report also suggest ungrouping objects as a (partial?) work-around.
If you do find a workaround for the bug, or have any other additional information to provide (such as a drawing that reproducibly exhibits the problem), you may want to add it to the bug report in order to help the developers pinpoint the cause and fix it.