You can use the Direct Selection Tool (white arrow) and merely toggle Outline/Preview Mode.
In Outline Mode you can click drag around anchors without having objects selected because you've clicked a fill.
Command/Ctrl+y to toggle the modes (View > Preview/Outline)
You don't have to switch back and forth between preview and outline. You can simply switch ...