When I click on the quick selection tool and try to select an object I get lines and it won't go around to select the object. Instead, I get a line. What should I check to ensure the quick selection tool is enabled properly. In the attached image I am trying to use the quick selection tool to select the rock hill top and you can see I just get the lines. I am using photoshop 2020. Thank you.
The quick select has no parameters other than brush size.
I think the trouble is there's not much of an edge for it to work with.
I had a quick go using your image & tbh it was easier to select it all & remove the 'sea' than start by adding 'rocks'. The end result wasn't magnificent & I'd be more inclined to use the lasso.
It's probably because there isn't much contrast between rocks and sea/water/surface. Both are darker at same time. Further, it would still might work if you had a high resolution image. But the one you've shared, has very low resolution, so quick selection doesn't work as intended.
You can try hit and trial and might end up getting an okay result, but I'd recommend you should use pen tool instead, in such cases. I always use pen tool for almost all selections if it's a professional project.
Quick selection works pretty well and sometimes can work even better than pen tool (For example selecting a square or pentagon with enough contrast and sharp edges).
But for this particular image, pen tool seems a better idea. You can use a temporary Levels adjustment layer for more visibility by adjusting brightness/contrast.
You can adjust mask options like feather.