As far as I know, you can't. The method suggested by Joonas and elaborated by Rachuru is a good option for many cases. When that isn't suitable, here's another option you can try.
Not the ideal solution you're asking for, but an option that can help in situations where you have up to 60 layers in Illustrator or maybe 40 in Photoshop - any more and it'd still be the tedious slow scroll:
Move your layers panel to the top of the list by click-dragging it upwards.
Then use the panel resize utility as you can see in the screenshot, and drag that panel height as far down as it can go.
On my screen I (quickly - could be wrong) count 32 layers that I can drag up and down immediately.