Pen tool - Precise placement of anchors and control over curves
Pencil Tool - Freeform creation of disjointed paths.
Brush Tool - Freeform creation of disjointed paths with automatic application of brush appearance
Blob Brush Tool - Freeform creation of a brush path suing a calligraphic brush which will auto-expand when the cursor is lifted, leaving a ...
You have already got an usable answer, so this is only a minor supplement.
You wanted complex edges. This is an easy way to get them without drawing the finest details. I have two layers. One is for fake paper and the other is for manual painting.
The painting layer is filled with black and two coarse white shapes are painted. The one in the left is a ...
You could copy and paste a rectangular piece of the land from a larger piece of the land mass, create a path over it, and apply that as a vector mask, then add a stroke effect.
Yes it is also possible to create a custom pattern in Photoshop, but I don't think that's how it was added to the map. Patterns repeat, but the example doesn't show that....