I have this antique style map:
I wish to customise it by adding more land.
Is it possible to create a custom pattern that represents the land? How would I add the "edge" of the land; perhaps as a stroke/outline on the brush preset?
You could copy and paste a rectangular piece of the land from a larger piece of the land mass, create a path over it, and apply that as a vector mask, then add a stroke effect.
For example
Yes it is also possible to create a custom pattern in Photoshop, but I don't think that's how it was added to the map. Patterns repeat, but the example doesn't show that. It's more likely a texture that has been created possibly using the clouds filter, or even a scan of some paper. Getting something that matches exactly would be difficult.
You have already got an usable answer, so this is only a minor supplement.
You wanted complex edges. This is an easy way to get them without drawing the finest details. I have two layers. One is for fake paper and the other is for manual painting.
The painting layer is filled with black and two coarse white shapes are painted. The one in the left is a few strokes with an ordinary brush and the other is a filled selection which is drawn with the polygonal lasso tool:
To get complex edges 3 filterings has been applied
A selection is made with the Magic wand. To avoid small islands one must check "contiguous". The tolerance affects radically:
Use that selection to cut a piece of the fake paper. Paste it place to generate a new layer. Here it has got Layer > Layer style > Stroke to generate the black edge:
As you see, there's small ponds near the coastline. To avoid them you can insert small areas to the selection (=hold shift and draw an elliptical selection over the hole). To get them more you can paint black inside the white shapes before blurring.
Less torn result can be got by smoothing the selection with Refine Edge: