One way to blur just the edge of something is to remove its background, duplicate it and blur the lower layer. That way you retain a sharp top layer and jus get the blurred edges. It may require some additional cleanup from the top layer, but with something like a map, maybe not. In most cases, this is pretty simple to do and also not incredibly helpful. Especially in this case, which is pretty extreme.
In this case, since the goal is to make the edge quality better, I would try the automatic image trace in Illustrator and use its edge. I mean... looking at the map I wonder if you could just use the vector image trace intraced by AI and call it a day... but assuming you have to keep most of the raster map as it is, you could try this:
- Trace the map leaving out the white background.
- Use the high fidelity photo preset for the image trace.
- Then back in PS, place that below the raster layer.
- Remove the background fromPlace the raster layer and place it in a clipping mask with the vector layer (Alt+click between the layers).