This question already has an answer here:
For lack of a better name, I'm trying to create a decreasing stippling effect like this:
My first inclination was that this could be created in Photoshop with some sort of filter effect. I was thinking along the lines of creating a smart object, applying a filter effect, then using a gradient on the filter mask to decrease the effect. This would allow for the possibility of a curved edged like you see in the image. No dice though, there doesn't seem to be an effect that quite does this.
The problem with searching for this effect is that I don't know what it would even be called. I've seen it implemented all types of posters though, so it seems to be generally used.