You could use the Wrinkle tool. It's located in the Tool bar as shown below. Double click the Tool icon to change settings as required. These are the settings I chose.
Then use the tool as you would normally use the Brush tool, over the edges of a rectangle or shape, until it's rough enough.
The other answer given covers the addition of stippling/shading. So I won't repeat that. However just to say that using the Wrinkle tool is a vector effect, rather than a raster filter.
Not sure about the exact effect, but you can use Illustrator's
Effect > Pixelate > Crystallize to create a crystallized look for borders.
For those dotted 'gradients', you can either buy or create a stipple shading brush or you can find some free to use brushes, like these:
(Above image credit: The same blog (spoongraphics) which I mentioned above)
That will give you similar look.
- Create your two rectangles with some different colored borders.
- Apply effect to both of them. Then go to
Object > Expand Appearance.
- Now you need to remove the unwanted border. For this use
Window > Image Tracefeature and then Expand it. Ungroup everything and delete unwanted shapes. You'll be left with this:
- Then use those Stipple brushes. Select darker stroke colors. And use Draw Inside mode.