A simple way to achieve a rough/torn edge look is to use an actual torn piece of paper as your mask. You can probably find a usable picture easily enough. Alternatively, you can tear a sheet of white paper in half, lay it on a dark or black surface and take a picture yourself.
Open the picture and tweak it as necessary using levels or brushes to make sure the paper is mostly white and the background is mostly black (or a non-white color). Use the magic wand selector on the paper. Tweak tolerance as necessary, but it doesn't need to be perfect - you're going for a rough look anyway - and use that selection as a mask.
To get a more subtly uneven look like the website you're referring to you'll probably want to shrink the mask in the direction perpendicular to the tear.
Here's a result of this method with the mask shrunk to 10% in one direction, and then applied to all 4 edges of a black box:
Keep in mind that you can do something like this with any edge you can find. A piece of paper may be the most obvious choice, but anything from landscape horizons to tolerance cut-offs from natural gradients in the color of the sky will work.