Something similar can be achieved entirely within Photoshop.
Create a rounded rectangle and add a stroke effect, and choose a pattern overlayfill. Although this will only work for small stroke sizes of around 2px.
You could also use an outer glow effect, and apply some noise to get a thicker irregular line. You could even combine both stroke and outer glow effects.
Screenshot showing an outer glow effect
Here's another where I also added an inner glow with noise to roughen up the inside line, in combination with the other two effects. The beauty of this method is that the effects are live and move when you rescale or change the size of the shape.