Something similar can be achieved entirely within Photoshop.
Create a rounded rectangle and add a stroke effect, and choose a pattern overlay. Although this will only work for small stroke sizes of around 2px.
You could also use an outer glow effect, and apply some noise to get a thicker irregular line. You could even combine both stroke and outer glow effects.
Screenshot showing an outer glow effect