It can also be done with a layer mask over the vector shape.
Create your polygon, then load the layers selection by Command-Clicking (Ctrl-click on pc) on the vector mask.
Then go to Select>Modify>Smooth. Enter the smooth amount.
After that it's just a matter of creating a layer mask based on the now rounded selection.
Now you have your rounded polygon fairly quickly and easily. You could also manually edit the shape with by adding points and using the convert point tool, but that would involve a great deal more work.