I am trying to export a clean mask of an image I created in photoshop to indesign. Whenever I utilize the work path, the wp does not adhere to the selections I've made, but rather creates jagged nasty looking edges instead. Is there a way to change the work path specifications (tolerance set at 0.5), or some other way to transfer the masked out of photoshop?
Not really... at least not automatically.
The "create work path" is a best guess feature. It tries to be as accurate to a selection as possible, but often it's not entirely precise.
In most instances, you can just grab the Direct Selection Tool (white arrow) or Pen Tool and refine the "work path" manually. Or, just create the path with the Pen Tool to begin with and not rely on the guesswork of the auto-path generator.
Why a path at all???
InDesign will honor Photoshop transparency. All you really need to do is have a transparent background in Photoshop, save, then place the .psd into InDesign. The background will be as transparent as it is in Photoshop. You can also set things like Text Wraps in InDesign to use the alpha transparency of a placed .psd files.