I've tried a number of different settings in the GUI, and have tried manual path tools from the path panel, but no matter what the inset path function completely mutilates the path into a blob. Distance of the inset or offset doesn't make a difference, and these are very simple paths like a square, or a circle. I'm on Mac, OS 10.6.
That sounds quite bizarre. I can't reproduce this, but I'm running on a Win7 system and my Mac isn't handy. Try, in sequence, the "usual suspects": quitting and relaunching Fireworks, logging out and back in, restarting the OS, deleting the Fireworks preferences file (it's Fireworks CS5 Preferences and it's in Library/Preferences in your user folder), resetting permissions, and, if all else fails, uninstalling and reinstalling the application.
I'm not on a Mac, but I did try to re-create the blob that Inset Paths generates. Unfortunately, I didn't get the blob with simple paths, but I did learn that rounded corners do not always play nice with concave vector nodes.
Just to be clear, here's what I did:
(1) add a Star Autoshape to the canvas - 1 px black stroke, no fill, centered path (2) ungroup the autoshape to make it a regular path (3) Open the Path panel and select the Inset / Expand Path option in the Alter Path section (4) Entered these settings:
inside, 10, miter, 10
(move the path inside the original by 10 px, mitered corner, miter limit = 10)
(5) The resulting path was a smaller version of the original star.
If the corners are "round" or "bevel" (instead of miter), then some of the vector nodes will be modified. "Round" appears to convert concave nodes into Bezier curves. "Bevel" appears to convert concave nodes into flattened angles rather than points.
I also found that I needed to use CTRL+K to "snap to pixel" to get rid of the non-integer pixel coordinate blurriness, but this is a common effect of numeric transforms or scaling.
I did not extensively test different shapes and settings, but when I used "miter" as the corner setting and kept the miter limit low, the results were pretty accurate.
Hope this helps,