As others have noted, this is almost certainly a pixel alignment issue.
Here's a simple example: a 29 × 29 pixel rendering of a circle and a square, first with pixel-aligned edges and second with the whole image shifted 0.4 pixels right and 0.2 pixels up:

To show more clearly what's happening, here are the same images scaled up by a factor of four:

Here, even the shifted version doesn't really look so bad, because the shapes are so simple that they look OK even with blurred edges. But if you have a lot of small details in your image, bad pixel alignment can easily turn them all into so much mush.
Fonts solve this problem with hinting, but unfortunately there's no way that I know of to hint an SVG image. (Well, I guess you could do it with scripting, in principle... let's say, no easy way that I know of.) Thus, you'll have to resort to other solutions. Here are a few tips I've found useful.
This is sort of a no-brainer, but if you're making your drawing with a particular output size in mind, make that the page size from the beginning. That makes setting up grids and such much easier, and also lets you easily preview the drawing at the target size by pressing 1 in Inkscape.
Create pixel grids for your document. In Inkscape, you can do this in the Grids tab under File → Document Properties.... Having a grid that shows where the pixel boundaries are makes it much easier to align the edges properly, especially with careful use of the snap to grid feature.
Note that you can have multiple grids in a single drawing, and toggle their visibility individually. Thus, you can create a separate grid for each of your target sizes (or at least for the ones where you expect problems). For this, it's most convenient to set your nominal page size to some number of pixels that is divisible by all of your target sizes: that way, all the grid spacings will be integer numbers of pixels.
If you just can't get your drawing to look good at all sizes, you may need to create multiple versions of it. For this, you can either create multiple copies of the entire document, or, if it's only part of the drawing that's causing problems, use Inkscape's layers feature. Just put the problematic parts of the image into a separate layer (or layers) and duplicate it, then tweak the different copies to work at the different target sizes. Then, when exporting a given target size, make the layers for that size visible and hide the rest. Don't forget to give the layers suitable names so that you'll remember which is which!
Unfortunately, as far as I know, Inkscape doesn't have any simple way to scale a page and its contents simultaneously, like GIMP's Scale Image tool. Of course, it's pretty easy if your drawing exactly fills the page (just resize the content and then auto-resize the page to match), but things get trickier if you want to keep some margin around the edges.
One thing you can do is create a "document rectangle": a simple white or transparent rectangle that exactly covers the page. (Set it to fill only, no stroke, and then adjust the dimensions to match the page.) It may be most convenient to put this rectangle in its own layer, since that way it stays out of the way, and you can easily select it even if it's fully transparent by going to that layer and using Edit → Select All (Ctrl+A).
Then, when you want to change the size of the drawing, you can select all the objects in it (using Edit → Select All in All Layers or Ctrl+Alt+A), scale them to the desired size using the transform tool controls bar (or Object → Transform...), and then resize the page to match by using Resize page to content from the Page Properties dialog.