When exporting / saving images with a transparent background that are intended for websites, should I save it as PNG or SVG? Why?
The simple answer here is use both.
The fact that you've named SVG as an option, means we can rule out photo graphics as an intended use case - because SVGs are only good for line-art graphics such as logos, icons and clip-art-like illustrations.
If you are considering this choice for photo graphics, there is no choice; PNG will probably always be better. For graphics where SVG is a viable option, SVG is the best option with a PNG/JPEG fallback. PNG has many strengths, but on scalability and file size, it will often not match up to SVG.
Using only one or the other means you will have to sacrifice either backwards compatibility or progressive enhancement.
Looking at what we know the future holds; forever higher resolutions, wider support and wider use of SVGs across the internet; it makes sense to build for what's coming.
In general, websites should be built to last for many years; in 5 years time your beautifully backwards compatible website might look amazing for the 2% of internet users still using old browsers, yet fairly poor on the up to date browsers with crazy resolutions - so it's very much a stack of hard choices on which way to compromise.
Your options in November 2014
For the sake of quality you'll need to serve at least five different versions depending on screen sizes and resolutions - and that's a very conservative guesstimate, you could end up with 10 - 15 versions of the same image if you wanted to be extremely thorough. This also takes some time to implement.
If you chose to serve a single graphic and have the browser scale it, the results will probably be less than perfect, and could even be ugly depending on the amount of scaling.
A large number of media queries could unnecessarily bloat the CSS and negatively impact on page loading speeds.
Will look great on older browsers and devices, but not so great on newer ones.
SVGs with single PNG/JPEG/GIF fallback
You could use SVGs everywhere and then have it fallback to another format for browsers that don't support SVG. The main advantage is you only need one file for all the different resolutions.
If you compromise and accept that users on outdated browsers can live with imperfectly scaled graphics, you would only need one other version of each file in PNG, JPEG or GIF format.
This would take a similar amount of time to implement as the PNG only media queries - possibly even less, meaning it would probably be around the same price.
Will look great on all new devices, with sacrifices being made on older technology.
SVG with multiple PNG/JPEG/GIF fallbacks dependant on resolution and screen size
You could serve SVG first, and then resolution-dependant PNGs for browsers that don't support SVG. This would be the most thorough, most backwards and forwards compatible, most consistent and most
It would probably take as much time as 1 & 2 combined, plus a little extra for working out the kinks.
Will look amazing on almost every device.
In summary, I think it depends on whether you're looking for more backward compatibility or more progressive enhancement, and how much
money time you have to spend.
SVG is scalable, if you have a vector-graphic that is a clear advantage. For pixel-graphics PNG is better. A downside is, that the Internet Explorer supports SVG only with the coming version 9 (before with plugin). Mobile browsers may also have limited support for SVG.
EDIT: As ClemDesm points out, older IE-versions don't support fully transparent PNG, since IE8 that is supported. Non-transparent PNGs work fine. The answer of Computerish has a great solution for handling vector images for now: Keep them as SVG, but export them for web as PNG. I fully agree with this solution.