I don't think you're entirely wrong.
If you're designing on a retina display, you know that what you're seeing as you design is exactly what is going to be seen by other retina users - but you're going to need to test the images on a lower res display to know for certain what your average user is going to see. The reverse is the case if you're designing on a low res display.
However, I think that your bullet point is nonsense. Yes, you shouldn't design your images on an iPhone, but lets assume you're using retina a desktop ;)
For one, if you open up a 120x120 image on your retina display, the OS will scale it up to appear to be the same size (if you measured with a ruler) as on a half resolution display (it may differ slightly depending on scaling settings - but Mac OS X don't even give you the option of viewing the image at actual size and doubles the size as standard). So you're not suddenly working on an image a quarter of the size.
Secondly, even if you were, it wouldn't matter - you'd just zoom in. Setting "pixel size" = zooming in and out. Photoshop is even kind enough to show you the outline of each pixel if you zoom in far enough (400% or so).
Just noticed that you said "pixels are an abstract unit of measurement". Again thats right and wrong (before the advent of the retina display, I'd probably have just said wrong).
In terms of your screen, a pixel is absolutely a physical thing (and hence in no way abstract), if you have a 1024x768 monitor, there are 768 rows of 1024 physical squares, each with a red, green and blue component.
So when you save a 100x100 image, that means when you open it up its going to occupy 100x100 physical pixels on someone's screen. If you have a non-retina iPhone, the image appears smaller purely because the physical pixels of the screen are smaller (and if you really had a 1024x768 16" display, guaranteed a non-retina iPhone would have smaller pixels), if you have a retina iPhone then this is still the reason why, but iOS is also making the image bigger because 100x100 would be so tiny otherwise.