The requirement is to scale a design from a 100ppi, 4inch display up to a 400ppi, 4 inch display. The design should scale such that a 40px X 40px button dimension has the same physical size across any pixel density display. Does this mean we should design using physical dimensions such as inch/cm rather than pixels?
Pixels for web or screen based design.
Inches for print design in US.
Mm for print design in UK and Europe.
It's important to steer clear of using inches, mm or cm for screen based design. 10cm on a 1900 x 1280 dpi screen is NOT 10cm on a 1024 x 768 screen.
If your goal is truly to have "the same physical size" across "all possible displays", then YES, you will need to specify that.
HOWEVER, you will need to be extremely explicit and verbose about this, since it is a very difficult goal. At the same time you will need to ensure you NEVER use anything other than inch/cm dimensions in all specifications, and force the designers to calculate the pixel dimensions for all possible display densities (or use 100% raster-free art).
The reason that screen specs are always given in pixels is because they are logical units independent of the vagaries of display manufacturing and technology. iOS has a scheme for providing artwork in multiple sizes so that the lower density displays have something and the higher density ones have something which can be scaled down (which usually offers better quality than going upwards).
If you have a specific subset of displays (such as iphone only), you have a better chance of achieving this, but even then, not all Retina displays are 2x pixel density. AFAIK, there are 3 variations minimum and probably more, and probably will change in the future.
I can think of at least one tile-based game with frame-animated sprites (composited animations) which had a zoom function. Every tile and sprite asset had a zoom version: twice the assets (and +4x the total file size) for all game pieces and tile sets.
TLDNR; [density] / [pixels] = [physical size]. If all you have control over is pixels, you have no control over the physical size. If you want to fix the physical size, you will need to have a variable pixel dimension and that is not possible without scaling a pre-rendered raster image.
Perhaps the best thing to do is identify the sizing specs for your target range of devices and then copy that for your own spec sheet.