You may or may not have a problem.
The user is able to preload the editor with a fixed canvas size..
either its 500 * 500 pixels or 300* 300 pixels ... my user draws in
pixels but i tell him that this is millimiters. The user has no idea
it is pixels
If you make the user aware that the canvas area on the screen represents a printable area of 80 mm square or 133 mm square (using your existing 3.77 px per mm = 96 px per inch (ppi) conversion factor) then that should be OK.
(I have to say, I think you may have started by looking at the wrong end of the problem. Are your print sizes actually intended to be 80 mm square or 133 mm square?)
One way of doing this by showing rulers with mm units around the canvas, as Horatio suggests. Although you don't necessarily need to be explicit, e.g. if your application was for printing on CDs and your canvas showed the outline of a CD it would be obvious.
But if you are relying on the canvas on the screen being true to print size, i.e. actually measuring 80 mm square or 133 mm square on the user's screen, you have a problem, as this will only be be true for monitors which display at 96 ppi, and where there is no additional scaling in the browser or OS.
Desktop and laptop screens commonly have a a ppi value between about 80 and 130, and much higher for some mobile devices, so there will be a great deal of difference.