I want to erase part of a rectangle but it gets erased in the way shown in the right hand image:
I want to be like this:
I'm guessing you're using a stroke layer style on your shape in Photoshop.
If you use a Layer Mask directly on the layer that has the shape with the effect, it will create the result you see on your top image. It won't delete the stroke but create a new "path" for it to follow instead, the stroke will simply wrap that shape and still be visible.
But if you put that layer into a "layer group" and add that layer mask to that group instead, you'll get the result on your second image. It will erase/mask the stroke layer effect as well or any other layer style you used on the layers within that group. This can be useful for drop shadows too.