The headline is actual question:) I want to add vector mask to the layer, but I want to soften it's edges because it looks a bit unnatural compared to the rest of the photo, thanks
when you select the vector mask just open the properties panel—if it's not open already—then you can feather it and adjust opacity.
feathering a raster mask is fine until you decide to resize it ;)
I think the closest you can get would be a setup like this:
Basically create your shape, apply a layer mask to the shape creating the feather how you'd like using whatever tool you want. I did airbrush but could just as easily select your shape, feather it, then make layer mask. Then use it as a clipping mask since it sounds like that's what you're after.
Here it is using the actual shape and feather:
You'd still have to reapply the Layer Mask each time you adjust the shape but that's not really too difficult and could make an Action to do it if you're using the Feather command instead of the Airbrush.