I have a file with a layer that has a smart object. I want to copy the whole layer and relink the smart object to another file. When I copy the whole group as a new layer, I somehow can't unlink the layer/
You can right click on the smart object layer and select "new smart object via copy" and this will give you a new separate independent smart object layer, which you can then copy over to the other file. Not sure if this solves you're issue.