The magic wand tool only really works well where there is a definite edge. Polar1ty's select/contract/erase answer is helpful as a quick fix if the purple edge is even. However, I believe your best bet is to learn how to use the pen and mask tools. The advantage of using the mask tool is that you can non-destructively hide sections of the image (unlike the eraser tool) which makes it far easier to fine-tune after. I'm going to show you how to do this with pen tools but there are many ways to make a good mask layer.
On the Paths window (Window > Paths) click the "Create New Path" button and ensure the path is selected. Now grab the pen tool (P) and trace around the image, clicking along the edge of the pixels that you want to keep till you have a full loop that encircles them all. You can click and drag to form bezier curves to go around curved objects. You can also hold Ctrl, while using the pen tool, to edit previously placed points easily. There are better guides, a quick search away, for using the pen tool if you find yourself getting stuck. The advantage of using paths instead of a selection tool is that you can edit the path, rather than start from scratch if you mess up.
I know my example doesn't have a purple edge but it's all the same, you want to place your path so that it doesn't include any of those purple pixels.
Once you've made your path, load it as a selection, either with the button in the Paths window, or by Ctrl clicking the path in the Paths window. Now with the selection highlighted, select the layer you want to mask to be applied to and add a layer mask with the button at the base of the layers window (Window > Layers).
This should leave you with a layer similar to above. If your image isn't very sharp, you may want to select the layer mask and in the Properties window (Window > Properties), drag the Feather slider up a bit to soften the edges. If you still have a bit of a purple edge to remove then you can use the brush tool with black to paint onto the layer mask to hide more. You can now add layers below.
Your results may vary and it will take a little bit of practice and experimentation to get used to using the path and mask tools. It really is worth the effort; these tools are some of the most useful tools that Photoshop has (or any other image editing software).
You can also combine Polar1ty's answer with this method. As you have a cut-out area already, you can Ctrl click the layer and contract the selection (Select > Modify > Contract) and then make a layer mask which you can neaten up after.
"Leaf blower man, oh leaf blower man... Please bugger off (away from my office window)" - marcusdoesstuff