In this spesific case there is a easy way to make a perfect selection. Go to channels copy the channel, and load selection.
Now even with this perfect selection there will be a fringe. The trick is to copy what you want to stay and then defringe this, or in this case the remove black matte option. Thats how you get rid of the residue. In general workflow its much better to think in terms of what you want to keep, instead of growing selection and geting slightly unperfect results. But thenagain this is far more common problem in motion graphics so i wouldnt expect this to be widely known outside those circles.
Again in this specific case there is a shortcut after selection paint it over and paste masked layers.
This is what you do. Go the the channels box copy one color channel. Select the copied channel. Magic wand select the outer white area, grow selection. The color areas act as a buffer you can safely grow into. fill with black.
Duplicate this channel, magic wand select one white area. Grow again and fill with black (if your fill wast good enough, undo and grow more then fill). Go to the copied channel. repeat for the other area.
Back to color mode. load selection of one area and make a fill layer. repeat for the other. Fill the background residue with white or put a pure white layer behind or leave it blank.