I have done some 3D renders and I know the frustration.
The theory behind it
Let us make a zoom on this circle and see what is happening. This is what you want. A red shape transparent on the borders. (If you put this image "A" on diferent background it will blend properly the rest are totally opaque.)
If we see the real image and the mask separately we will notice something. The transparent red pixels are actually red and we have a mask where we expect it to be.
But when we have a pre rendered image, with anti alias on, thoose pixels are no more red, but diferent grades of pink.
Most of the masking techniques rely on reducing the mask, so the transparency is "eating" more where you have not the modified color, so you do not see this halo.
A method of using your exact image as a mask
1) Open your image
2) Go to the channels panel and select the most contrasted one. In this case Red.
3) Duplicate it. Name it for example "MASK". Now select it and turn the eye on.
4) Image > Adjust > Levels (Ctrl+L) and pick the white droper on the correct area, the darkest hair fill. Press Ok.
- Depening on the image you could need to move the slider a bit more to the left.
Now you have an alpha channel that uses the exact same antialiased the original image used. A beauty.
5) You can load this channel Selection > Load Selection and choose "MASK" Channel.
Let us work with your image
6) Invert the selection, we will work with the black area.
7) Now we need to expand this selection.
- Selection > Modify > Expand > And use 2 pixels.
From this selection make a new layer (Copy Paste) Name it "Halo1" and you will have a layer like this.
Note that we now have a slight brown halo of two pixels.
8) Duplicate this layer. Move this layer with the keyboard one pixel to the right and one down.
And move the other "Halo" Layer one pixel to the left and one up.
On this both layers use the blending mode "Lighter Color"
Remove any selection and flaten the layers.
You have now a slightly "fatter" image.
Let us compare it versus the original (on the right):
9) Now go back to your Alpha Cannel MASK where you saved the original selection.
- Selection > Load Selection > And choose your alpha channel.
10) Copy + Paste and now you have a neat image with no halos.
There are still a thin dark part on the tips of the hairs. To remove them try to use the "halo" layers, instead of moving them 1 pixel, move them 2 or 3.
An automatic quicker method (in recent versions of photoshop)
But I have not compared it to the other result.