The short answer to your question would be to include a step between making your selection and inverting it: choose Select > Modify > Feather, and choose a 1~2px value.
Do yourself a huge favour, however, and rather than inverting your selection and hitting Delete, click on the 'Add Layer Mask' icon in the Layers palette. It looks like a rectangle with ...
On a small image like this, there's a quick cheat you can use…
Make your mask by selection any way you like - wand is fine, though I tend to use 'Select & Mask' these days for it's finer edge detection & smoother masking - then show the mask only (alt-click on it)
De-select the image, then select the Paint Bucket tool & black colour. Opacity 100%,...
There's no actual mask in the SVG. The entire design is made of white filled objects, which are grouped, and placed over a black filled rectangle. Shortcuts below are for PC, for Mac use Command instead of Ctrl.
Using the Selection Tool (V) select the graphic by clicking on a white area
Do Object > Compound Path > Make (Ctrl+8)
You can try to place the images to different layers in Photoshop and let the upper layer have blending mode Difference. That makes black all those pixels which are same in both images. The black area can be selected and used to generate a layer mask for the image which contains the interesting target.
In theory it should be perfect. It will work exactly in ...
Select > Select All
Type > Create Outlines
File > Export
Edit > Undo Create Outlines
By creating outlines you eliminate the path the type is sitting on and the export won't have any problems.
Be certain to Undo after the export... or copy/paste to a new file then create outlines and export and then merely don't save the new file.