I followed this tutorial on GIMP to make an image transparent. Here is a snippet of the original image opened inside Windows Image Viewer (so the grey background is part of Windows Image Viewer, I wanted the white background to show up which was hard to notice if I had just uploaded the original image here rather than a snippet image),
Inside GIMP the image appears to be transparent (again this is a snippet so the grey background is not part of the image),
But when I export the image and open it again in Windows Image Viewer the white background is still present. I need the white background to disappear so that when I add it as an icon in a program it blends in seamlessly.
Kudos to Volker Siegel!
It turns out I was exporting as a JPG. After exporting as a PNG I got the full transparency shown here in MS Image Viewer again (so MS IV has a greyish background and you can see the image is transparent),