It's gone, which is a real shame, I should've made more use of it. Where's the best place to go to get free ampersands that I can then embed using
@font-face in CSS?
Check out The League of Movable Type over at http://www.theleagueofmoveabletype.com/. All of their fonts are free, designer-created, and come with the fonts needed for @font-face usage.
Google Web Fonts allows you to do pretty much what the ampersand site did, and perhaps with even more fonts to choose from. Check out the GWF Getting Started page linked below, and look for the section titled "Optimizing your font requests (beta)" which talks about how to link a small subset of any Google Web Font for use on your site. This will allow you to link just the ampersand and nothing else, like you wanted.
Don't forget... you have to encode the ampersand symbol when using this method. The ampersand encoding is just %26 for URLs. Here's an example using the encoding to link the ampersand from a font called Inconsolata:
<link rel="stylesheet" type="text/css" href="http://fonts.googleapis.com/css?family=Inconsolata&text=%26">