The issues with this situation are the "personal favor" approach with your client and that there is nothing in writing. The latter actually benefits you as the creator of the work:
When you create artwork of any kind, the copyright law automatically
makes you the owner of a copyright of that work. Whether you create it
first and sell it later or whether you create it for a client, unless
you are an employee, you start out as the owner of the copyright. As
the owner of the copyright you have control of five things that can be
done with the work.
- You control reproductions of the work.
- You control derivative works made from the work.
- You control display and exhibition of the work.
- You control public performance of the work.
- You control distribution of the work.
Source - Graphic Artists Guild, To Sell or to Rent: The Difference
Between Copyright License and Transfer
Since there is nothing specifically stated contractually to prove otherwise, you legally own the right to the disputed artwork.
On the personal aspect of this case, I'd say you have a professional choice to make regarding the this situation. If I may speculate, it sounds like the falling out was ugly and your ex-client/friend is just trying to be spiteful about it. My advice would be to take the high road and move on. The sad truth is this most likely won't be the last time you run into this problem and it's never fun to cut work out of your portfolio for reasons outside of your control. But sometimes it makes more sense to cut your losses and focus on developing your skills and professionalism. It'll make you a better designer and person in the long run. (Caveat: I'm not a lawyer or a therapist, just speaking from experience)