Like the one shown on the RSVP design. Thank you.
For whatever reason "bracket frame" just isn't cutting it for me. Seems like a more stylized derivative of the barbed quatrefoil.
It's killing me that I can't find a more "official" name for it, but the first thing I thought of was this episode of 99% Invisible from a few weeks back. Maybe someone else feels like digging into The Grammar of Ornament for more context.
Or just stick with bracket frame.