Yes, ideally all files that need bleed should have it on all sides.
The bleed is an extra area that you add to your design to make sure there's no "white border" once the card or flyer is cut. The cutters that print shop use is not 100% perfect, pages might not be aligned or printed perfectly; when you add bleed, you make sure there's color everywhere no matter what.
That goes for business cards, postcards, flyers, covers, banners, etc. Even giant size banner need it unless you don't mind having a banner that is a bit smaller; it will be trimmed smaller if there's no bleed added to your layout!
The minimum amount of bleed should be around 0.125" (3mm) outside your document final size, ideally 0.25" (6mm). Each printer has his
own requirement for this.
The only time you don't need to use bleed is when there is absolutely nothing printed on all sides (eg. a design with a white border.) Some small ads in magazine or newspapers don't require it but it's better to provide a file with some bleed if you're not certain.
In this case, you can simply provide your final print-ready at its final size, without any extra bleed.
Another situation where you might not need to add bleed on all sides is when your file is provided as a "printer spread"; for example, a book cover design with the spine, front and back cover on the same layout OR a brochure with a fold OR a greeting card. The bleed will only be necessary for the printer on the outside edges. It's still a good idea to create your own designs (front cover and back cover) with bleed on all sides; it will be easier for you to merge them together later when you'll prepare the final print-ready file.
For magazine, they usually provide a template or precise instructions for bleed but the file still need to have bleed on all sides. The part where the fold or binding is will still be cut or folded. Printers also need bleed if the magazine or book is thick because they need to adjust the "creep"; pages are not cut exactly at the same exact width on thicker magazine or book, otherwise the pages in the middle would look wider than the pages that are closer to the covers.
For the same reasons, you need to be careful and keep your texts or important elements within the "safe margin" of your design and
nothing important should be on the "bleed" part!
The safe margin should be 0.125" (3mm) inside your design. With books, some printers can even require up to 0.5" (13mm!)
Some details on bleed here
Extra info about trim/crop marks