Is there any difference between the three besides that the brush and pencil are much more flexible than the pen?
Pen tool - Precise placement of anchors and control over curves
Pencil Tool - Freeform creation of disjointed paths.
Brush Tool - Freeform creation of disjointed paths with automatic application of brush appearance
Blob Brush Tool - Freeform creation of a brush path suing a calligraphic brush which will auto-expand when the cursor is lifted, leaving a shape rather than a stroked path
There are not vast, major difference on the whole. You can use the Pen Tool to draw a path and then apply a brush stroke to that path.. so...
It's not designed to be "one tool for one specific task". It's merely a set of tools to allow the user to draw/create in the manner they feel most comfortable with or the way they feel is most useful.
Highly related: What's the difference between the Brush and Pencil tools?