Brushes in Photoshop are always raster. There's no such thing as a "vector brush" in Photoshop. And as with all raster content, scaling, especially enlarging, can cause degradation.
If you wish to use a brush and prevent any loss of quality, create the brush at the largest size you need. Reducing the brush size won't show the pixel issues as much. But it will still be raster in nature.
The only real way to get around this is to use Custom Shapes or Shape ayersLayers, not brushes.