Smart objects are only as good as their initial resolutions.
If you enlarge a smart object beyond it's original size, you fall victim to the same broken pixels you would get if the image was not a smart object.
Whether you tunnel into a smart object and enlarge the original element, or simply enlarge the smart object, you should not enlarge beyond the maximum size of the original image.
I suspect, regardless of what steps you take, you are simply enlarging your image too much to maintain a detailed appearance. However, without fully knowing the original dimensions of Smart Object elements and the new, desired, dimensions this is only a guess.
Smart Objects do not allow a user to make a tiny image huge. Smart Objects only ensure the original dimensions/resolutions are used upon output. This is beneficial for reductions or multiple transformations, but it doesn't create an infinitely enlargable image.
The exceptions to all of the above are live text and vector/shape layers. These can be enlarged well beyond their original sizes and maintain full document resolution.
Edited to add:
If they are vector smart objects you should have no issue enlarging the entire document.
If you need to enlarge just a groups specifically -
- right-click/control-click the group
- choose Duplicate Group
- select "new" from the drop down menu
- A new Document will open
- Resize that via Image size
- Drag the group back to the original document.