There are a few things you need to understand.
1) You are viewing the said layer at 400% zoom in Photoshop. This will cause the image to pixelate anyways. Since Photoshop is a raster editing program, its rendering engine works differently than Illustrator's and will show pixelation on zooming more than 100%.
2) By looking at the icon of the layer, I can tell that you have rasterized your smart object already. You cannot resize the smart object without losing quality after rasterizing it.
Smart objects have an icon that icon looks like this:
3) A problem similar to this occurs if you make a smart object in photoshop and then enlarge that beyond its original dimension. In that case, resizing will cause pixelation even if you have used vector layers. To solve that issue, open smart object and resize it according to the resolution you need it to be in original document.
NOTE: This problem mostly occurs only in smart objects that were created in Photoshop.