I drew a 8.5"x 11"rectangle, then convert it to smart object and distort it according to the image size.(am creating a mock) But when i copy my document from illustrator and paste it on the rectangle in Photoshop every thing gets pixelated. How can i stop this from happening.
Looks like you are trying import your smart object into a low resolution image.
It also looks like you have zoomed in on the image. If you zoom in far enough on any raster image you will see pixels. Don't zoom in. View the image at 100% zoom.
Note that everything in Photoshop is raster. Even vectors are rendered as raster images.
Also note that Smart objects are rendered at the resolution of the image you are importing them into, so if you begin with a low resolution image and import a vector image as a smart object it will look like cr*p, especially if you zoom in on it.
The solution is to begin with a higher resolution image, then paste your smart object into that.