Basically I have a composition layer that needs to have sort of fading out effect on it. I can achieve this in PS CS6 by aplying a vector mask on a layer and then choosing opacity gradient and applying it on created mask. But I'm struggling to achieve this effect in AE. Any suggestions?
Example:
There are MANY ways to achieve a certain task in After Effects and while the above answer will work, there is a much easier approach.
Select the layer you want to adjust go to your "Effect" tab select either "Radial Wipe" or "Linear Wipe" (depending on what fade you want) Then just choose the % fade, the angle of the wipe, and the feather
Hope this helps. Good luck and happy editing!
Applying a mask worked for me. Just remember to hit the chain icon to manipulate whether you want to apply the feather to the X-axis or Y-axis. 
What you can do is pre-compose the layer (in your example, the blue ball), then apply a rectangular mask to it, cutting the bottom part off. Next you need to go into the mask's properties and increase Mask Feather.
Here's what it looks like when I tried it:
This solution lets you animate the mask, too, so the fade-out effect can smoothly change (if you need it to, that is).
Hope this helps!
