For example in After Effects you are using mask for adjustments of your shape. You can subtract something with it what you needed. Can't find related future in AI.
Here is an image which is demonstrating my question for AI in AE:
If you want to make a shape by subtracting one shape from another, then you are probably looking for the Pathfinder. In the main menu, click
Pathfinderto bring it up.
Create two shapes in Illustrator that overlap, and select both. Then in the Pathfinder, there are several functions to try. Probably the easiest to get a grasp on is
Minus Front will use the shape on top to punch through the shape below. I have circled the button in red.
Experiment with the other functions in the Pathfinder. Most are fairly self explanatory.
There are other methods that can also be used in Illustrator, such as clipping masks, and in the newest Illustrator CC, the new Shape Builder tool functionality is worth having a look at. There are plenty tutorials online.
In Illustrator there's the Clipping Mask. A closed shape on the top of another shape or a group of shapes define what's actually visible. See a cartoon:
1) Select all your objects and make a group
2) Draw or drag the mask object over the objects to be masked. Bring it to the front.
3) Select the mask object and the group of other objects. Goto Object > Clipping mask > Make
Nothing is deleted until you expand the result. Until this you can release the Clipping mask and have the objects all visible. The mask object unfortunately loses its color when it's released
4) For inverse mask draw another (=complementary) shape (=red rectangle) that covers all objects
5) In the pathfinder panel minus the top object (=Circle) from the complementary shape. The result is filled with grey only for demonstrating. NOTE: This works only for closed shapes. It's not available for a group of various objects.
6) Use the resulted difference as the clipping mask. The blue hairline as the border of the result is visible only as the selection. Normally it's invisible.