The R shortcut is to select the rotation tool.
When you'll press ALT and click on a corner, you should see 3 little dots near your pointer.
Then this window is supposed to appear. If it doesn't work for you, I have no explanation why, and I think the ALT key is the same on Mac or Windows.
But you're not stuck, you can always use the TRANSFORM panel to change the angle of your shape. You can select your shape and enter the angle you want in the field for the rotation.
You'll find it in the menu "window" and select "transform."
You can press the link "Shape" in the top menu bar and the same menu will appear.
If you don't see that option in the menu bar of Adobe Illustrator, click on the small icon on the extreme top right on the menu bar and select "Shapes."