Flatness of an arc (or circle) determines the number of points needed along the arch to create a visually smooth curve. With things like circular selections, what is really created is a series of straight lines between points. The more points, the more straight lines there are. When the number of points reaches a visual tolerance, the straight paths visually appear to be an arc. However, it is still a series of tiny straight paths.
Some aspects of digital reproduction can not draw curves. Not even one. So the flatness setting is used and determines the number of these points and straight paths needed to draw a curve. Things like laser printers, imagesetters, and other output devices all use a flatness setting to produce visually smooth curves. Photoshop's selection tools use flatness as well.
When you contract a circular selection, you are asking Photoshop to remove points along the arcs. The more you contract, the more points you remove. Eventually you can remove so many points that the appearance of smooth arcs is lost and you end up with what you see in your example. If there were a direct method for adjusting the flatness settings for Photoshop's tools (there isn't) you could lower the flatness setting and see this effect on all curves, not simply contracted curves.
Many software applications used to have an adjustable flatness preferences. But with the increased performance of software, most flatness adjustments are done directly in the output device or in its settings rather than the applications themselves. It was one of those preferences users didn't fully understand but could absolutely wreak havoc on artwork if it were played with. It was probably best the developers removed those preferences so users can't improperly set them within the applications.
There is still one area in Photoshop where you can adjust flatness, clipping paths.
Of course, you never want to put a value in that field unless you know the proper flatness setting you need to alter your clipping path. If you do not use a value, then the path uses the output device settings for flatness, which is preferred). On screen, you'll see no visible change when applying a flatness setting. You only see the change upon output, which is why it can be a dangerous setting.