At first glance, everything looks fine... Circle is 2p and square behind has rounded corners of 1p but if the stroke is removed, there is clearly some inconsistency. What is the reason for this behavior?
Here is the same process but done in Illustrator, where rounded corners actually work:
Probably because neither the rounded corners nor the circles are true mathematically perfect arcs/circles.
Applications like InDesign, Illustrator and others that are used for vector graphics, use Bézier curves to approximate arcs/circles. They are close, but not perfect.
Also another thing is that transitions from a straight line into a sudden arc can sometimes look a bit odd - so there may some smoothing of the curves going on, to make the transition less extreme, so that it's more pleasing to the eye.
4*(Math.sqrt(2)-1)/3 ~ 0.5523 of that distance.
I was hoping to find an answer for this as well. Surely if round corners behave one way in Illustrator, that same behavior would be possible in InDesign.
Because of the obvious oblong appearance of corners in InDesign, editing corners in InDesign alone is not a viable option in my work. My workaround is creating all rounded objects like buttons or holding shapes in Illustrator and transferring them to InDesign instead. Unfortunately this method is destructive (live corners don't translate into InDesign), but this is the trade-off for greater control over the appearance of the shape.
They are not perfectly round because InDesign's graphic model does not use circle fragments; all circles and circle sections are created with Bézier curves, and ((in)famously) with these you cannot draw a perfect circle, only approximate it. This is also the case in Illustrator:
Fig. 1. Illustrator circle and rounded corners
But there are various levels of accuracy in "approximating".
Circles are done the same as in Illustrator, but for the Rounded Corner option, the InDesign engineers chose the easiest – and most naïve – solution: add a single Bézier control point at the intersection of the horizontal and vertical lines:
Fig. 2. InDesign's circle and rounded corners
As can be seen in the zoomed-in image, this is off in both outwards (at the starts and end) and inwards (at the outermost point) directions.
Fig. 3. The intersection of a circle and a rounded corner at a modest 800% zoom
However, mathematically speaking, the nearest approximate of a Bézier curve to a quarter circle segment needs its control points at a horizontal and vertical distance of 4*(Math.sqrt(2)-1)/3 ~ 0.5523 and not 1.000 (related Stack Overflow question; the article Approximating Circular Arcs with Cubic Splines by Philip Todd (PDF) discusses the accuracy of a number of approaches).
We can only wonder why the more accurate approximation is used in Adobe InDesign for circles but not for rounded corners, while in Adobe Illustrator it is used for both.
