The round tool in the effects menu called
Round Corners is broken (and has been forever). Either use the live corners feature (CC and newer) or the round any corner script described by @Joonas in a comment. But here's a perfect opportunity to bring in mathematical thinking.
You can do this on your own.
See the round has to occur at the point where of 2 perpendicular lines with the specified length (radius of circle) meet. Now there happens to be a function that does trace the line where the object happens to be x units away from the path. This function is called offset and is present (Object → Path → Offset Path...) to enable the system to stroke paths.
Image 1: Round (fillet) has to be where the line ends meet while being perpendicular to curve.
By drawing the offset curves all you need to find the place where the curves intersect. This is your round (fillet) circles center location, Intersection is easy to snap to when you have smart guides enabled.
Image 2: The offset lines and their intersection.
All that remain is to draw the circle at the intersection and cutting this in pieces.
Image 3: After drawing the circle and cutting it.
Why bother? Well its true that you might not want to do this in illustrator and instead use script or builtin function where present. But this also works if you draw things with a compass and a ruler, and some esoteric apps might also not have this. Besides it shows you how mathematical thinking can help in design. Good to know in any case.
TL;DR Use a script or the live corners feature.