I've always looked at it that elements at the same level of hierarchy in the containment structure should have the same border radius. That way, then sibling elements will always play nicely when sitting next to each other, irrespective of their relatives sizes and proportions.
So an outer container has a larger border radius than an inner container, and any container has a larger border radius than the buttons contained within it, but all those child/leaf elements should use a consistent radius - again so that they sit nicely next to each other.
Typically for buttons and other controls, I would use a border radius between 8% and 15% of the smallest dimension, so a 32px button might have a border radius of 3 pixels and importantly: that will define all the other controls as well.
Then a toolbar that contains those elements and has a height of 40 to 50 pixels might have a border radius of 4 to 5 pixels, but it depends on the padding and margins so that the curves look good in close proximity.
However this doesn't then mean that the outer container which may have a width of 800 should have an 80 pixel border radius. As you go up in the hierarchy, the border radius should be step increased, not percentage increased.
You let the inner controls drive the border radii from the inside-out.
So here's an example.
No increase between hierarchy levels - the outer boxes start to look too square compared to the inner buttons
Increasing by percentage - the outer boxes start to look too curved and comical compared to the inner buttons
Increasing by step increments - the outer boxes are in synch with the inner butons - no single part looks out of place.