Illustrator does not really excel at this sort of operation. This is why I linked to Excentro in the comment above. Excentro is specifically designed to create guillochés and it's artwork is exportable (as vector) to Illustrator. Of course, I realize Excentro is an extra cost.
The ways I can think to try and pull it off with only Illustrator include:
- Pattern brushes
- Transform Effects
With a Pattern Brush, you simply create tile-able base artwork:
Drag that to the Brushes Panel and choose Pattern Brush for the first pop-up window.
The second window will allow you to set some auto-generated corners (Illustrator CC or newer only):
You can try a few of the auto-corner options while the dialog is open to see if any fit well. If not, you may need to generate a custom corner. Unfortunately, custom corner creation is much more intricate and dependent upon the actual artwork. So, I won't go into custom pattern brush corner creation here, but there are tutorials on the web.
After clicking OK in the brush dialog you can simply draw a shape and apply the brush to it.
The more intricate and detailed the original brush artwork is the more extensive the overall final appearance can be.
Effects which create smooth curves really come down to Stylize > Scribble and Distort & Transform > Zig Zag. Neither of these on their own does, what I would consider a good job:
Zig-Zag would require rounded corners to work if no manual alteration is desired. But ultimately it's just one path and not what I would consider a guilloché. You'd have to double-up objects and carefully configure things to get any sort of guilloché aspect.
Lastly you could combine an effect with a pattern brush:
So I start with simple base art and apply an effect to create a guilloché-like circle:
I then apply another effect (saying OK to the warning that pops up) to move and copy this object horizontally.
Then I draw a rectangle with no fill and no stroke and move it behind all other artwork. The rectangle "splits" the first and last circle in half. This ensures repeating brush patterns will repeat at that "half circle" or "half emblem", thus lining up the area of repeat.
I then drag that artwork to the Brush Panel and choose Pattern Brush when asked. I set the auto-corner option to Auto-Slice and click OK.
Then simply draw a rectangle and click the brush I just created:
My pattern-defining rectangle is slightly off on one end. This is resulting in a small overlap where the repeat happens. Easily corrected, but I closed the file (without saving it) before I noticed the repeat issue.
Is this technically a intertwining guilloché? No, but it does look like it. Depending upon your original base art, you can create some very nice, intricate, appearing brushes in this way.
