Start by trying to utilize more what you already have learned to use. Program draw.io seems to have what you want: https://drawio-app.com/introducing-draw-ios-new-sketch-feature/
Tried it by myself, too. At least it works:
The effort needed to get something less clinical than the perfect geometry basic shapes is zero, only set option Sketch =ON. Then every already drawn and selected item and every inserted new item will be sketchized. Making the same in a general purpose vector drawing program needs much more work. The difference can easily be 500 or 1000%. The difference reduces substantially if you prepare a big enough set of your own shapes that you only copy, paste and stretch to different sizes.
BTW the same core drawing tool is available in several different packages.I tried portable (=no install) Windows version of diagrams.net 15.7.3 Its diagrams can be exported as SVG which can be opened in Inkscape.
It's useful to see how the roughness is constructed by using only simple paths which could be as well drawn manually in Inkscape with no effects. Every connector and shape edge line is duplicated. Both copies have a little different random bendings. In Inkscape or other drawing program they are born without asking if you use the pencil tool and draw everything twice. The scribble fill in draw.io is a single path.
Here's a 100% manually drawn example in Inkscape.
The texts are not roughened in draw.io, they use normal computer fonts. Some of them can resemble manual writings. Beware: some commonly used ones have collected passionate haters and that can harm the reading. The example in my 1st image uses one. I am not at all sure would the text in my 2nd image be treated better, no matter it's unique.