I suggested this already in a comment, but added the same now as an answer. The comment is removed.
Have the needed numbers and arrows as a prepared collection.
Drag with the normal selection tool (=F1) one number and arrow at the same time to its place. Let the number land to its final place, drag with the node tool (=F2) the arrow ends to their final places
If some curvature is needed, click a node and drag the handle holding the Shift key at the same time. In complex cases doubleclick the arrow where you want a new node. Simple curves need no new nodes. This case needed only one handle dragging.
You cannot skip a number nor use one twice if you take them from a prepared collection.
A number and an arrow in the collection can be a group, it makes moving them together a little easier, but doesn't affect the node tool.
Arrow stroke widths and arrowheads do not get scaled when the ends are dragged with the node tool. If you need scaling, allow it in the program preferences and drag the approximately right size with the normal selection tool.
Arrows and numbers can be useful to be kept in a new layer. This way they do not get hided accidentally and the unused one can be moved easily. The actual drawing can be locked as whole when you place or move numbers and arrows.