It does not matter if you only have 2 swatches in the INDD file and then add "some RGB images in the document", this likely cannot result in having a 2-color PDF export.
Normally with a 2-color print requirement, you should attempt to do everything in InDesign using shapes and fonts exclusively laid out in InDesign, without pulling in external (raster) links, which may push this back to a 4-color PDF.
If the RGB images are full color, like pictures, you are sending off a full 4-color PDF to print.