It does not matter if you only have 2 swatches in the INDD file and then add "some RGB images in the document", this likely cannot result in having a 2-color PDF export.
Normally with a 2-color print requirement, you should attempt to build everything in InDesign using shapes and fonts exclusively laid out in InDesign, without pulling in external (raster) links, which may pushforce this back to having a 4-color PDF.
If the RGB images are full color, like pictures, you are sending off a full 4-color PDF to print.
In that case you need to remove the images, and find a way only to design this by only using the 2 swatches in the INDD file for everything.