The whole idea with RGB to CMYK conversion is to create a CMYK image which on print will look as similar as possible to the original RGB image.
Since there are different printing devices, different kinds of paper, different inks and different ways to calibrate the equipment etc., the same set of CMYK values will look different on different setups. A CMYK color is not an objective color, it's simply a set of technical instructions for the printing device on which halftone percentages to use.
This is why color profiles were invented. They are mainly based on measurements on actual physical prints, not theoretical assumptions. Each print house decides which standard they are going to follow (for each device, paper type etc.) and calibrate their equipment to match.
So if your screen is calibrated according to a standard (often sRGB or Adobe RGB) and the print house has calibrated their equipment according to a standard, Photoshop is able to show you a (more or less) precise preview showing how your CMYK colors would look on print.
There is no such thing as generic CMYK. The only way Photoshop can show us a preview on screen is because it knows (or assumes) which CMYK color profile is in use.
In other words: If you don't know which CMYK profile your print house use, you can't do a proper RGB to CMYK conversion and you can't trust the preview on your screen!
Your custom rich black isn't neutral
You have chosen a rich black with the values (20, 20, 20, 100). This seems a little random to me (although I know that it's common practice for many).
The CMY inks does not have the same properties so mixing them equally will not create a neutral color. Your color will have a slight color tint depending on which standard the print house follows.
See here how an RGB black converted to CMYK (darkest possible neutral color) compares to your custom rich black when proofed with different color profiles:
A CMYK conversion will try to maintain the colors of the original. You are trying to force the darkest part of the images to have a specific non-neutral color, so you are (unintentionally?) trying to use CMYK conversion to change the perceived color. This is normally a thing you try to avoid.
Your custom rich black isn't very black
Uncoated paper usually allows 260-300% total ink and coated paper 300-340%. It differs from color profile to color profile.
The total ink of your custom rich black is only 160%. This can make it easier for the printer if you have large areas, but as the darkest point in an image, it's quite low.
A bright red color in the image might be around CMYK(0, 100, 100, 0) so it has a total ink of 200%. That's more than you allow in the darkest part of the image, so the transitions in the image might end up looking rather strange and some kind of banding might occur.
How it could be done using Custom CMYK
The only way I can see for you to force the darkest colors to CMYK(20, 20, 20, 100) using Custom CMYK is quite hacky. You would have to enter Ink Options > Ink Colors > Custom...:
Then you would have to manually adjust the CMY inks and overprints until you have invented a set of colors where equal amounts of ink gives a neutral color.
The result might look OK on screen, but the preview would be based on a custom set of inks no-one uses in real life. So to get a correct preview you would have to then assign the CMYK color profile provided by your print shop and a noticeable shift of colors would occur.
How I would do
The simplest way of matching the dark areas of a photo to a background in InDesign is in my opinion:
- Make sure that all images are RGB and that they all have the same color profile.
- Make sure that the darkest point of every image is RGB(0, 0, 0).
- Make sure that Document RGB is the same profile as used in the images.
- Place RGB images in InDesign.
- Use an RGB(0, 0, 0) swatch for the dark areas.
- Convert to the correct CMYK profile on export. Make sure to choose Convert To Destination (Preserve Numbers) to preserve Black text (and other deliberate clean inks).