Dynamic 3D model rendering with physics using known keypoints position [closed]

Question

(A bit of background: I am a C++ engineer with 0 graphical design experience)

I have a following task:

1. Capture a 3D keypoints position from live actor using multi-camera motion capture technique
2. Adjust the position and skeleton of a 3D character model
3. Apply necessary environment physics (hair physics or actor interacting with virtual object)
4. Render the final result (with green-screen support)

Capturing is not a real-time thing in my case (I capture and save at 60 FPS, but can only process ~2 FPS for accurate estimation, so all the processing is done after actor recording)

I have step 1 complete and it provides me a set of ~100 3d points position (it is running in a C++ application so I can export it to any format I want) including body and face tracking. But I am not sure which tools are good to use for other 3 steps. The concept is that the final render may have a green-screen as a background so I can post-process it as I like (or reuse the render without rendering the scene multiple times)

The scenes are not overly complex (up to 100k polygons total) so I don't expect the rednering to be overly demanding. As for the character 3D model I decided to go with FBX models, but I would like to hear an opinion about what is good for all other objects in the scene which I am not manually animating

I am not really restricted by the tool selection but I am restricted in knowledge in what is good for what. I tried using bs framework (C++ library) which lets me do some basic model bone adjustments but all the cool rendering effects and physics (especially) is left to me and I am not really looking forward doing this logic since I am 100% other tools provide such functionality out-of-the-box.

From my research it may be a good idea to go with Unreal or Unity but since I have no experience with both I am still looking for some advice.

Bonus points if the tools you advise will have a capability to do real-time rendering (once I go from ~2 FPS to ~20 FPS for motion capture processing) so it can be streamed to some content platform (Twitch, YouTube, etc), the current hardware I use is 8th gen Intel i7 and RTX2060, but will be upgrading depending on the requirements

Answer 1

Probably one option is using blender directly.

1. Motion capture info

2. Position and skeleton

3. Environment physics

4. Render, green screen or even composite a 3D environment or doing camera tracking

5. Realtime rendering, Not as powerful as a gaming engine, but pretty interesting.

6. Open source, so you could tweek some things if you need or can.

7. For the models you could export Daz 3D models into blender and rig them, if you want less detail there are some other options, Makehuman for example.