(A bit of background: I am a C++ engineer with 0 graphical design experience)
I have a following task:
- Capture a 3D keypoints position from live actor using multi-camera motion capture technique
- Adjust the position and skeleton of a 3D character model
- Apply necessary environment physics (hair physics or actor interacting with virtual object)
- Render the final result (with green-screen support)
Capturing is not a real-time thing in my case (I capture and save at 60 FPS, but can only process ~2 FPS for accurate estimation, so all the processing is done after actor recording)
I have step 1 complete and it provides me a set of ~100 3d points position (it is running in a C++ application so I can export it to any format I want) including body and face tracking. But I am not sure which tools are good to use for other 3 steps. The concept is that the final render may have a green-screen as a background so I can post-process it as I like (or reuse the render without rendering the scene multiple times)
The scenes are not overly complex (up to 100k polygons total) so I don't expect the rednering to be overly demanding. As for the character 3D model I decided to go with FBX models, but I would like to hear an opinion about what is good for all other objects in the scene which I am not manually animating
I am not really restricted by the tool selection but I am restricted in knowledge in what is good for what. I tried using bs framework (C++ library) which lets me do some basic model bone adjustments but all the cool rendering effects and physics (especially) is left to me and I am not really looking forward doing this logic since I am 100% other tools provide such functionality out-of-the-box.
From my research it may be a good idea to go with Unreal or Unity but since I have no experience with both I am still looking for some advice.
Bonus points if the tools you advise will have a capability to do real-time rendering (once I go from ~2 FPS to ~20 FPS for motion capture processing) so it can be streamed to some content platform (Twitch, YouTube, etc), the current hardware I use is 8th gen Intel i7 and RTX2060, but will be upgrading depending on the requirements