Hi, I want to do a (rough) coordinate transformation between an object's pos in 3D space and the corresponding coordinates in 2D from the camera's viewpoint. I am aware the reverse transformation (2D to 3D) could be done by calling mjr_select to get the object and then querying its position, but not sure whether there is any API combinations I could use for the forward transformation (3D to 2D).
Mapping 3D to 2D is essentially what rendering does. You need to know the camera projection matrix and do the computation yourself. The API does not have a function to automate this. Note however that OpenGL has functions to get the current projection and modelview matrices.
Yujin I think it is too late but I had the same problem and wrote the code to do a 2D projection. Check my repo. https://github.com/yusukeurakami/mujoco_2d_projection