Hello everyone, in Mujoco 2.00, a new segment renedering mode was introduced. The change notes of 2.00 say: I am repeatedly running into problems mapping the pixel values do the correct geom IDs. I found that adding/removing visualization and enabling/disabling visualization of geoms (by alpha = 0 or via groups) changes the "IDs" of the drawn segments. This is true for both SEGMENT and IDCOLOR rendering modes. For example, if I have a scene with 16 boxes, and enable the SEGMENT mode, the color of each box changes for different label or frame rendering settings. The same happens when geoms are not rendered (e.g. by disabling their group). For the SEGMENT mode, this would not be a problem, as the colors are arbitrary anyway. However, this behavior makes using the IDCOLOR mode to match a pixel to a geom via its ID very difficult. It appears to me that "rendered IDs" (the pixel values) are assigned depending on the currently drawn geometries (including frames, labels, sites, ...), skipping not rendering geometries. While it would be possible to capture skipped geoms, also considering visualization geometries in all of its complexity seems like a very great challenge. I have attached several screenshots of an example scene illustrating the observations, as well as the corresponding model file. Would it be possible to change this behavior in a new version, where pixel values truely match the respective geom ID (and have everything else e.g. values above the number of geoms)? What is the recommendation on handling the rendered segment IDs / pixel values?
I was able to solve the problem by myself. I found the member mjvGeom* geoms; // buffer for geoms in mjvScene, which contains the information needed for the mapping. It still did not work out of the box. I had to shift the segment IDs by +1. For example, in geoms[6], segid=6, and objid=6 (objtype=geom), but actually, geom 6 corresponded to segment 7. Another example: I had geoms[23] with segid=23 with objtype=camera and objid=0 (a camera existing and shown in my scene), but segment 23 corresponded to geom 26 (which was specified by geoms[22] with segid=22). Anyway, it works for me now.
The segment id corresponds to the order of mjvGeoms in mjvScene, and they in turn point to mjModel objects, but not always. Some decorative elements (such as contact points and force arrows) are inserted in mjvScene without having a corresponding model element. The code that computes the segmentation color is: unsigned char seg[4] = { (geom->segid+1) & 0xFF, ((geom->segid+1)>>8) & 0xFF, ((geom->segid+1)>>16) & 0xFF, 0xFF }; glColor4ubv(seg); Everything is shifted by one to avoid completely black pixels (which would be indistinguishable from the background).