Hi I'm trying to build an interaction environment with Mujoco and OpenAI-Gym. I'm using a robot arm and a shadowhand model. When I use mocap to position the shadowhand in space, everything works fine. As soon as I put a second hand into the environment (for interaction), the hand is able to penetrate the second hand (see attached picture), even though there seems to be a resistance initially. In the case where I put actuators into the xml file and use "self.sim.data.ctrl" to control the robot without any mocap, the hand is not able to penetrate through other objects. I'm trying to understand why this happens, i.e., why there is this difference when controling the robot with mocap instead of direct actuation? Hope this is a feasible question, I'm new to Mujoco and trying to understand it better. Maybe I'm missing something in the documentation. Best, Sammy
mocap body positions are treated as constants by the simulator, even though you can change them at runtime. So if you set the mocap bodies into a configuration that implies penetration, there is nothing the simulator can do to remove that penetration. The recommended way to tele-operate MuJoCo simulations is to hook up a motion capture system to mocap bodies, then connect these mocap bodies with (soft) equality constraints to the dynamic bodies that you want to move around. This allows the simulator to displace the dynamic body from the mocap body and avoid penetration. You have to set the softness of the equality constraints and contacts appropriately, to decide which one wins in case of conflict. This requires some experimentation. Look at the Modual Prosthetic Limb model available on the Forum. This was designed for DARPA and used in a tele-operation setting.