Modelling Gelsight Sensor in Mujoco

Discussion in 'Modeling' started by Frederik Ebert, Jul 30, 2018.

  1. Hi,

    I'd like to build a model for a Gel-sight tactile sensor in mujoco (like the one used here: https://arxiv.org/pdf/1710.05512.pdf) for the purpose of simulating robotic manipulation with tactile-feedback policies.
    The Gelsight sensor has a layer of transparent gel where the deformation due to contact is observed by a camera and can be used to estimate contact forces, surface properties etc. It can also be used as an input for various learning algorithms.
    I know that mujoco currently does not support simulating soft objects like rubber, I'm also aware of the possibility for setting soft contacts in mujoco.
    However the soft contacts currently implemented in mujoco would not be sufficient for simulating a Gel-sight sensor since this would require (roughly) simulating the deformation of the Gel in contact with an object. Would it be possible to write a simple simulator for the Gel (like a small FEM simlation engine) and combine this with the mujoco physics simulator? How likely is this computationally feasible?

    The simulation does not need to be extremely accurate (we don't expect results to transfer quantitatively to real-world settings), therefore results that simulate the behavior of a soft surface and sensor readings qualitative would suffice.

    Thank you very much.

    Frederik