I am simulating two sphere-like agents that move around in a space with 10 boxes at random positions. One of the agents can pull individual boxes using equality constraints that are activated and deactivated. Occasionally, I get warnings such as "Inertia matrix is too close to singular" that I can successfully catch and recover from. However, at other times I get the error "ERROR: Rank-defficient Hessian in HessianDirect" (sic!) which stops the program and is therefore impossible to recover from. From observing the simulation it is non-obvious to me when this error arises. How can it be avoided?
You don't want to have a simulation where the inertia matrix is too close to singular. You may be able to side-step it somehow, but it will usually cause numerical issues elsewhere. So try to increase masses, or armatures, or maybe scale everything to avoid very small objects. The error you are seeing should perhaps not be an error; I may change it to a warning in a future release and implement some recovery mechanism. It happens when the inertia matrix is near-singular and you are using the Newton solver, and there are not enough constraints to mask the singularity of the inertia matrix... so one way to avoid the error for now is use a different solver (CG or PGS). But again, try to get rid of the singular inertia warning; that will solve both the Hessian error and general numerical issues with your model.