I want to implement deep reinforcement learning using a UR5 robot. A little research told me nowadays researchers are using openAI gym, Mujoco, rllab as their frameworks. The thing is, I want to train my UR5 robot in virtual environment and transfer the learning to my real robot. The task can be peg insertion. Previously I worked on a project using ROS with the UR5 robot and a Kinect. In that project, we are using MoveIt for motion planning. Everything has already been set up and I'm quite familiar with ROS and MoveIt. Do you think I should bother myself modeling the UR5 robot and adding RGBD camera using Mujoco (I am not sure if I can do this in Mujoco), and after that training it using openAI gym framework, and finally transferring learning to the real robot using ROS? Or should I stick with ROS and MoveIt while incorporating openAI gym only as my reinforcement learning framework? I just don't see why I need Mujoco if I can use MoveIt and gazebo for modeling, which is already done. Thanks for your advice on this.
Getting learning to work is much harder that modeling or moving the results from one environment to another (over ROS or another communication layer). If you are just getting started with reinforcement learning, you have a lot to catch up, and the best way to do it is to follow what others have done, at least initially. ROS and Gazebo and MoveIt have been around for a while, and are free. If they were suitable for reinforcement learning, people would not be buying MuJoCo licenses. Indeed MuJoCo was developed precisely because existing tools were not adequate for control and learning. Note that MuJoCo can load URDF files. You can also model RGBD cameras (including off-screen rendering in the cloud).
Thanks for your reply, Professor Todorov. I just want to ask an added question: Do you know if there's any way to transfer the trained model, say, in openAI gym, Mujoco, etc to the real robot? My professor values real robot implementation much more than simulation. So whatever I do, there should be a demo on the real robot.
Transfer from simulation to real hardware is one of the big open problems in the field. It depends on how accurate your model is (you should do system identification to fit the model parameters and make it more accurate), how accurate your state estimator is, and how the control policy was trained. With regard to training, ensemble methods help a lot, where you train the same policy with respect to several different models. See this paper for example: Ensemble-CIO: Full-body dynamic motion planning that transfers to physical humanoids Mordatch I, Lowrey K and Todorov E (2015). In IEEE/RAS International Conference on Intelligent Robots and Systems Movie
I have the model of my robot in mjcf format and want to use moveit to setup controller and trajectory planning, but moveit_setup_assistant cannot load the xml file. How can I use moveit with mujoco? I don't want to convert it to urdf format with the mujoco tag, because as I read in the mujoco book, it is not as complete as mjcf file. I want all the mujoco features. What is your suggestion for control and trajectory planning of a robotic arm described in mjcf format?