The human excellence stems from gazillion different aspects, but honestly speaking, none contribute to it more than our tendency of growing under all circumstances. This willingness to get better, no matter the situation, has already enabled the world to clock some huge milestones, with technology emerging as quite a major member of the group. The reason why we hold technology in such a high regard is, by and large, predicated upon its skill-set, which guided us towards a reality that nobody could have ever imagined otherwise. Nevertheless, if we look beyond the surface for one hot second, it will become abundantly clear how the whole runner was also very much inspired from the way we applied those skills across a real world environment. The latter component, in fact, did a lot to give the creation a spectrum-wide presence, and as a result, initiated a full-blown tech revolution. Of course, the next thing this revolution did was to scale up the human experience through some outright unique avenues, but even after achieving such a notable feat, technology will somehow continue to bring forth the right goods. The same has turned more and more evident in recent times, and assuming one new discovery ends up with the desired impact, it will only put that trend on a higher pedestal moving forward.
The researching team at Massachusetts Institute of Technology CSAIL (Computer Science and Artificial Intelligence Laboratory) has successfully developed a brand-new LLM system called GenSim, which is designed to code new tasks for robots without extensive human intervention. To understand the significance of such a development, we must acknowledge how, at present, it’s overtly expensive to train a robotics machine on manipulation tasks. Even if we look past the costs, these simulations present a limited range of tasks because each behavior has to be coded individually by human experts. This translates to a reality where many bots cannot complete prompts for chores they haven’t seen before. So, how will the new development solve such a conundrum? Well, the answer resides in its make-up which consists of two particular modes i.e. goal-directed and exploratory. Talk about goal-directed setting first, GenSim here takes the chore a user programs in at the beginning, and then breaks down each step needed to accomplish that objective. On the other hand, with the exploratory setting, the system comes up with new tasks. In both the modes, however, the process starts with an LLM generating task descriptions and the code needed to simulate the behavior. Once that bit is done, the model would use a task library to refine the code, leading up to the final draft where these instructions can eventually create comprehensive simulations that teach robots how to do new chores.
“In the beginning, we thought it would be amazing to get the type of generalization and extrapolation you find in large language models into robotics,” said Lirui Wang, Ph.D student at MIT CSAIL, who is also a lead author of the paper explaining this study. “So we set out to distill that knowledge through the medium of simulation programs. Then, we bootstrapped the real-world policy based on top of the simulation policies that trained on the generated tasks, and we conducted them through adaptation, showing that GenSim works in both simulation and the real world.”
The researchers have already conducted some initial tests on their latest brainchild, tests that saw them pre-training the systems on 10 different tasks. After the pre-training was done, GenSim was able to automatically generate a sizeable 100 new behaviors. Furthermore, the stated technology successfully assisted robotic arms in several demonstrations, demonstrations during which simulations trained the machines to execute tasks like placing colored blocks, and they did so at a notably higher efficiency rate than comparable approaches. Owing to these findings, the team’s proposed use cases, as of now, reach across areas like kitchen robotics, manufacturing, and logistics etc.
“Robotic simulation has been an important tool for providing data and benchmarks to train and assess robot learning models,” said Yuke Zhu, Assistant Professor at The University of Texas at Austin, who is not involved with GenSim. “A practical challenge for using simulation tools is creating a large collection of realistic environments with minimal human effort. I envision generative AI tools, exemplified by large language models, can play a pivotal role in creating rich and diverse simulated environments and tasks.”
Having covered the capabilities, it’s important to mention that the system, in its current form, can only be useful when the task demands a simple pick-and-place mechanism. The researchers, though, will hope to eventually eliminate this limitation, replacing it with the ability to perform complex and dexterous tasks like using a hammer, opening a box, and more. One more shortcoming of GenSim is how it is prone to hallucinations and grounding problems, thus posing a potential safety concern. Assuming the said concern is taken care of in the near future, GenSim should be able to make the human input around here largely redundant, instead handing cutting-edge and comprehensive LLMs the responsibility to conceive new robotic activities.
“A fundamental problem in robot learning is where tasks come from and how they may be specified,” said Jiajun Wu, Assistant Professor at Stanford University, who is not involved in the work. “The GenSim paper suggests a new possibility: We leverage foundation models to generate and specify tasks based on the common sense knowledge they have learned. This inspiring approach opens up a number of future research directions toward developing a generalist robot.”
