ePrints@IIScePrints@IISc Home | About | Browse | Latest Additions | Advanced Search | Contact | Help

Translating Natural Language Instructions to Computer Programs for Robot Manipulation

Venkatesh, SG and Upadrashta, R and Amrutur, B (2021) Translating Natural Language Instructions to Computer Programs for Robot Manipulation. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, 27 September - 1 October 2021, Prague, pp. 1919-1926.

IROS_2021.pdf - Published Version

Download (1MB) | Preview
Official URL: https://doi.org/10.1109/IROS51168.2021.9636342


It is highly desirable for robots that work alongside humans to be able to understand instructions in natural language. Existing language conditioned imitation learning models directly predict the actuator commands from the image observation and the instruction text. Rather than directly predicting actuator commands, we propose translating the natural language instruction to a Python function which queries the scene by accessing the output of the object detector and controls the robot to perform the specified task. This enables the use of non-differentiable modules such as a constraint solver when computing commands to the robot. Moreover, the labels in this setup are significantly more informative computer programs that capture the intent of the expert rather than teleoperated demonstrations. We show that the proposed method performs better than training a neural network to directly predict the robot actions.

Item Type: Conference Paper
Publication: IEEE International Conference on Intelligent Robots and Systems
Publisher: Institute of Electrical and Electronics Engineers Inc.
Additional Information: The copyright for this article belongs to the Authors.
Keywords: Actuators; Object detection; Program translators; Robotics; Robots; Translation (languages), Constraint solvers; Imitation learning; Learning models; Natural languages; Neural-networks; Non-differentiable; Object detectors; Robot actions; Robot manipulation; Teleoperated, Forecasting
Department/Centre: Division of Electrical Sciences > Electrical Communication Engineering
Division of Interdisciplinary Sciences > Robert Bosch Centre for Cyber Physical Systems
Date Deposited: 23 May 2023 04:10
Last Modified: 23 May 2023 04:10
URI: https://eprints.iisc.ac.in/id/eprint/81721

Actions (login required)

View Item View Item