ChatGPT for Robotics: Design Principles and Model Abilities

This paper presents an experimental study regarding the use of OpenAI's ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineering and the creation of a high-level function library which allows ChatGPT to adapt to different robotics tasks, simulators, and form factors. We focus our evaluations on the effectiveness of different prompt engineering techniques and dialog strategies towards the execution of various types of robotics tasks. We explore ChatGPT's ability to use free-form dialog, parse XML tags, and to synthesize code, in addition to the use of task-specific prompting functions and closed-loop reasoning through dialogues. Our study encompasses a range of tasks within the robotics domain, from basic logical, geometrical, and mathematical reasoning all the way to complex domains such as aerial navigation, manipulation, and embodied agents. We show that ChatGPT can be effective at solving several of such tasks, while allowing users to interact with it primarily via natural language instructions. In addition to these studies, we introduce an open-sourced research tool called PromptCraft, which contains a platform where researchers can collaboratively upload and vote on examples of good prompting schemes for robotics applications, as well as a sample robotics simulator with ChatGPT integration, making it easier for users to get started with using ChatGPT for robotics.


Introduction
The rapid advancement in natural language processing (NLP) has led to the development of large language models (LLMs), such as BERT [2], GPT-3 [3], and Codex [4], that are revolutionizing a wide range of applications.These models have achieved remarkable results in various tasks such as text generation, machine translation, and code synthesis, among others.A recent addition to this collection of models was the OpenAI ChatGPT [1], a pretrained generative text model which was finetuned using human feedback.Unlike previous models which operate mostly upon a single prompt, ChatGPT provides particularly impressive interaction skills through dialog, combining text generation with code synthesis.Our goal in this paper is to investigate if and how the abilities of ChatGPT can generalize to the domain of robotics.
Robotics systems, unlike text-only applications, require a deep understanding of real-world physics, environmental context, and the ability to perform physical actions.A generative robotics model needs to have a robust commonsense knowledge and a sophisticated world model, and the ability to interact with users to interpret and execute commands in ways that are physically possible and that makes sense in the real world.These challenges fall beyond the original scope of language models, as they must not only understand the meaning of a given text, but also translate the intent into a logical sequence of physical actions.
In recent years there have been different attempts to incorporate language into robotics systems.These Figure 1: Current robotics pipelines require a specialized engineer in the loop to write code to improve the process.Our goal with ChatGPT is to have a (potentially non-technical) user on the loop, interacting with the language model through high-level language commands, and able to seamlessly deploy various platforms and tasks.
efforts have largely focused on using language token embedding models, LLM features, and multi-modal model features for specific form factors or scenarios.Applications range from visual-language navigation [5,6], language-based human-robot interaction [7,8], and visual-language manipulation control [9,10,11].However, despite the potential advantages of using LLMs in robotics, most of the existing approaches are restricted by a rigid scope and limited set of functionalities, or by their open-loop nature that does not allow for fluid interactions and behavior corrections from user feedback.
Models such as GPT-3, LaMDA, and Codex also show promise in zero-shot robotics scenarios when tasked with high-level agent planning [12,13] or code generation [14,15].These early demonstrations inspired us to investigate ChatGPT as a potentially more versatile tool for the robotics domain, as it incorporates the strengths of natural language and code generation models along with the flexibility of dialogue.ChatGPT's ability to engage in a free-form dialog and capture long context allows users to interact with the model in a more natural fashion, with flexible behavior correction.
In this paper, we aim to demonstrate the potential of ChatGPT for robotics applications.We outline a key concept that unlocks the ability to solve robotics applications with ChatGPT, which is the creation of a high-level function library.Given that robotics is a diverse field where several platforms, scenarios, and tools exist, there exists an extensive variety of libraries and APIs.Instead of asking LLMs to output code specific to a platform or a library, which might involve extensive finetuning, we instead create a simple high-level function library for ChatGPT to deal with which can then be linked in the back-end to the actual APIs for the platforms of choice.Thus, we allow ChatGPT to parse user intent from natural dialog, and convert that to a logical chaining of high-level function calls.We also outline several prompt engineering guidelines that help ChatGPT solve robotics tasks.
Our research shows that ChatGPT is capable of solving various robotics-related tasks in a zero-shot fashion, while adapting to multiple form factors, and allowing for closed-loop reasoning through conversation.In addition, we aim to show current model limitations, and provide ideas on how to overcome them.Our main contributions are listed below: • We demonstrate a pipeline for applying ChatGPT to robotics tasks.The pipeline involves several prompting techniques such as free-form natural language dialogue, code prompting, XML tags, and closed-loop reasoning.We also show how users can leverage a high-level function library that allows the model to quickly parse human intent and generate code for solving the problem; • We experimentally evaluate ChatGPT's ability to execute a variety of robotics tasks.We show the model's capabilities and limitations when solving mathematical, logical, and geometrical operations, and then explore more complex scenarios involving embodied agents, aerial navigation, and manipulation.We include both simulation and real-world experiments that result from ChatGPT's plans; • We introduce a collaborative open-source platform, PromptCraft, where researchers can work together to provide examples of positive (and negative) prompting strategies when working with LLMs in the robotics context.Prompt engineering is a mostly empirical science, and we want to provide a simple interface for researchers to contribute with knowledge as a community.Over time we aim to provide different environments where users can test their prompts, and welcome new contributions; • We release a simulation tool that builds on Microsoft AirSim [16] combined with a ChatGPT integration.This AirSim-ChatGPT simulation contains a sample environment for drone navigation and aims to be a starting point for researchers to explore how ChatGPT can enable robotics scenarios.
With this work we hope to open up new opportunities and avenues for future research fusing LLMs and robotics.We believe that our findings will inspire and guide further research in this exciting field, paving the way for the development of new, innovative robotics systems that can interact with humans in a natural, intuitive manner.For more details, we encourage readers to view detailed videos of our experiments in the project webpage.

Robotics with ChatGPT
Prompting LLMs for robotics control poses several challenges, such as providing a complete and accurate descriptions of the problem, identifying the right set of allowable function calls and APIs, and biasing the answer structure with special arguments.To make effective use of ChatGPT for robotics applications, we construct a pipeline composed of the following steps: 1. First, we define a high-level robot function library.This library can be specific to the form factor or scenario of interest, and should map to actual implementations on the robot platform while being named descriptively enough for ChatGPT to follow; 2. Next, we build a prompt for ChatGPT which describes the objective while also identifying the set of allowed high-level functions from the library.The prompt can also contain information about constraints, or how ChatGPT should structure its responses; 3. The user stays on the loop to evaluate code output by ChatGPT, either through direct analysis or through simulation, and provides feedback to ChatGPT on the quality and safety of the output code; 4.After iterating on the ChatGPT-generated implementations, the final code can be deployed onto the robot.
We show a visual depiction of this pipeline in Figure 2 for the example of a household robot.

Construction and description of the robotics API library
Robotics being a well-established field, there already exists a multitude of libraries, either black-box or open-source, that can be used for basic functionalities in both the perception and action domains (e.g.object detection and segmentation, mapping, motion planning, controls, grasping).If properly specified in the prompt, the LLM is able to use these pre-defined functions for robot reasoning and execution.
One important prompt design requirement is that all API names must be descriptive of the overall function behavior.Clear names are essential to allow the LLM to reason over functional connections between APIs and produce the desired outcome for the problem.Hence, we can define high-level functions, which act as wrappers over actual implementations from the respective libraries.For example, a function named detect_object(object_name) could internally link to an OpenCV function or a computer vision model, whereas something like move_to(x, y, z) could internally invoke a motion planning and obstacle avoidance pipeline along with the appropriate low-level motor commands for a drone.Listing such a collection of high-level functions in the prompt is key in allowing ChatGPT to create logical sequences of behavioral primitives, and in generalizing to different scenarios and platforms.
Depending on the context, we recommend explaining the function of APIs and if needed, breaking them down into sub-components with clear inputs and outputs, similar to code documentation.In Figure 3 we present an example of a good API prompting strategy for a home cook robot scenario.The strategy presented allows ChatGPT to reason about the order and content of tasks according to the functions the robot is actually able to execute.In contrast, we refer the interested reader to Appendix A.1 for an example of how ChatGPT reasons when no API guidance is given, which leads to a unbounded text-based answer, or to Appendix A.2 for an example of API under-specification, which leads to hallucinations over function call parameters.
We note that unlike the brittle structure of classical symbolic AI, which required rigid pre-defined relationships between objects and functions, LLMs are capable of defining new functions and concepts altogether when relevant to a particular problem.This capability confers flexibility and robustness to LLMs when dealing with robotics applications.Figure 4 shows how ChatGPT can create new high-level concepts and even low-level code when needed to solve a problem, even fusing existing APIs.The user on the loop can take advantage of this ability as a design strategy, and iteratively define new APIs with the help of the LLM when the current ones are not sufficient to solve the task at hand.

Clear description of the task details in the prompt
By providing a clear and concise description of the desired robotics task and its context, ChatGPT can generate more accurate responses.A good context description should contain, besides robot APIs: • Constraints and requirements: specify constraints or requirements that are relevant to the task.If the task involves moving objects, you might specify the weight, size, and shape of the objects to be moved.• Environment: describe the environment in which the robotics task is taking place.For example, if the task is to navigate a maze, you might describe the size and shape of the maze, as well as any obstacles or hazards that need to be avoided.• Current state: describe the current state of the robotics system.For example, if the task is to pick up an object, you might describe the current position and orientation of the robot and the object.• Goals and objectives: state the goals and objectives of the task.If the task is to assemble a puzzle, you might specify the number of pieces that need to be assembled and the desired completion time.
• Solution examples: demonstrate how similar tasks can be solved as a means to guide the LLM's solution strategy.For example, if a task involves interactions with the user, we can describe an example of how and when the robot should be asking for the user's input (see Fig. 5).Note that priming can also introduce biases, so we should provide a diverse range of examples and avoid overly prescriptive language.
Imagine we are working with a household robot.The job of this robot is to make an omelette.Can you make use of these to write code to go to the kitchen and make an omelette?
Sure, here is an example Python code that utilizes the provided functions to prepare an omelette for an elderly person: fridge_location = locate_object("fridge") bowl_location = locate_object("bowl") pan_location = locate_object("pan") stove_location = locate_object("stove") oil_location = locate_object("oil") # Start the omelette-making process go_to_location(fridge_location) pick_up("eggs") go_to_location(bowl_location) use_item("eggs") # Crack eggs into the bowl go_to_location(pan_location) pick_up("pan") use_item("pan") # Place the pan on the stove go_to_location("oil") pick_up("oil") go_to_location(pan_location) use_item("oil") # Pour oil into the pan go_to_location("bowl") pick_up("bowl") go_to_location(pan_location) use_item("bowl") # Pour the egg mixture into the pan go_to_location(stove_location) use_item("stove") # Turn on the stove print("Enjoy your omelette!")Even a well designed prompt might not contain all necessary information needed to solve the problem, or in some cases ChatGPT is not able to generate the correct response in a zero-shot fashion.In these cases, we find that a simple and effective strategy a user can take is to send additional instructions to ChatGPT in the chat format describing the issue, and have it correct itself.Previous approaches that rely on GPT-3 or Codex models [15,14] require the user to re-engineer the input prompt and generate new outputs from scratch.The dialog ability of ChatGPT, however, is a surprisingly effective vehicle for behavior correction.Section 3.2 and the supplementary videos show examples of interactive behavior between the user and ChatGPT.

Special arguments to bias the answer's structure
Different prompting methodologies can be used to force the output of the model to obey some specific pattern.For instance, a user might want to automatically parse ChatGPT's output so that it can be used in other scripts for real-time execution.As shown in Figure 3, a simple strategy is to directly ask ChatGPT to produce code in a specific language (e.g.Python, C++).What usually follows is a free-form text paragraph followed by a code block.A more structured response can be produced by requesting the model to use XML tags to help us parse the output automatically, as shown in Figure 5.
In other settings users might want to force the model to output answers following a list pattern as opposed to code or free-form text.Appendix A.3 displays an example of such approach, where the last line of the user's prompt directs the model's output.Can you make use of these to write code to go to the kitchen and make an omelette?Try to use these low level functions to construct more high level behaviors and then solve the task using those high level primitives.
(rest of the response omitted for brevity) Imagine you are helping me interact with a mobile ground robot.You are in a room that contains several objects: three chairs, a glass table, a mirror, and two couches.All of your outputs need to be identified by one of the following tags: <question> Always ask me a clarification questions if you are unsure </question> <reason> Explain why you did something the way you did it </reason> <command> Output code command that achieves the desired goal </command> For example: Me: Go to the chair You: <question> There are three chairs in the world, which one do you want me to go to?</question> Me: Go to "chair_3" You: <command> target_location = get_location("chair_3"); move_to_location(target_location); </command> <reason> I found the location of the chair with the get_location function, and moved the robot towards the desired location </reason> Are you ready?
Yes, I am ready.Please give me your first command.
Go to the couch <question> There are two couches in the room.Which couch do you want me to go to?</question> Go to the second sofa <command> target_location = get_location("second_sofa"); move_to_location(target_location); </command> <reason> I used the get_location function to find the location of the second sofa and then moved the robot towards that location.</reason> Now move to the other one <command> target_location = get_location("first_sofa"); move_to_location(target_location); </command> <reason> I used the get_location function to find the location of the first sofa and then moved the robot towards that location.</reason> with the user to ask for help.In addition, we force responses to be framed into specific XML tags which can aid code parsing in a post-processing stage.

ChatGPT abilities for solving robotics problems
In this section take a deeper look into ChatGPT's abilities for solving robotics problems.Specifically, we investigate how well ChatGPT performs when dealing with a variety of robotics-related tasks, from simple spatio-temporal reasoning problems all the way to real-world aerial agent and manipulation deployments.We highlight several interesting capabilities we observed during these experiments.While ChatGPT's capabilities are impressive, practical deployment safety considerations should not be overlooked, especially in the case of physical robotics deployments.As shown in Figure 2, we find it necessary to have a human on the loop to monitor and intervene in case of unexpected behaviors generated by ChatGPT.Additionally, the use of simulators can be particularly helpful to evaluate the model's performance before deployment in the real world.We emphasize that the use of ChatGPT for robotics is not a fully automated process, but rather acts as a tool to augment human capacity.
We provide all initial prompts for the problems in this section in Appendix B. For the sake of brevity we only include brief excerpts from each task.The full conversations can be found in the following repository: https://github.com/microsoft/PromptCraft-Robotics.

Zero-shot task planning
First, we note that ChatGPT is able to solve several robotics tasks in a zero-shot fashion, with access only to the prompt and the description of a function library, and no requirement of actual code examples.

Spatio-temporal reasoning: catching a basketball with visual servoing
In this example, we ask ChatGPT to control a planar robot equipped with an upward-facing camera.The robot is expected to catch a basketball using a visual servoing method based on the appearance of a basketball.We see that ChatGPT is able to appropriately use the provided API functions, reason about the ball's appearance and call relevant OpenCV functions, and command the robot's velocity based on a proportional controller.Even more impressive is the fact that ChatGPT can estimate the appearance of the ball and the sky in the camera image using SVG code.This behavior hints at a possibility that the LLM keeps track of an implicit world model going beyond text-based probabilities.
[Initial model prompt found in Appendix B.1] Write a python script that executes a visual servoing approach towards catching a basketball in a court.You can use opencv functions to detect the ball as an orange blob.
Here's a sample implementation of the script: This implementation uses a simple visual servoing approach where the robot moves towards the ball at a constant velocity.The velocity is proportional to the difference between the ball's and robot's locations, so the robot will move faster as it gets closer to the ball.You can adjust the gain (i.e., the multiplication factor of 0.5) to control the speed of the robot.

Aerial robotics: real-world drone flight with intuitive human-robot interface (Click for video)
We also used ChatGPT with a different API implementation to control a real drone.ChatGPT provided an extremely intuitive natural language interface between the user and the robot, writing robot code based on ambiguous and ill-defined instructions, and asking clarification questions when necessary.The model was also able to write complex code structures for drone navigation (circular and lawnmower inspection) based solely on the prompt's base APIs.
[Initial model prompt found in Appendix B.2] I am thirsty.Find me a drink Question: What type of drink are you looking for?There are several options in the scene, including "coconut water" and "diet coke can".I just came back from the gym and am looking for a healthy option.
In that case, you might like "coconut water".Here is the code to fly to the "coconut water" object:

fly_to(drink_position)
Explanation: This code first retrieves the X, Y, Z, and angle position of the "coconut water" object from the dict_of_objects dictionary.The fly_to method is then used to fly the drone to the retrieved position.The drone will fly to the position of the "coconut water" object and look in the direction of the angle.I am still thirsty.Can you find something with sugar and a red logo?I'm assuming you are referring to a can of "regular coke".Here is the code to fly to the "regular coke can" object: drink_position = self.dict_of_objects["regularcoke can"] self.tello.fly_to(drink_position) Fantastic.Now I want to take a look at my orchid.However, the chair might be blocking its view, so I want to inspect it from 3 points on a half-circle from -60deg to -120, 1m away.Make sure that the drone's heading points towards the orchid from each point.
Here's the code to inspect the orchid from 3 points on a half-circle:

Aerial robotics: AirSim industrial inspection (Click for video)
We also applied ChatGPT to a simulated domain, using the Microsoft AirSim [16] simulator.We explored the idea of a potentially non-technical user directing the model to control a drone and execute an industrial inspection scenario.We observe from the following excerpt that ChatGPT is able to effectively parse intent and geometrical cues from user input and control the drone accurately.
[Initial model prompt found in Appendix B.3] Fly to the turbine.
Question -There are two turbines in the world, which one do you want me to fly the drone to?Turbine 1 or Turbine 2?
Turbine 2, please.Fly to the turbine while keeping 12 meters away from the turbine along the X axis, and reach an altitude of 50 meters.
Here's the code to fly the drone to Turbine 2 while maintaining a distance of 12 meters along the X-axis and reaching an altitude of 50 meters: Here's the code to fly the drone in the bottom-right direction at an angle of -30 degrees from the horizontal axis, in the YZ plane for a distance of 30 meters, and then back to the current position:

User on the loop: interactive conversations for complex tasks
As a next step, we focused on the ability of ChatGPT to perform more complex tasks with a user on the loop offering textual feedback.We observed that this modality of interaction allows for complex constructs such as curriculum learning, where the system is taught smaller-scale skills which can be combined towards larger and more intricate tasks.In addition, we found that ChatGPT is able to take high-level textual feedback regarding generated code or its performance and map it to the required low-level code changes, thus enabling potentially non-technical users to interact with it with ease.

Manipulation with curriculum learning (Click for video)
As a first example, we set up a block arrangement task using a robot arm, for which we devised a curriculum.ChatGPT was asked to learn the simpler skills of picking up objects, and then placing them.Subsequently, when asked to use these skills to perform a more involved task of block arrangement, ChatGPT chains these learned skills together logically.In addition, the model displays a fascinating example of generalizability when bridging the textual and physical domains when tasked with building the Microsoft logo out of wooden blocks.This task requires memory of how the corporate logo looks like, including its colors, followed by building an abstraction of the logo into physical parts which are constructible by the existing robot actions.
[Initial model prompt found in Appendix B.7] [The beginning of the conversation is omitted for brevity.The following excerpt requires ChatGPT to build a new function using previously defined APIs, and later use this function for more complex tasks.]I want you to learn how to place an object in a position.Considering you are holding a generic object, you first need to first move to a safe position above the target location, bring the object down and then release it.
Understood.Here's a Python function that takes the position where the object should be placed, moves the robot arm above the position, brings the object down, and releases it.Note that the function assumes that the robot arm is holding an object.The Microsoft logo is a colored rectangular with four squares arranged in a grid pattern.The colors used in the logo are blue, green, orange, and yellow.The logo has gone through several iterations over the years, but this is the most recent and recognizable design.
Sure! Here's the Python code to build the Microsoft logo on top of the white pad using the blocks: [Code not show for brevity.Please consult the link for the full conversation] This code gets the positions of the white pad and all four blocks using the get_position function, picks up each block in turn using the pick_up_object function we defined earlier, calculates the position to place the block on the white pad, and places the block on the white pad using the place_object function we defined earlier.We use the block colors to ensure that the logo is built correctly.We place the blue block in the bottom left position, the yellow block in the bottom right position, the red block in the top left position, and the green block in the top right position, as specified in the prompt.
[snapshots from manipulator video]

Aerial robotics: AirSim obstacle avoidance (Click for video)
We tasked ChatGPT to write a goal-reaching algorithm with obstacle avoidance for a drone equipped with a forward facing distance sensor.ChatGPT built most of the key building blocks for the avoidance algorithm, but required some human feedback on steps it missed regarding the drone's orientation.Although the feedback was provided entirely in high-level text, ChatGPT improved its solution with localized changes to the code where appropriate.
[Initial model prompt found in Appendix B.4] Assume I give you some XYZ coordinates as a goal.I want you to control the drone such that it moves towards the goal while avoiding obstacles in front of it.Instead of flying directly to the goal, the drone should take small steps while reassessing the obstacle situation at every step.The drone is equipped with a distance sensor that returns the distance to the closest obstacle directly in front of the drone.You can access it by calling get_distance().I want you to make sure to fly only if there is at least 10 meters of distance in front of the drone.If not, you should rotate the drone until there is at least 10 meters of clearance in the front and then take a step in the direction that the drone is currently facing.Don't forget to face the goal again after taking a step.Got it?
Yes, I got it.Here's how I would approach the problem.After you take a step, you need to face the goal again.
You're right.After taking a step, we need to recalculate the yaw angle to face the goal again.Here's the modified code that includes this step: '''Same code as before''' yaw = math.atan2(goal_pos[1]-current_pos [1], goal_pos[0]-current_pos[0]) * 180 / math.piset_yaw(yaw) This code should ensure that the drone faces the goal after taking a step, and continues moving towards the goal while avoiding obstacles in front of it.You are not using the current yaw angle when computing the step vector.Remember there is a get_yaw() function.
You're right.After taking a step, we need to recalculate the yaw angle to face the goal again.Here's the modified code that includes this step: # Take a step towards the goal while maintaining the yaw angle step_size = 5 # take steps of 5 meters current_yaw = get_yaw() step_vector = [step_size * math.cos(current_yaw),step_size * math.sin(current_yaw),0]

Perception-action loops
We also evaluate ChatGPT's ability to reason about perception-action loops.At a first level, we outline the model's ability to make use of the API library to construct perception-action loops in its code output.The model correctly employs perception functions such as image acquisition and object detection to extract the relevant information for robot navigation and controls.
At a second level of complexity, we try to answer the question of whether ChatGPT's dialogue system can serve as a closed feedback perception-action loop in itself.We explore the idea of continuously feeding the model with perception information via textual dialog, where we input in observations (converted into a textual format) to ChatGPT during a conversation.We find that ChatGPT is able to parse this stream of observations and output relevant actions.

Embodied agent: closed loop object navigation with API library (Click for video)
We provided ChatGPT access to a computer vision model as part of its function library, and tasked it to explore an unknown environment and navigate to a user-specified object.The object detection API (YOLOv8 [17] in the back-end) returned bounding boxes, and ChatGPT generated the code to estimate relative object angles and navigate towards them.When we offered ChatGPT additional information from a depth sensor it produced an improved algorithm with pixel depth masking, expressed as a modular pipeline.
[Initial model prompt found in Appendix B.6] I need you to help me find an object.Objects might be scattered around the scene, so if you don't find it right away, a good strategy is to turn around a few times to see if it becomes visible.Give me code to explore and find the bottle.
Got it.Here's some code that will help you explore the environment and find the bottle: This code will keep exploring the environment until a microwave is found.Once a microwave is found, the agent will move towards it and stop when it gets within 30 cm of it.
[snapshots from embodied object navigation]

Embodied agent: closed loop visual-language navigation using ChatGPT's dialogue (Click for video)
In experimental character, we also evaluated ChatGPT's ability to act as a perception-action control feedback loop using its dialogue function.In this mode of operation new state observations are fed into the system as dialogue text, and the model's outputs are executed by the robot at each step, generating new observations.We note that this type of behavior is heavily dependent on a dialog-based LLM such as ChatGPT, which can naturally take receive a stream of observations.We constructed a scenario using the Habitat simulator [18] and tasked the agent to navigate to an area of interest.At each step we provided ChatGPT with a description of the scene in terms of polar coordinates of the visible objects relative to the agent.We constrained the output of ChatGPT to only return a forward distance of motion and the turning angle.We verify that the model is able to complete simple navigation tasks, but more research is needed in terms of properly describing the scene context in textual or vector form for more complex tasks and environments.For additional experiments, we refer the reader to Appendix C.

Reasoning and common-sense robotics tasks
We tasked ChatGPT with answering simple logical reasoning questions, as well as basic robotics problems, shown in Appendix D. We observe that ChatGPT offers a rich foundation of common-sense and logical reasoning capabilities upon which more advanced robotics abilities can be built.This logics grounding allows the user to interact with the model more naturally, as opposed to having to prompt or define every concept from scratch.Furthermore, ChatGPT's out-of-the-box understanding of basic robotics concepts such as control, camera geometry, and physical form factors makes it an excellent choice to build upon for generalizable and user-friendly robotics pipelines.

PromptCraft, a collaborative tool for LLM + Robotics research
Prompting is a crucial component to generate the desired behaviors in large language models (LLMs).Prompt engineering is particularly challenging at the intersection of LLMs with robotics, where there is a lack of comprehensive and accessible resources that provide examples of positive (and negative) interactions.To address this gap, we introduce PromptCraft 1 , a collaborative open-source platform for researchers to share examples of prompting strategies and test their algorithms in sample robotic environments.
PromptCraft is a Github-based platform that allows researchers to share examples of prompt engineering strategies within different robotics categories, such as navigation, grasping, and manipulation.Users can submit their examples and rate others' submissions, which we hope will create a community-driven resource for researchers working with LLMs.Submissions of prompts and dialogues are primarely based on text, but we encourage users to share videos and images depicting the robot's behavior, especially for real-world deployment scenarios.
In addition to providing a platform for sharing prompt examples, PromptCraft also offers an AirSim [16] environment with a ChatGPT wrapper for researchers to prototype prompts and algorithms in a controlled simulated setting.We welcome contributions of new test environments to expand the range of scenarios where researchers can test their algorithms.
With Promptcraft we aim to support the empirical science of prompt engineering and enable researchers to advance the field.

Related Work
Natural language and robotics: Natural language processing (NLP) has long been recognized as a crucial component for human-robot interaction.There are many applications where robots can benefit from NLP, including but not limited to task instruction, navigation, and information retrieval.Classically, modeling human-robot interactions using language is challenging because it forces the user to operate within a rigid set of instructions [19], or requires mathematically complex algorithms to keep track of multiple probability distributions over actions and target objects [20,21].More recent works explore neural networks to implicitly keep track of the complex mapping between language and actions, but such techniques often require vast amounts of labeled data for training [22,5,6,23] Large (vision and) language models for robotics: The Transformer architecture, introduced in the paper by [24], has revolutionized NLP and has also shown great promise in robotics.Transformers have been used for robot control and planning [25,26,27], object recognition [28], and robot navigation [29].A more common use of transformers in robotics has been as feature extraction modules for one or more modalities simultaneously.These systems are often coupled with additional features from pretrained large-scale vision and language models models [30,10,31,32,11,9].
Models such as SayCan [31] focus on grounding LLMs so that free-form text commands are used to compute a value function to rank the best action types within a robot-specific library.RT-1 [33], on the other hand, takes an end-to-end approach to learn the mapping between language commands low level actions, without the use of intermediate high-level functions.Recent works have also explored the ability of large language models (LLMs) for zero-shot high-level robotics task planning [15,14,12].These models make use of prompting structures with pre-defined functions, behaviors, and examples to guide the generation of the model's answers.[13] also explore the use of interactivity between user and LLM for table-top manipulation settings.
Conceptually, the main difference of these approaches with respect to our work, which leverages ChatGPT [1], is the conversational ability of our LLM, which allows the user to interactively improve and correct the robot's behavior (as opposed to re-engineering the prompt from scratch and generating another zero-shot answer).In addition, our works aims to provide a generalizable pipeline and set of principles to be used by researchers in different fields of robotics, as opposed to focusing on a single domain such as table-top manipulation or task planning.
Prompting LLMs with APIs, and its connections to symbolic AI: When designing LLM prompts for robotics applications, users often make use of high-level library of APIs to represent specific behaviors to be used.We can draw a connection between this approach with classical symbolic AI, which uses logic and rules to represent and reason about knowledge [34].While the traditional symbolic AI approach presented difficulties in new knowledge acquisition and dealing with out-of-distribution data, we believe that LLMs can overcome these challenges.As we showed in Section 2.1 and Section 3, models such as ChatGPT can compose new primitive functions based on the context and generate code for them automatically.

Conclusions and Future Work
We presented a framework for using ChatGPT for robotics applications.The framework entails designing and implementing a library of APIs that for robot control which are amenable to prompt engineering for ChatGPT.We discussed design principles for creating such APIs and prompting strategies that can be used to generate code for robotics applications via ChatGPT.The proposed framework allows the generated code to be tested, verified, and validated by a user on the loop via a range of methods including simulation and manual inspection.We demonstrated how the framework can be used for multiple applications ranging from simple common-sense robotics knowledge tasks all the way to deployments in aerial robotics, manipulation and visual navigation.
We believe that this work presents only a small fraction of what is possible within the intersection of large language models operating in the robotics space.We hope to not only inspire other researchers to take these next steps, but to also help them achieve results with the use of the PromptCraft collaborative tool.
We emphasize that these tools should not be given full control of the robotics pipeline, especially for safetycritical applications.Given the propensity of LLMs to eventually generate incorrect responses, it is fairly important to ensure solution quality and safety of the code with human supervision before executing it on the robot.We expect several research works to follow with the proper methodologies to properly design, build and create testing, validation and verification pipelines for LLM operating in the robotics space.
Most of the examples we presented in this work demonstrated open perception-action loops where ChatGPT generated code to solve a task, with no feedback was provided to the model afterwards.Given the importance of closed-loop controls in perception-action loops, we expect much of the future research in this space to explore how to properly use ChatGPT's abilities to receive task feedback in the form of textual or specialpurpose modalities.

ChatGPT for paper writing
Please note that this paper was largely written with the assistance of ChatGPT, with prompts provided by the authors.The model's output was thoroughly revised and adapted, we note that the use of LLMs can significantly speed up the writing process, and we recommend their use to the interested reader.

Figure 2 :
Figure 2: Robotics pipeline employing ChatGPT with the user on the loop to evaluate the output's quality and safety.

Figure 3 :
Figure 3: Example of a good prompting practice when defining different APIs for a robotics problem.Clear function definitions that can be executed by the robot, and the task objectives are explicitly stated.
Imagine we are working with a household robot.The job of this robot is to make an omelette.The objects available around are: fridge, bowl, pan, oil, stove The main functions you can use are: locate_object(object_name): Returns the XYZ coordinates of an object of interest.go_to_location(object_name): Moves robot to a location specified by XYZ coordinates.Returns nothing.pick_up(object_name): Picks up the object of interest.Returns nothing.use_item(object_name): Takes the name of an object as input.Performs an appropriate action on the object at hand (pour oil, crack eggs).Returns nothing.

Figure 4 :
Figure 4: Example of new API development using LLM intrinsic reasoning capabilities.The model improves its answer in comparison to Figure 3, which contained the same task but with a fixed set of APIs.

2 #
def search_for_bottle(): num_turns = 4 # Turn around 4 times to look for the bottle for i in range(num_turns): # Turn left 90 degrees to face a new direction turn_left() img = get_image() obj_list, obj_locs = detect_objects(img) if 'bottle' in obj_list: # Get the coordinates of the bottle in the image bottle_idx = obj_list.index('bottle')bbox = obj_locs[bottle_idx] # Compute the center of the bounding box center_x = (bbox[0] + bbox[2]) / 2 center_y = (bbox[1] + bbox[3]) / Print the center coordinates of the bottle print(f"Found the bottle at ({center_x}, {center_y})") Great!Now let's say we did find the bottle.Now I want to move towards it.The only information we have is the location of the bottle bounding box in the image.Can you give me code to make this happen?... (model comments omitted for brevity) ... '''prior code omitted for brevity''' center_x = (bbox[0] + bbox[2]) / 2 center_y = (bbox[1] + bbox[3]) / 2 # Compute the distance and angle to the center of the bounding box img_width, img_height = img.sizeangle = math.atan2(center_x-img_width / 2, img_height / 2 -center_y) distance = (center_x -img_width / 2) / (img_width / 2) if abs(distance) > 0.05: if distance > 0: # Bottle is to the right, so turn right and move forward turn_right() forward() else: # Bottle is to the left, so turn left and move forward turn_left() forward() '''later code omitted for brevity''' I want you to modularize this code a bit.Why don't you try coming up with a few high level functions and use them?Good idea!Here's a modularized version of the code that uses some helper functions: def get_object_distance_and_angle(img, depth_img, obj_name): #Code omitted for brevity def turn_towards_angle(angle): #Code omitted for brevity def search_for_object_and_move_to_it(obj_name): #Code omitted for brevity Now, let's assume you have access to all these functions you just created.Let's get to work solving some tasks.Find out where I can warm up my lunch.Sure, to find out where you can warm up your lunch, we can use the search_for_object_and_move_to_it function to search for a microwave in the scene.Here's an example code snippet: # Search for a microwave search_for_object_and_move_to_it("microwave")

Figure 6 :
Figure 6: Promptcraft open-sourced repository.Researchers can upload and vote on examples of LLM prompts for various robotics categories.
time.sleep(0.01)# Start the balancing process balance_ball()This code uses a loop to continuously check the position of the ball using the get_ball_position function.If the ball is off-center, the code calculates the necessary pitch and roll adjustments to center the ball and sets the plate angles using the set_plate_pitch and set_plate_roll functions.The code limits the pitch and roll adjustments to the maximum range of [-0.1, 0.1].The loop waits for a short period before checking the ball's position again to prevent over-adjusting the plate.
The objects available around are: fridge, bowl, pan, oil, stove The main functions you can use are: locate_object(object_name): Returns the XYZ coordinates of an object of interest.go_to_location(object_name): Moves robot to a location specified by XYZ coordinates.Returns nothing.pick_up(object_name): Picks up the object of interest.Returns nothing.use_item(object_name): Takes the name of an object as input.Performs an appropriate action on the object at hand (pour oil, crack eggs).Returns nothing.