Autonomous Quadrotor Control Library (AQC)

Repository developed by Nick Hinke (nhinke1) for EN.530.707 Robot System Programming (Spring '22)

README Contents

1. Description
2. Demo Videos
3. Software Description
   1. Dependencies
   2. Testing Installations
   3. What is an Input Client?
   4. Using an Existing Input Client
   5. Writing a New Input Client
   6. Full SITL Simulation
4. Hardware Description
   1. Requirements
   2. Running AQC on Physical Hardware
5. Future Work
6. Brief Overview of Each Package
7. Personal Remarks

Description

The Autonomous Quadrotor Control Library (henceforth known as AQC), was created with the intention of making software development for autonomous quadrotors easier. AQC currently only targets the PX4-Autopilot software, but could be extended to additionally support Ardupilot. Additionally, despite its name, AQC supports any flavor of multicopter supported by PX4.

At its core, AQC exists to serve two main purposes:

1. Provide a framework within which to easily (and quickly!) develop deployable software that targets real, physical hardware.
2. Provide an interface that is agnostic to the output so that all of your code can remain exactly the same (with the exception of a single launch file) when switching between simulation and physical hardware.

To accomplish these objectives, AQC implements a variety of features that will help speed up your development of any autonomnous behaviors, including:

• A variety of controllers to which vehicle setpoints can be published (currently supports both position and velocity setpoints)
• A variety of servers capable of changing the state of the vehicle (e.g. arm/disarm the vehicle, change vehicle flight mode)
• A one-stop shop to retrieve useful information regarding the state of the vehicle
• An extensive set of chained launch files that allow you to launch everything you need for a full Gazebo simulation (or with real hardware!) with just one file
• Support for any type of input that you want to provide (see the Future Work section)

In order to take advantage of these features, one need only launch the AQC driver (with your desired arguments set accordingly) on the appropriate device (aqc_driver.launch for physical hardware, or aqc_sim.launch for simulation). This will launch everything required for AQC to run as a "server side" for any autonomous behaviors, and will remain entirely consistent in its functionality across both physical hardware and Gazebo simulations. From there, you can either use one of the provided example input clients (see the Using an Existing Input Client section) to interact with AQC and control the vehicle, or define your own for your application (see the Writing a New Input Client section).

Before jumping into more details, it is first necessary to make a quick disclaimer. Developing autonomous behaviors for aerial vehicles is inherently dangerous. There is certainly a tradeoff between flexibility of features and implemented safety measures, and AQC has not been rigorously tested at a production level. With that in mind, please perform any experiments with physical hardware at your own risk.

It should also be noted that AQC was created using ROS Melodic on Ubuntu 18.0.4 LTS, and its formatting was partially inspired by the ROS Industrial universal robot meta-package.

Demo Videos

See YouTube video descriptions at the links below for more information regarding each demo video.

1. Demo1 - brief hardware tour of physical hardware used to test AQC
2. Demo2 - demonstration of using one of the provided AQC input clients to publish position setpoints (the highest level of control available) from a laptop ground control station (GCS) to the AQC driver running on the vehicle's companion computer

Software

Software Dependencies

1. ROS Melodic running on Ubuntu 18.04 LTS

2. QGroundControl

This is a ground control software that enables communication between a "ground control station" (GCS) and the flight controller unit (FCU). It also provides an easy interface to update firmware and/or configuration parameters of the FCU, and an extremely helpful GUI that displays a lot of information about the vehicle during flight (both in real life and in simulation). Note that the latest stable version of QGC only officially supports Ubuntu 20.0.4 and later due to video streaming issues, but I have had experienced no errors thus far. If you do encounter problems, older stable versions can be downloaded here.

To install:

sudo usermod -a -G dialout $USER
sudo apt-get remove modemmanager -y
sudo apt install gstreamer1.0-plugins-bad gstreamer1.0-libav gstreamer1.0-gl -y
sudo apt install libqt5gui5 -y

Log out and log back in for user permissions to change
Download QGroundControl.AppImage

To run:

chmod +x ./QGroundControl.AppImage
./QGroundControl.AppImage

3. PX4-Autopilot

This is the autopilot software that will be running on the FCU with real hardware, and that will be running "in-the-loop" during simulation. Alternatively, you could use Ardupilot, but that will not be compatible with this repository.

To install:

git clone https://github.com/PX4/PX4-Autopilot.git --recursive

4. mavros

In short, this provides the necessary bridge for communication between ROS and the Pixhawk. In slightly more detail, the PX4-Autopilot software running on the Pixhawk communicates using the Mavlink protocol commonly used on drones (although I believe it is now capable of communicating with ROS2 directly). Mavros simply provides the bridge from any ROS communications to Mavlink communications.

To install:

sudo apt-get install ros-melodic-mavros ros-melodic-mavros-extras

5. GeographicLibDatasets

These are geographic datasets that are utilized by the ground control software.

To install:

wget https://raw.githubusercontent.com/mavlink/mavros/master/mavros/scripts/install_geographiclib_datasets.sh
sudo bash ./install_geographiclib_datasets.sh

6. AQC!

To install:

source /opt/ros/melodic/setup.bash
mkdir -p catkin_ws/src && cd catkin_ws
git clone https://github.com/nhinke/rsp-project-repo.git src/aqc-repo
catkin build
source devel/setup.bash

See next section here regarding how to test to ensure that everything was installed and set up correctly.

Testing Installations

After doing all that work to install the dependencies and set up your environment, it would be certainly be a good idea to make sure it all works! If everything built correctly, you're already off to a great start. To truly test if everything is installed correctly, you can launch a slightly simplified version of the simulation that allows for inputs through the terminal window in which it was launched.

To launch:

# assuming you are starting in the catkin_ws where aqc_repo was just built
chmod +x src/aqc-repo/scripts/test-sim-env.sh
./src/aqc-repo/scripts/test-sim-env.sh

Running this script should open QGroundControl and Gazebo, and you should see something akin to the following with the vehicle sitting in the middle of Decker Quad at JHU:

As long as you ran the script in the foreground, hitting enter in the same terminal window from which you ran it should provide you with a command line interface designated by pxh> . This should allow you to control the vehicle in the simulation, which you can try by using commands such as commander arm and commander takeoff. For a full list of the available commands, simply enter commander help.

Note that the script used to launch this test simulation references this script which is used to launch all simulations from within AQC. Notably, this script uses the shell command locate to find your installations of QGroundControl and PX4-Autopilot. If you find that this takes too long to launch the simulations, simply define the appropriate installations paths in that script.

What is an Input Client?

As previously mentioned, the typical use case for AQC will first involve launching the aqc_driver (either the physical hardware version at aqc_driver.launch or the simulation version at aqc_sim.launch) to act as a server side running on the vehicle. Indeed, running this driver is what provides all of the functionality described above. After launching the driver, all of the necessary communications bridges have been made to the PX4 software stack, and so the system is just waiting for some good input commands. In order to provide these commands in a standardized way, we define the notion of an "input client" which is responsible for communicating all commands to the aqc_driver. While it is possible (and encouraged!) to define your own input clients specific to your application as will be discussed briefly here and here, there are several pre-existing input clients that are ready for you to use right away. They can all be found in the directory titled example_aqc_input_clients.

The reason for keeping these input clients separate from the rest of the AQC library is because, frankly, they do not belong in here. This is because for most applications, the companion computer onboard the vehicle will manage the aqc_driver and the ROS Master, but will never utilize one of these inputs clients (rather, they will only receive commands from them via pre-defined communication channels). Consequently, there is no reason to waste any memory on a resource-constrained companion computer, and they are only included in this repo for illustrative purposes. If you wish to use them, copy them into a different repository specific to your application and work with them there. That being said, however, if you wish to simulate both the input client and the aqc_driver on the same machine during development, it may make sense to keep the input clients here until it comes time to deploy your software onto physical hardware.

How to Use an Existing Input Client

Using one of the pre-defined input clients is very straightforward, both in simulation and when using physical hardware (since not a single line of code in the input client needs to change!). Currently, there are three fully-implemented input clients provided here, and one other work-in-progress. The three working clients are:

1. aqc_input_raw_twists

   • Publishes arm and disarm commands
   • Publishes commands to change the vehicle flight mode
   • Publishes velocity setpoints (x, y, z, and yaw) to the aqc_driver in the form of a twist
   • Subscribes to a ROS topic ("/input_cmd_twist" by default) to receive a custom message from the user which defines each of the three commands above, which can be published by first typing rostopic pub -1 /input_cmd_twist and then using tab completion to fill out the rest of the message

2. aqc_input_dynamic_reconfigure_vel

   • Publishes arm and disarm commands
   • Publishes commands to change the vehicle flight mode
   • Publishes velocity setpoints (x, y, z, and yaw) to the aqc_driver in the form of a twist
   • Uses ROS dynamic reconfigure to receive inputs from the user which define each of the three commands above

3. aqc_input_dynamic_reconfigure_pos

   • Publishes arm and disarm commands
   • Publishes commands to change the vehicle flight mode
   • Publishes position setpoints (x, y, z, and yaw) to the aqc_driver in the form of a custom defined message
   • Uses ROS dynamic reconfigure to receive inputs from the user which define each of the three commands above

Each of these input clients comes complete with a launch file of the form aqc_input_*.launch containing several useful arguments including the rate (in Hz) at which the input client operates as well as the names of all of the relevant topics. As such, all that is required to use one of these input clients is to launch the appropriate launch file with any arguments that you wish to specify! Then, assuming the aqc_driver is already running on the vehicle or in simulation with the appropriate controllers enabled, you'll be completely set up to send commands to the vehicle at will!

It is worth reiterating that all of the commands being published by the input clients described above are being received, processed, and appropriately forwarded in a standardized way to the vehicle's FCU by the aqc_driver. This demonstrates the real power of AQC, since now a user need only be concerned with the higher level questions like what kinds of setpoints should be generated for their application or even how to collect user input (see here for more on this); indeed, this is due to how easy it is to define one of these input clients within this standardized framework, and crucially, to the ability of the aqc_driver to execute your commands without you needing any knowledge of how it's actually happening. As such, gone are the days when you have a great autonomous behavior in mind for your application, but cannot get the FCU to listen to your commands (or even establish reliable communication with it).

How to Write a New Input Client

At their fundamental level, any input client implementation must do only three things:

1. Communicate arm and disarm commands to the aqc_driver
2. Communicate change flight mode commands to the aqc_driver
3. Communicate motion setpoints (e.g. position, velocity, acceleration, etc.) to the aqc_driver

Considering a slightly more detailed view, both of the first two commands are handled by actionlib servers running within the aqc_coordinator node, which is one of the most important nodes operating within aqc_driver. The quickest way to handle the first two commands within your input client would likely be to simply copy the code in the provided examples corresponding to these actionlib servers (ArmAction and ChangeFlightModeAction, respectively).

After very easily completing the first two required tasks, the third one (regarding motion setpoints) requires a little more care. Currently, only two types of controller have been fully implemented, namely using east-north-up (ENU) absolute position setpoints (plus yaw) or ENU velocity setpoints (plus yaw rate). These controllers operate within the aqc_driver, and their implementations can be found here and here, respectively. Both of these controllers listen for published setpoints on ROS topics ("/cmd_position" and "/cmd_vel" by default).

Consequently, to communicate motion setpoints to the appropriate controllers, your input client need only publish a message containing your setpoint on the appropriate topic. The controller will then handle the rest. You do have to be careful, however, to ensure that your controller was enabled when launched the aqc_driver (using either aqc_driver.launch or aqc_sim.launch. Enabling and disabling controllers--as well as remapping the topics they subscribe to--can all be done through arguments passed to one of these two launch files. As a result, after completely defining your own new input client, the only thing you have to do to ensure that the aqc_driver will be able to process your commands is to correctly set the arguments to a single launch file. Thus, it has once again been shown (and is demonstrated in Demo2 on physical hardware), that AQC can completely control the vehicle just by appropriately launching two launch files: one for the aqc_driver and the other for the input client.

Configuring and Running SITL Simulation

TL/DR:

Each provided input client has an "end-to-end" (e2e) launch file entitled "demo_sim_e2e.launch" for testing (found here, here, and here). To get started running your first full SITL simulation (the PX4 software stack is considered "in-the-loop" here), simply launch any one of these files (and give it a second to find and load everything); for example:

# assuming you are starting in the catkin_ws where aqc_repo was built
source devel/setup.bash
roslaunch aqc_input_dynamic_reconfigure_vel demo_sim_e2e.launch

Note that you will have to arm the vehicle before it is able to takeoff and fly in any flight mode. Typically, the workflow entails: the vehicle is first armed and taken off to a safe altitude, the vehicle then enters hold mode to hold its current state and wait for further instruction, then (after ensuring that an appropriate input client is alive and running) the vehicle is switched into offboard flight mode in order to test the input client.

Considering more detail:

It should be mentioned that since the goal of AQC is to speed up and simplify the development of autonomous behaviors for multicopters, the only real priority when simulating the system was to provide a simulation environment that interfaces with your code in exactly the same way as the aqc_driver for physical hardware. As such, only a very basic simulation was utilized for first testing the functionality of the aqc_driver and subsequently for testing the performance of the input clients. That being said, it would be very straightforward to jazz up the simulation quickly to fit the needs of your application (e.g. using a world with lots of obstacles if your application requires obstacle avoidance capabilities). Additionally, if it is desired to simulate the robustness of your implementation to uncertainty or noise for your application (e.g. PID trajectory tracking controller performance, active disturbance rejection at high altitudes, etc.), a variety of parameters could be introduced to the simulation via Gazebo Plugins or otherwise including wind, sensor noise, gps errors, etc. PX4 even provides a good resource here with some ideas and implementations for simulating such things. All that being said, however, the priority here was simply to prove the functionality of AQC, so only a rather minimalist simulation enviroment was required.

Recall that when considering running AQC on physical hardware, the typical use case will almost always involve the aqc_driver running on the vehicle's companion computer, whereas the input client will be running on a laptop/GCS. As such, since the two sides of the system are divided as server (aqc_driver) and client (input_client), ideally they should be simulated as such. In that vein, there is a single launch file for launching the aqc_driver in simulation called aqc_sim.launch. When launching this file, you can pass it many arugments including the controllers to be enabled and their frequency, the names of the relevant topics, and even the simulated FCU url if your application requires the use of one other than the default set by PX4 here. Thus, once you know which input client you would like to test (i.e. the controllers it requires and the topics to which it will be publishing setpoints), you can launch just the single aqc_sim.launch file to begin the simulation in Gazebo. Note that this file actually launches Gazebo and QGroundControl using the same script as used when testing the installations.

All that being said, in order to speed up development times, you're going to want to just launch a single file to test your code since both the aqc_driver and the input client will (very likely) be running on the same machine. To do so, this single launch file should only need to launch the input client you wish to test along with aqc_sim.launch (and likely some shared arguments between the two). As a new combined launch file such as this would be required for every new input client implementation, it is recommended that a launch file of this nature be included under the launch directory within the test client package. Three such examples of these combined launch files exist within the example input clients directory, one for each of the provided input clients, namely:

1. demo_sim_e2e.launch for aqc_input_raw_twists
2. demo_sim_e2e.launch for aqc_input_dynamic_reconfigure_vel
3. demo_sim_e2e.launch for aqc_input_dynamic_reconfigure_pos

Each of these files has several arguments that you can pass to it during launch, including the controller rate in aqc_driver, the input command rate, relevant topic names, etc. If you look closely at each of the files, you will notice that they all simply launch their target input client in addition to launching aqc_sim.launch with the approrpriate arguments. As such, this strucutre of launch file would be applicable to any input client that you define.

To try one out:

# assuming you are starting in the catkin_ws where aqc_repo was built
source devel/setup.bash
roslaunch aqc_input_dynamic_reconfigure_vel demo_sim_e2e.launch

Note that you should not need to pass any arguments to any of the launch files for them to work. Additionally, while each of the three launch files have very similar arguments that can be passed to it, the demo_sim_e2e.launch for aqc_input_dynamic_reconfigure_pos notably has an additional argument entitled "use_relative_xy_setpoints" (which takes a bool value, and is false by default) that converts the input commands (that were set using the rqt_reconfigure_gui) for the east (x) and north (y) positions into relative setpoints, while keeping the up (z) position and yaw angle absolute for safety reasons. The planar (x,y) position referenced as (0,0) in this relative mode is set to the planar position of the vehicle when offboard mode was first launched. For this reason, if you were to drive the vehicle to the planar position (5,5) using the input client with the vehicle in offboard mode, then switch the vehicle to hold flight mode for some reason, and then switch the vehicle back into offboard mode before resetting the sliders in the rqt_reconfigure_window to 0, the vehicle would immediately begin travelling to planar position (10,10).

To try this out:

# assuming you are starting in the catkin_ws where aqc_repo was built
source devel/setup.bash
roslaunch aqc_input_dynamic_reconfigure_pos demo_sim_e2e.launch use_relative_xy_setpoints:=true

Hardware

TODO finish fleshing out the remaining sections
This section will focus exclusively on the hardware setup that I used for testing this library; however, there are infinitely many other hardware combinations (e.g. choice of airframe, companion computer, FCU, radios, etc.) that would be supported by this library. For reference, there are some images below of my personal hardware setup.

Hardware Requirements

Basic requirements including what I used:

1. Companion Computer -- NVIDIA Jetson Nano 2GB
2. Multicopter -- Holybro x500 v2 Quadrotor
3. FCU -- Holybro Pixhawk5x
4. GPS -- Holyrbo M9N
5. Telemetry radios -- mRobotics 915 Mhz
6. RC receiver/transmitter -- FlySky FS-i6
7. UART Connection -- USB-to-TTL for reliable communication between FCU and Jetson
8. Wifi adapter -- 802.11ac wifi adapter to allow the Jetson to create its own network to which you can connect the GCS
9. Power -- LiPo battery with appropriate power distribution modules, and a USB-C power bank to power the Jetson

Configuring and Running AQC on Hardware

Steps required for my hardware implementation:

1. Connect Jetson to WiFi connection with internet, then pull AQC repository from GitHub and build
2. Connect Jetson to its own (hidden) WiFi network (without internet) and note its IP address
3. Connect machine that will be running the input client to hidden Jetson WiFi network and note its IP address (referred to as jetson-ip)
4. In ~/.bashrc on GCS machine, set ROS_IP to its own IP address and set ROS_MASTER_URI to http://jetson-ip:11311/ and resource it
5. Power vehicle and ensure it connects to GCS machine (in QGC) using telemtry radio connection
6. ssh into Jetson using its IP address
7. Run the terminal as a root user using the command sudo -s in order to obtain access to the serial ports (for mavros)
8. cd into the AQC workspace on the Jetson and source it
9. Launch aqc_driver on the Jetson using the command roslaunch aqc_driver aqc_driver.launch with any other desired arguments
10. Launch appopriate input client on GCS machine
11. Have fun (and be safe!) controlling the vehicle in offboard mode!

Future Work

• extend aqc_driver to support more controllers (and therefore, allow more types of input from the input clients)
• develop really cool/interesting input clients for niche applications
  • e.g. a bluetooth headset connect to a Raspberry Pi that could be carried around in your backpack while acting as the GCS machine, where the Raspberry Pi running a automatic speech recognition neural network like wav2vec (likely with additional hardware like a TPU accelerator to reduce latency) to extract setpoint commands from speech
  • e.g. running input client in addition to aqc_driver on companion computer with depth camera (would likely required more powerful device such as NVIDIA Jetson Xavier) to track and follow another moving multicopter

Brief Overview of each Package

In the order in which they are listed in the repository:

1. aqc_controllers - this is a meta-package which contains all controllers that are used within aqc_driver
   1. aqc_controller_pid - INCOMPLETE, this package will be used to define a PID position controller that uses velocity setpoint commands, but it is currently incomplete
   2. aqc_pos_controller_px4 - this package contains a position controller which communicates position setpoints to the FCU (which is the highest available method of control input supported by PX4) and can be configured in either absolute position mode or relative position mode
   3. aqc_pos_contoller_rtt - INCOMPLETE, this package was going to be used to define position setpoints using [Orocos RTT], but was put on the back burner given that many companion computer may struggle to run this anyway
   4. aqc_vel_controller_raw_twists - this package contains a velocity controller which communicates velocity setpoints to the FCU
2. aqc_coordinator - this package is primarily responsible for handling any arm requests or change flight mode requests coming from the input client, but it also publishes a lot of useful information regarding the state of the vehicle within the "/aqc/" namespace that input clients may rely on
3. aqc_driver - this meta-package provides a very easy way to launch all of the apporpriate nodes/packages on the vehicle's companion computer (or within a Gazebo simulation) through just a single launch file
4. aqc_mode_manager - this package provides a bridge between the aqc_coordinator package and mavros (for communicating arm requests and change flight mode request to the FCU); this was done to simplify the development of any future input clients by ensuring that you can get and set everything you want/need through the aqc_coordinator package while keeping it as lightweight and easy to understand as possible
5. aqc_msgs - this package contains all of the custom .msg, .src, and .action files used throughout AQC
6. doc - this is not a package, just a directory containing the images used in this README
7. example_aqc_input_clients - this is a meta-package which contains all of the provided AQC input clients
   1. aqc_input_dynamic_reconfigure_pos - this package contains an input client which generates position setpoints (and support either relative or absolute positions) for the vehicle using an rqt_reconfigure_window
   2. aqc_input_dynamic_reconfigure_vel - this package contains an input client which generates velocity setpoints for the vehicle using an rqt_reconfigure_window
   3. aqc_input_raw_twists - this package contains an input client which generates velocity setpoints for the vehicle by subrscribing to a topic that a user can publish commands to (see here for how to do this)
   4. aqc_input_teleop_twist - INCOMPLETE, this package contains a partially implemented input client based on a c++ implementation of the ROS teleop_twist_keyboard which will be used to generate velocity setpoints for the vehicle through keybindings
8. scripts - this is not a package, just contains a single script for testing your dependency installations as described here

Personal Remarks

I had a ton of fun working on this project. I actually had a lot of the hardware (including Jetson, x500, etc.) after starting to get into this stuff last Summer (2021), but I never had a chance to even set up everything once school started again last Fall. I will absolutely continue to work on this library, as I have a lot of ideas for future projects that would be benefitted greatly by using AQC. Also, I wrote this README as if other people are going to come along and use this repository; I do not expect that to happen (at least, certainly not anytime soon), but it was good practice and I honestly had fun doing it. If you are someone else here to use this repo, however, cheers to you! Anyways, I would like to quickly thank you to Prof. Leonard for providing me with the tools necessary to actually pull this off. Happy flying!