Cover of Zhejiang University Research Summit Issue: Micro aerial robot clusters like birds freely passing through dense forests

More than 10 miniature aerial robots the size of palms and the weight of a single machine is less than a cola, rising in a bamboo forest in Anji, Zhejiang Province, without GPS, motion capture systems, remote communication operations or pre-established environmental maps, each independently and cooperatively through low shrubs, sloping bamboo poles, undulating ground, dense branches, like a cooperative but autonomous flock of birds.

This is one of the latest research results of Gao Fei and Xu Chao's team in the School of Control Science and Engineering of Zhejiang University on the fully autonomous micro-aerial robot. The team developed miniature and fully autonomous aerial robots capable of flying in clusters based on limited information provided by airborne sensors in highly chaotic field environments. The trajectory planning algorithm on the robot can independently plan the flight path, and at the same time meet the requirements of flight efficiency, obstacle avoidance, mutual collision avoidance between robots, dynamic feasibility, and cluster coordination.

On May 4, local time, the above results were titled "Swarm of Micro Flying Robots in the Wild" and published as a cover article in science robotics, the international top journal in the field of robotics. The first author of the paper is Zhou Xin, a doctoral student at the School of Control of Zhejiang University, and the corresponding authors are Dr. Gao Fei and Professor Xu Chao.

In an interview with the surging news (www.thepaper.cn) reporter, Gao Fei, doctoral supervisor of the School of Control of Zhejiang University, said that this achievement will lay the foundation for the future application of micro-aerial robot clusters and related algorithms in outdoor disaster relief, field biological research, collaborative transportation and other fields.

As a symbol of future technology, the flexibility of multi-robot systems in the air has been presented in many science fiction movies. In Prometheus, astronauts release several miniature flying devices to explore an unknown alien spacecraft before deciding which way to go. In Ender's Game, the aerial robot swarm system surrounds the spaceship, forming a shield against alien attacks, clearing a path for humans to win battles. In Star Wars Prequel III and Blade Runner 2049, air traffic between skyscrapers operates in a busy and orderly manner on a high-tech planet.

With the development of processor computing power, perception and communication, aerial robots such as quadcopters have entered the public life, which are not only highly maneuverable, but also inexpensive. According to the Nihon Keizai Shimbun, DJI's Mavic Air 2 is currently one of the best-selling aerial robots, which has functions such as obstacle avoidance, tracking, and has a communication distance of 10 kilometers, and its hardware cost is only about $135. In addition, there are still a lot of possibilities in the aerial robot market, and by 2028, the overall market value of aerial robots is expected to reach $500 billion.

However, although autonomous navigation by aerial robots has developed rapidly in both industrial and academic practices, highly chaotic environments, such as dense forests, are still difficult for aerial robots to enter and travel freely, not to mention swarms of aerial robots. In these scenarios, unknown complex environments and narrow feasible spaces can pose great challenges to the coordination of aerial robot clusters.

"In the research of aerial robots, the shift from a single robot to a collaborative cluster system is also an inevitable trend in the development of robot intelligence." Gao Fei introduced himself to the surging news reporter. Aerial robotics was Goofy's main research area when he was pursuing his Ph.D. at the Hong Kong University of Science and Technology, when his research direction was mainly airborne robot single-machine navigation. At the end of 2019, Gao Fei went to Zhejiang University to take up a position and began to lead the team to carry out research on autonomous navigation and cluster technology of aerial robots, and the team prepared for the field aerial robot cluster paper for about two years.

It is understood that the reason why it is difficult to navigate the aerial robot cluster in the chaotic field environment is because the field environment puts forward four requirements for the aerial robot cluster, referred to as TEEM - trajectory optimality, extensibility( scalability), low-cost computing requirements (economical computing) and mini size (miniature size).

Trajectory optimization reflects mission quality and flight time, an efficiency that is critical in emergency rescue situations. Trajectory optimization requires a variety of complex environments that allow aerial robots to fly and traverse quickly, safely and stably, a performance that is critical in emergency rescue situations and chaotic and narrow environments.

Scalability refers to the expansion of the needs of aerial robots for different tasks in software and hardware, such as multi-machine collaborative target tracking tasks, which need to add recognition functions in software and expand multi-faceted lenses on hardware.

The need for low computing costs is important, which allows aerial robots to carry smaller onboard processors and reduces response times to changing environments and unexpected situations, thus reserving as many available computing resources as possible for other user-defined tasks such as object detection and decision-making.

Finally, all of these capabilities should be placed in the smallest aircraft, because weight and volume are directly related to the robot's endurance and ability to traverse narrow spaces.

However, the requirements of these four aspects are contradictory, and precise trade-offs are required to be achieved at the same time. For example, higher trajectory optimization requires more iterations in complex modeling and solution spaces, which can significantly increase computational time and computational cost. Greater scalability requires problems and tasks to be defined in a more general form, at the expense of potential task-specific optimizations.

"Meeting safety, kinetic feasibility, minimizing time and maximizing trajectory smoothness is already a huge challenge for aerial robot clusters, and even more difficult to implement on micro platforms. That's why previous research has been unable to move from a structured, man-made environment to an uncertain wild environment. The paper said.

In the real world, Intel, High Great, and CollMot have already demonstrated impressive aerial robot formation performances. However, behind the large-scale and successful commercial uses, formations of aerial robots positioned using GNSS follow only pre-set trajectories and cannot operate in obstructed field locations.

To solve this problem, robotics researchers have tried to find inspiration from nature, that is, to observe how nature responds to this navigation challenge. In nature, insects perform short-term reactions, while birds prefer relatively long-term smooth movements. This is because birds have sharper visual and motor sensations, a more free motor system, and a larger brain capacity than insects.

The two major flying species of insects and birds have also inspired two mainstream aerial robot navigation methods: insect-based response and bird trajectory-based planning methods. Of the two approaches, the former includes an extremely lightweight and efficient solution in terms of compute and memory, allowing for lighter clusters of aerial robots, while the latter shows greater optimization and flexibility. To improve mission efficiency and scalability, Goofy's team chose the latter.

After studying various aerial robot cluster applications, the team found that the key to solving the "TEEM" problem is the robot trajectory spatiotemporal planning, that is, not only the ability to change the shape of the trajectory, but also to adjust the time distribution to maximize the use of space. If only space deformation is carried out, the aerial robot will often detour and wait for other robots when passing through the narrow passage, which will hinder the flight of subsequent aerial robots, resulting in a poor or even unsafe flight trajectory. Therefore, simultaneous planning of the shape and time of the flight trajectory, also known as spatio-temporal trajectory planning, is the key to the safe and efficient flight of aerial robots. Still, this joint optimization is a huge challenge for multi-rotor vehicles, as the spatial and temporal parameters that jointly determine the trajectory are highly coupled.

"Realize joint spatio-temporal trajectory planning to ensure that every robot in the robot cluster can achieve the optimal spatio-temporal trajectory." This is undoubtedly the most difficult in the study. Goofy said.

In its proposed method, the team realized the linear complexity mapping between the optimization variable and the intermediate variable representing the trajectory by decoupling the space-time parameters in the objective function calculation, thus realizing the space-time optimization. As a result, even in the most restricted environments, aerial robots can obtain high-quality trajectories in just a few milliseconds.

In particular, in the team's cluster of aerial robots, each aerial robot has complete sensing, positioning, planning and control functions, and shares trajectories through broadcast networks to achieve group collaboration. This is similar to the ability of birds to fly freely in the forest while avoiding obstacles and other moving creatures. For example, in short-range navigation, birds rely primarily on the eye and vestibular systems, and accordingly, the team developed an improved visual-inertial odometry. In addition, birds can adjust their path and speed at the same time to avoid collisions, while considering flight time and smoothness to save energy, so the team proposed a joint optimization method for multi-objective spatio-temporal trajectories. In addition to the capabilities of small birds, the team further leveraged the advantages of robotics, an artificial electronic system, to use high-fidelity wireless communication for motion trajectory sharing and high-speed computing for rapid planning. In addition, the team's solution satisfies the distributed coordination of individual intelligence and swarm intelligence, improving the robustness of the system.

At present, the team has released cutting-edge algorithm software for aerial robot cluster research in the paper, "developers can deploy and use these software to validate their algorithms from the simulation environment, and in the future, these algorithms can be applied to logistics trolleys, robot distribution, field search and rescue and other fields and environments." Goofy said.

It is worth mentioning that in the paper, the team highlighted four challenging applications through experiments in the real world. On the same day as the paper was published, the Goofy team released a video of these four experimental scenes on the well-known domestic video website Bilibili, and as of the time of writing, it has received 56,000 views.

Fly over dense forests

Challenging wild navigation with bamboos and various other obstacles.

This experiment aims to show that in a highly dense wild environment, that is, in a bamboo forest, aerial robot clusters can achieve fully autonomous group navigation without harming the robot body or plants. The trajectories presented in the image show the significant advantages of trajectory planning: the planned trajectories are always directly and smoothly connected to each gap.

In these environments, in addition to vertically growing bamboo, there are also sloping bamboo, tree trunks, low bushes, overgrown ditches, uneven ground, windblown leaves and other obstacles, which require systematic three-dimensional planning of the trajectory. This unstructured environment of irregularly shaped, densely distributed obstacles validates the ability of clusters of aerial robots to navigate most chaotic places, such as disaster scenarios.

Formation navigation in the wild

Swarm navigation in formation with prior-unknown obstacles.

This experiment demonstrates the scalability of the proposed unified trajectory planning. Here, the formation is defined as maintaining a desired moving shape, which means that the aerial robot moves in a fixed relative position. At the same time, each aerial robot can also navigate independently, avoiding obstacles. In this experiment, the density of obstacles was reduced to make the formation clearly recognizable compared to the "Bamboo Shuttle" experiment, but there were still bushes, high and low trees, and two artificial iron pillars.

Following the planned trajectory, aerial robot clusters fly through the woods in fixed formations. As can be seen from the deformation curve and speed curve, although aerial robots sometimes have to deviate from the route to avoid obstacles that are unknown in advance, they will speed up again after this to catch up with the formation, so that the swarm remains in formation. The average speed automatically decreases when aerial robots avoid trees, and increases when they fully return to open space. In this case, some individual velocity changes can also be propagated throughout the formation without explicit preprogramming. This result shows a hidden balance between safety and flight time, with deceleration near obstacles retaining more reaction time to potential collisions, while accelerating as much as possible reduces flight time in open areas.

Intensive mutual obstacle avoidance assessment

Evaluation of intensive reciprocal collision avoidance with unexpected events

The setup of the experiment simulated the most basic requirements for dense air traffic between skyscrapers: safe, efficient, and independent navigation. To test this capability of 10 aerial robots, the experimenters randomly assigned the target position to the aerial robot on a circle with a radius of 3 meters. In order to better simulate the real flight situation, in addition to the dense tree trunks and tripods of the camera, the flight area also simulates new buildings with box and cylindrical obstacles, and the aerial robot cluster also needs to pass through large moving obstacles in the area. Next, the experimenter closes all ground positioning tabs (used only in this experiment) to simulate a temporary loss of global positioning.

Since safety and efficiency are the two main concerns of the transport system, the researchers evaluated the minimum collision distance and the total number of deliveries completed in the 3-minute flight (the total number reached), and throughout the flight, the researchers modeled each aerial robot as a sphere with a radius of 7cm. A single aerial robot manages to maintain a safe distance from obstacles and other robots. Experiments have shown that the number of targets reached increases linearly over time. Experimental results show that at different obstacle densities, since the planned trajectory is locally optimal, a nearly constant growth rate can be obtained.

Aerial robot swarms in the case of target occlusion

Multidrone tracking with target occlusion.

This experiment demonstrates the potential to add computationally intensive hardware and run additional tasks on micro platforms. Cluster tracking can be used for multi-view aerial photography and video, allowing for a comprehensive recording of participants and more footage for post-editing. In the experiment, the focus was on a human participant moving through the woods. To track targets while avoiding obstacles and other aerial robots, the experiments designed constrained penalties for tracking to plan the desired trajectory. From the results, it can be seen that the tracked person can move forward without worrying about the collision or drop of the robot in the air.