Hybrid automata-guided multi-agent TD3 for UAV swarm decision-making in adversarial environments

Abstract

UAV swarms must coordinate under partial observability, intermittent communication, and adversarial electronic-warfare (EW) interference. End-to-end multi-agent reinforcement learning (MARL) such as MADDPG can learn cooperative behaviors, but monolithic policies are brittle across mission phases and during rare safety-critical events such as sudden jamming. Building on a previously published hybrid framework that couples a supervisory hybrid automaton (HA) with three discrete modes (Search, Track, Safe) and guarded transitions with MARL under centralized training and decentralized execution (CTDE), this work replaces the MADDPG controller with multi-agent Twin Delayed Deep Deterministic Policy Gradient (MATD3). The HA enforces mission-level logic and safety while MATD3 optimizes continuous actions within each mode. In a Gym-PyBullet-Drones testbed with an EW jammer we compare HA+MATD3 against HA+MADDPG under identical HA logic. HA+MATD3 reaches the same final return about 30% sooner and more smoothly, attains success rates of 96% (no jam) and 93% (jam), searches faster (22.3 s vs 24.0 s), and tracks evasive targets more reliably (92.5% vs 89.7% in-view). It also scales more gracefully to larger swarms, retaining 87% mission success under jamming with ten UAVs versus 82% for the baseline controller. The results indicate that pairing formal mode supervision with a twin-critic MARL algorithm yields more robust, sample-efficient swarm control than the baseline.

Keywords: multi-agent reinforcement learning, UAV swarm, hybrid automata, TD3, MADDPG, electronic warfare, CTDE.