Untitled

Funder

Organizational Unit

Publications

Deep Q-Networks for Aerial Data Collection in Multi-UAV-Assisted Wireless Sensor Networks

Publication . Emami, Yousef; Wei, Bo; Li, Kai; Ni, Wei; Tovar, Eduardo

Unmanned Aerial Vehicles (UAVs) can collaborate to collect and relay data for ground sensors in remote and hostile areas. In multi-UAV-assisted wireless sensor networks (MA-WSN), the UAVs' movements impact on channel condition and can fail data transmission, this situation along with newly arrived data give rise to buffer overflows at the ground sensors. Thus, scheduling data transmission is of utmost importance in MA-WSN to reduce data packet losses resulting from buffer overflows and channel fading. In this paper, we investigate the optimal ground sensor selection at the UAVs to minimize data packet losses . The optimization problem is formulated as a multiagent Markov decision process, where network states consist of battery levels and data buffer lengths of the ground sensor, channel conditions, and waypoints of the UAV along the trajectory. In practice, an MA-WSN contains a large number of network states, while the up-to-date knowledge of the network states and other UAVs' sensor selection decisions is not available at each agent. We propose a Multi-UAV Deep Reinforcement Learning based Scheduling Algorithm (MUAIS) to minimize the data packet loss, where the UAVs learn the underlying patterns of the data and energy arrivals at all the ground sensors. Numerical results show that the proposed MUAIS achieves at least 46\% and 35\% lower packet loss than an optimal solution with single-UAV and an existing non-learning greedy algorithm, respectively.

2021-07-02Journal article

Open access

Data-driven Flight Control of Internet-of- Drones for Sensor Data Aggregation using Multi-agent Deep Reinforcement Learning

Publication . Li, Kai; Ni, Wei; Emami, Yousef; Dressler, Falko

Energy-harvesting-powered sensors are increasingly deployed beyond the reach of terrestrial gateways, where there is often no persistent power supply. Making use of the internet of drones (IoD) for data aggregation in such environments is a promising paradigm to enhance network scalability and connectivity. The flexibility of IoD and favorable line-of-sight connections between the drones and ground nodes are exploited to improve data reception at the drones. In this article, we discuss the challenges of online flight control of IoD, where data-driven neural networks can be tailored to design the trajectories and patrol speeds of the drones and their communication schedules, preventing buffer overflows at the ground nodes. In a small-scale IoD, a multi-agent deep reinforcement learning can be developed with long short-term memory to train the continuous flight control of IoD and data aggregation scheduling, where a joint action is generated for IoD via sharing the flight control decisions among the drones. In a large-scale IoD, sharing the flight control decisions in real-time can result in communication overheads and interference. In this case, deep reinforcement learning can be trained with the second-hand visiting experiences, where the drones learn the actions of each other based on historical scheduling records maintained at the ground nodes.

2022-05-06Journal article

Open access

Data-driven Deep Reinforcement Learning for Online Flight Resource Allocation in UAVaided Wireless Powered Sensor Networks

Publication . Li, Kai; Ni, Wei; Kurunathan, Harrison; Dressler, Falko

In wireless powered sensor networks (WPSN), data of ground sensors can be collected or relayed by an unmanned aerial vehicle (UAV) while the battery of the ground sensor can be charged via wireless power transfer. A key challenge of resource allocation in UAV-aided WPSN is to prevent battery drainage and buffer overflow of the ground sensors in the presence of highly dynamic lossy airborne channels which can result in packet reception errors. Moreover, state and action spaces of the resource allocation problem are large, which is hardly explored online. To address the challenges, a new data-driven deep reinforcement learning framework, DDRL-RA, is proposed to train flight resource allocation online so that the data packet loss is minimized. Due to time-varying airborne channels, DDRL-RA firstly leverages long short-term memory (LSTM) with pre-collected offline datasets for channel randomness predictions. Then, Deep Deterministic Policy Gradient (DDPG) is studied to control the flight trajectory of the UAV, and schedule the ground sensor to transmit data and harvest energy. To evaluate the performance of DDRL-RA, a UAV-ground sensor testbed is built, where real-world datasets of channel gains are collected. DDRL-RA is implemented on Tensorflow, and numerical results show that DDRL-RA achieves 19\% lower packet loss than other learning-based frameworks.

2022-05-16Journal article

Open access

Employing Intelligent Aerial Data Aggregators for Internet of Things: Challenges and Solutions

Publication . Li, Kai; Ni, W.; Noor, Alam; Guizani, Mohsen

Internet-of-Things (IoT) devices equipped with temperature and humidity sensors, and cameras are increasingly deployed to monitor remote and human-unfriendly areas, e.g., farmlands, forests, rural highways or electricity infrastructures. Aerial data aggregators, e.g., autonomous drones, provide a promising solution for collecting sensory data of the IoT devices in human-unfriendly environments, enhancing network scalability and connectivity. The flexibility of a drone and favourable line-of-sight connection between the drone and IoT devices can be exploited to improve data reception at the drone. This article first discusses challenges of the drone-assisted data aggregation in IoT networks, such as incomplete network knowledge at the drone, limited buffers of the IoT devices, and lossy wireless channels. Next, we investigate the feasibility of onboard deep reinforcement learning-based solutions to allow a drone to learn its cruise control and data collection schedule online. For deep reinforcement learning in a continuous operation domain, deep deterministic policy gradient (DDPG) is suitable to deliver effective joint cruise control and communication decision, using its outdated knowledge of the IoT devices and network states. A case study shows that the DDPG-based framework can take advantage of the continuous actions to substantially outperform existing non-learning-based alternatives.

2022-04-01Journal article

Open access

Work-In-Progress: Worst-Case Response Time of Intersection Management Protocols

Publication . Reddy, Radha; Almeida, Luis; Gutiérrez Gaitán, Miguel; Kurunathan, John Harrison; Santos, Pedro M.; Tovar, Eduardo

Intersections are critical elements of urban traffic management and are identified as bottlenecks prone to traffic congestion and accidents. Intelligent intersection management plays a significant role in improving traffic efficiency and safety determining, among other metrics, the waiting time that vehicles incur when crossing an intersection. This work presents a preliminary analysis of the worst-case response time of intersection management protocols that handle mixed traffic with autonomous and human-driven vehicles. We deduce theoretical bounds for such time considered as the interval between the injection of a vehicle in the road system and its departure from the intersection, considering different intersection management protocols for mixed traffic, namely the Synchronous Intersection Management Protocol (SIMP) and several configurations of the conventional Round-Robin (RR) policy. Simulation results validate the analytical bounds partially. Ongoing work addresses thequeue dynamics and its reliable detection by traffic simulators.

2021-12-01Conference object

Open access