| 197 | 1 | 101 |
| 下载次数 | 被引频次 | 阅读次数 |
为无人机提供网络连接是未来蜂窝网络系统的一个主要应用，无人机在蜂窝网络中作为移动基站或移动用户设备时，需要在不同的基站之间切换，以保持高速可靠的网络连接。针对无人机移动性强、飞行环境复杂造成无人机在蜂窝基站间发生频繁切换、切换失败等问题，提出了一种基于深度强化学习的无人机连接蜂窝网络切换优化方法。基于深度强化学习框架，实现无人机自适应基站切换的在线学习和决策，克服了以往算法中当状态空间过大而导致训练时间长、泛化能力差的缺点；融合参考信号接收功率和切换次数两项指标作为联合奖励函数，保证无人机在稳定蜂窝网络连接的前提下，减少了无人机在蜂窝基站间的无效切换次数。实验结果表明，所提出的算法经过1 000轮训练，无人机的平均切换次数显著降低，有效避免了不必要的切换，降低了切换失败的概率，提升了无人机连接蜂窝网络时的信号接收功率。
Abstract：Providing network connections for drones is a major application of future cellular network systems. When drones serve as mobile base stations or mobile user equipment in cellular networks, they need to switch between different base stations to maintain high-speed and reliable network connections. Aiming at the problems of frequent handovers and handover failures of UAVs between cellular base stations caused by high mobility of UAVs and complex flight environment, a method for optimizing handover of UAVs connected to cellular networks based on deep reinforcement learning is proposed. First of all, based on a deep reinforcement learning framework, online learning and decision-making for adaptive base station switching of UAVs are realized, which overcomes the shortcomings of previous algorithms that result in long training time and poor generalization ability when the state space is too large. Secondly, two indicators of reference signal received power and handover times are integrated as a joint reward function to ensure that the UAV has a stable cellular network connection and reduces the number of invalid handovers between the UAV and the cellular base station. Experimental results show that after 1 000 rounds of training, the proposed algorithm has significantly reduced the average number of handovers for UAV,effectively avoiding unnecessary handovers, reducing the probability of handover failures, and improving the receive power of UAV when connecting to cellular networks.
[1] HANS I.Survey on UAV Deployment and Trajectory in Wireless Communication Networks:Applications and Challenges[J].Information,2022,13(8):389.
[2] ZENG Y,WU Q Q,ZHANG R.Accessing from the Sky:A Tutorial on UAV Communications for 5G and Beyond[J].Proceedings of the IEEE,2019,107(12):2327-2375.
[3] FOTOUHI A,QIANG H R,DING M,et al.Survey on UAV Cellular Communications:Practical Aspects,Standardization Advancements,Regulation,and Security Challenges[J].IEEE Communications Surveys & Tutorials,2019,21(4):3417-3442.
[4] KIM M,LEE W.Adaptive Success Rate-based Sensor Relocation for IoT Applications[J].KSII Transactions on Internet and Information Systems (TIIS),2021,15(9):3120-3137.
[5] MISHRA D,NATALIZIOE.A Survey on Cellular-connected UAVs:Design Challenges,Enabling 5G/B5G Innovations,and Experimental Advancements[J].Computer Networks,2020,182(7):107451.
[6] MEER I A,OZGER M,SCHUPKE D A,et al.Mobility Management for Cellular-connected UAVs:Model-based Versus Learning-based Approaches for Service Availability[J].IEEE Transactions on Network and Service Management,2024,21(2):2125-2139.
[7] SHAYEA I,DUSHI P,BANAFAA M,et al.Handover Management for Drones in Future Mobile Networks-A Survey[J].Sensors,2022,22(17):6424.
[8] FAKHREDDINE A,BETTSTETTER C,HAYAT S,et al.Handover Challenges for Cellular-connected Drones[C]//Proceedings of the 5th Workshop on Micro Aerial Vehicle Networks,Systems,and Applications.New York:ACM,2019:9-14.
[9] EULER S,MAATTANEN H L,LIN X Q,et al.Mobility Support for Cellular Connected Unmanned Aerial Vehicles:Performance and Analysis[C]//2019 IEEE Wireless Communications and Networking Conference (WCNC).Marrakesh:IEEE,2019:1-6.
[10] ZHONG J H,ZHANG L,SERUGUNDA J,et al.An Improved Q-learning Based Handover Scheme in Cellular-connected UAV Network[C]//2022 25th International Symposium on Wireless Personal Multimedia Communications (WPMC).Herning:IEEE,2022:520-525.
[11] CHOWDHURY M M U,SAAD W,GüVEN? I.Mobility Management for Cellular-connected UAVs:A Learning-based Approach[C]//2020 IEEE International Conference on Communications Workshops (ICC Workshops).Dublin:IEEE,2020:1-6.
[12] YAJNANARAYANA V,RYDéN H,HéVIZI L.5G Handover Using Reinforcement Learning[C]//2020 IEEE 3rd 5G World Forum (5GWF).Bangalore:IEEE,2020:349-354.
[13] GALKIN B,FONSECA E,AMER R,et al.REQIBA:Regression and Deep Q-learning for Intelligent UAV Cellular User to Base Station Association[J].IEEE Transactions on Vehicular Technology,2021,71(1):5-20.
[14] AZARI A,GHAVIMI F,OZGER M,et al.Machine Learning Assisted Handover and Resource Management for Cellular Connected Drones[C]//2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring).Antwerp:IEEE,2020:1-7.
[15] HELLAOUI H,YANG B,TALEB T.Towards Using Deep Reinforcement Learning for Connection Steering in Cellular UAVs[C]//2021 IEEE Global Communications Conference (GLOBECOM).Madrid:IEEE,2021:1-6.
[16] HAN M H,ZHANG L X,WANG J,et al.Actor-critic Reinforcement Learning for Control with Stability Guarantee[J].IEEE Robotics and Automation Letters,2020,5(4):6217-6224.
[17] SIVANESAN K,ZOU J L,VASUDEVAN S,et al.Mobility Performance Optimization for 3GPP LTE HetNets[M]//ANPALAGAN A,BENNIS M,VANNITHAMBY R.Design and Deployment of Small Cell Networks.Cambridge:Cambridge University Press,2015.
[18] ASADI A,PINKLEY S N,MES M.A Markov Decision Process Approach for Managing Medical Drone Deliveries[J].Expert Systems with Applications,2022,204:117490.
[19] HOFMANOVá M,ZHU R C,ZHU X C.Global-in-time Probabilistically Strong and Markov Solutions to Stochastic 3D Navier-Stokes Equations:Existence and Nonuniqueness [J].The Annals of Probability,2023,51(2):524-579.
[20] YUAN Y X,LEI L,VU T X,et al.Energy Minimization in UAV-aided Networks:Actor-critic Learning for Constrained Scheduling Optimization[J].IEEE Transactions on Vehicular Technology,2021,70(5):5028-5042.
[21] GONG X Y,YU J Y,LU S W,et al.Actor-critic with Familiarity-based Trajectory Experience Replay[J].Information Sciences,2022,582(3-4):633-647.
[22] MAENG S J,OZDEMIR O,GUVENC I,et al.LTE I/Q Data Set for UAV Propagation Modeling,Communication,and Navigation Research[J].IEEE Communications Magazine,2023,61(9):90-96.
[23] ALMASRI M,MARJOU X,PARZYSZ F.Reinforcement-Learning Based Handover Optimization for Cellular UAVs Connectivity[J].WSEAS Transactions on Computer Research,2022,10:93-98.
[24] CHEN Y,LIN X Q,KHAN T,et al.Efficient Drone Mobility Support Using Reinforcement Learning[C]//2020 IEEE Wireless Communications and Networking Conference (WCNC).Seoul:IEEE,2020:1-6.
基本信息:
DOI：
中图分类号:TN929.53;TP18;V279
引用信息:
[1]段盈江,赵一帆,丁广恩等.基于深度强化学习的无人机切换管理研究[J].无线电通信技术,2024,50(05):949-957.
基金信息:
国家自然科学基金(61963038,62063035)~~