XCOL: xApp-based Framework for Detecting Layer-2 DoS Attacks in 5G NR

The growing adoption of 5G in industrial deployments expands the attack surface for adversarial exploitation. Fake Base Stations (FBSs), which lure benign UEs in a fake attach procedure, and Malicious User Equipment (MUEs), which simultaneously flood legitimate gNBs, can be used to launch combined Denial-of-Service (DoS) attacks. Mobile networks lack the capability to detect Layer-2 DoS attacks originating from these FBSs and MUEs. In this work, we propose an O-RAN-compliant XCOL framework that uses a passive air-interface sniffer (SNI5GECT) to collect Layer-2 telemetry alongside standard E2 KPM telemetry from the O-DU, and deploys an xApp on the Near-RT RIC to detect combined FBS- and MUE-induced attacks. The xApp employs sequence models (BiLSTM-Attention, Conv1D) over per-UE RACH event traces, leveraging three engineered feature families: Temporal Burstiness, RAR suppression ratio, and RACH Completion Ratio. We show that the proposed XCOL achieves an F1 score of 1.0 for gNodeB-level MUE flood detection. At the UE level, it attains an F1 score of 0.678 with 100% precision for MUE detection and an F1 score of 0.903 for FBS-victim identification, outperforming threshold baselines while satisfying Near-RT RIC latency requirements.