Driver array-index-out-of-bounds errors when device is plugged in

[groeneveld]

Hi, I compiled and installed your driver with "make arch=arm64" on ubuntu 22.04, kernel 5.15, on a raspberry pi. When I plug in the wifi adapter, dmesg shows a variety of array-index-out-of-bounds.

[40586.289261] usb 1-1.3: new high-speed USB device number 4 using dwc2
[40586.390944] usb 1-1.3: New USB device found, idVendor=0bda, idProduct=885c, bcdDevice= 0.00
[40586.390988] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[40586.391012] usb 1-1.3: Product: 802.11ax WLAN Adapter
[40586.391033] usb 1-1.3: Manufacturer: Realtek
[40586.391065] usb 1-1.3: SerialNumber: 00e04c000001
[40587.913738] 8852au: loading out-of-tree module taints kernel.
[40588.018469] 8852au: module verification failed: signature and/or required key missing - tainting kernel
[40588.046902] systemd-udevd: page allocation failure: order:4, mode:0xdc0(GFP_KERNEL|__GFP_ZERO), nodemask=(null),cpuset=systemd-udevd.service,mems_allowed=0
[40588.046987] CPU: 3 PID: 15815 Comm: systemd-udevd Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40588.046999] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40588.047005] Call trace:
[40588.047009]  dump_backtrace+0x0/0x1f0
[40588.047027]  show_stack+0x24/0x30
[40588.047036]  dump_stack_lvl+0x8c/0xb8
[40588.047048]  dump_stack+0x18/0x34
[40588.047056]  warn_alloc+0x11c/0x1b0
[40588.047066]  __alloc_pages_slowpath.constprop.0+0x908/0x924
[40588.047075]  __alloc_pages+0x2bc/0x33c
[40588.047083]  __get_free_pages+0x28/0x50
[40588.047092]  ftrace_allocate_pages+0xac/0x240
[40588.047104]  ftrace_process_locs.isra.0+0x6c/0x300
[40588.047116]  ftrace_module_init+0x34/0x40
[40588.047128]  load_module+0x60c/0xbb0
[40588.047137]  __do_sys_finit_module+0xa8/0x11c
[40588.047146]  __arm64_sys_finit_module+0x2c/0x40
[40588.047154]  invoke_syscall+0x50/0x120
[40588.047168]  el0_svc_common.constprop.0+0x180/0x1a0
[40588.047180]  do_el0_svc+0x34/0xa0
[40588.047190]  el0_svc+0x4c/0x1c0
[40588.047202]  el0t_64_sync_handler+0xa4/0x12c
[40588.047212]  el0t_64_sync+0x1a4/0x1a8
[40588.047231] Mem-Info:
[40588.047242] active_anon:1097 inactive_anon:6408 isolated_anon:0
                active_file:30040 inactive_file:24632 isolated_file:0
                unevictable:6549 dirty:40 writeback:1
                slab_reclaimable:10642 slab_unreclaimable:10411
                mapped:12363 shmem:31 pagetables:746 bounce:0
                kernel_misc_reclaimable:0
                free:3708 free_pcp:41 free_cma:708
[40588.047268] Node 0 active_anon:4388kB inactive_anon:25632kB active_file:120160kB inactive_file:98528kB unevictable:26196kB isolated(anon):0kB isolated(file):0kB mapped:49452kB dirty:160kB writeback:4kB shmem:124kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:2960kB pagetables:2984kB all_unreclaimable? no
[40588.047295] DMA free:14832kB min:2460kB low:3072kB high:3684kB reserved_highatomic:0KB active_anon:4316kB inactive_anon:25616kB active_file:120160kB inactive_file:98532kB unevictable:26196kB writepending:164kB present:458752kB managed:424876kB mlocked:26196kB bounce:0kB free_pcp:164kB local_pcp:0kB free_cma:2832kB
[40588.047321] lowmem_reserve[]: 0 0 0 0
[40588.047369] DMA: 1925*4kB (MEC) 704*8kB (UMEC) 90*16kB (MC) 10*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15092kB
[40588.047523] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[40588.047537] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=32768kB
[40588.047549] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[40588.047562] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=64kB
[40588.047574] 59209 total pagecache pages
[40588.047583] 2984 pages in swap cache
[40588.047592] Swap cache stats: add 335543, delete 332563, find 82814/158926
[40588.047604] Free swap  = 2023164kB
[40588.047613] Total swap = 2097148kB
[40588.047622] 114688 pages RAM
[40588.047631] 0 pages HighMem/MovableOnly
[40588.047640] 8469 pages reserved
[40588.047648] 16384 pages cma reserved
[40588.796728] usbcore: registered new interface driver rtl8852au
[40588.980865] rtl8852au 1-1.3:1.0 wlx90de8025f143: renamed from wlan1
[40590.897439] ================================================================================
[40590.906400] UBSAN: shift-out-of-bounds in /home/pi/rtl8852au/phl/hal_g6/phy/bb/halbb_interface.c:136:40
[40590.916308] shift exponent 32 is too large for 32-bit type 'unsigned int'
[40590.923399] CPU: 2 PID: 665 Comm: NetworkManager Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40590.923419] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40590.923427] Call trace:
[40590.923431]  dump_backtrace+0x0/0x1f0
[40590.923448]  show_stack+0x24/0x30
[40590.923457]  dump_stack_lvl+0x8c/0xb8
[40590.923469]  dump_stack+0x18/0x34
[40590.923477]  ubsan_epilogue+0x10/0x54
[40590.923485]  __ubsan_handle_shift_out_of_bounds+0xf0/0x150
[40590.923497]  halbb_set_reg+0xb4/0xb8 [8852au]
[40590.924093]  halbb_digital_cfo_comp_init+0x30/0x6c [8852au]
[40590.924559]  halbb_cfo_trk_init+0xbc/0x13c [8852au]
[40590.925017]  halbb_dm_init+0xd8/0x1d8 [8852au]
[40590.925480]  rtw_hal_bb_dm_init+0x24/0x30 [8852au]
[40590.925934]  hal_start_8852a+0x140/0x250 [8852au]
[40590.926392]  hal_start_8852au+0x60/0x8c [8852au]
[40590.926847]  rtw_hal_start+0x44/0x120 [8852au]
[40590.927298]  rtw_phl_start+0x40/0x174 [8852au]
[40590.927751]  rtw_hw_start+0x38/0x68 [8852au]
[40590.928201]  netdev_open+0xd0/0x1a4 [8852au]
[40590.928655]  __dev_open+0x12c/0x224
[40590.928670]  __dev_change_flags+0x1a0/0x240
[40590.928681]  dev_change_flags+0x30/0x70
[40590.928692]  do_setlink+0x220/0x9e0
[40590.928703]  __rtnl_newlink+0x494/0x820
[40590.928712]  rtnl_newlink+0x5c/0x90
[40590.928722]  rtnetlink_rcv_msg+0x140/0x3a0
[40590.928732]  netlink_rcv_skb+0x68/0x134
[40590.928742]  rtnetlink_rcv+0x24/0x30
[40590.928752]  netlink_unicast+0x2c8/0x31c
[40590.928761]  netlink_sendmsg+0x280/0x470
[40590.928770]  sock_sendmsg+0x60/0x70
[40590.928780]  ____sys_sendmsg+0x290/0x2d4
[40590.928790]  ___sys_sendmsg+0x84/0xd0
[40590.928801]  __sys_sendmsg+0x74/0xd0
[40590.928812]  __arm64_sys_sendmsg+0x30/0x3c
[40590.928823]  invoke_syscall+0x50/0x120
[40590.928836]  el0_svc_common.constprop.0+0x6c/0x1a0
[40590.928848]  do_el0_svc+0x34/0xa0
[40590.928858]  el0_svc+0x4c/0x1c0
[40590.928871]  el0t_64_sync_handler+0xa4/0x12c
[40590.928881]  el0t_64_sync+0x1a4/0x1a8
[40590.929085] ================================================================================
[40592.198434] ================================================================================
[40592.207250] UBSAN: array-index-out-of-bounds in /home/pi/rtl8852au/phl/phl_cmd_dispatcher.c:1936:7
[40592.216621] index 16 is out of range for type 'u8 [16]'
[40592.222100] CPU: 0 PID: 15833 Comm: disp_eng_share_ Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40592.222120] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40592.222126] Call trace:
[40592.222130]  dump_backtrace+0x0/0x1f0
[40592.222148]  show_stack+0x24/0x30
[40592.222157]  dump_stack_lvl+0x8c/0xb8
[40592.222168]  dump_stack+0x18/0x34
[40592.222176]  ubsan_epilogue+0x10/0x54
[40592.222184]  __ubsan_handle_out_of_bounds+0x80/0x90
[40592.222196]  dispr_send_msg+0x158/0x360 [8852au]
[40592.222774]  phl_disp_eng_send_msg+0x90/0x98 [8852au]
[40592.223232]  _phl_cmd_scan_req_acquired+0x108/0x164 [8852au]
[40592.223680]  register_cur_cmd_req+0x4c/0x7c [8852au]
[40592.224133]  dispr_process_token_req+0x64/0x108 [8852au]
[40592.224588]  _handle_token_op_info+0xac/0x1a8 [8852au]
[40592.225042]  token_op_hanler+0x54/0xb4 [8852au]
[40592.225499]  dispr_thread_loop_hdl+0x40/0x1f4 [8852au]
[40592.225952]  dispr_share_thread_loop_hdl+0x1c/0x28 [8852au]
[40592.226407]  share_thread_hdl+0x8c/0x144 [8852au]
[40592.226857]  kthread+0x12c/0x140
[40592.226870]  ret_from_fork+0x10/0x20
[40592.227109] ================================================================================
[40592.235884] ================================================================================
[40592.244624] UBSAN: array-index-out-of-bounds in /home/pi/rtl8852au/phl/phl_cmd_dispatcher.c:557:5
[40592.256465] index 16 is out of range for type 'u8 [16]'
[40592.264495] CPU: 0 PID: 15833 Comm: disp_eng_share_ Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40592.264510] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40592.264517] Call trace:
[40592.264520]  dump_backtrace+0x0/0x1f0
[40592.264538]  show_stack+0x24/0x30
[40592.264547]  dump_stack_lvl+0x8c/0xb8
[40592.264559]  dump_stack+0x18/0x34
[40592.264568]  ubsan_epilogue+0x10/0x54
[40592.264576]  __ubsan_handle_out_of_bounds+0x80/0x90
[40592.264588]  set_msg_bitmap+0xdc/0xe0 [8852au]
[40592.265332]  dispr_send_msg+0x1e0/0x360 [8852au]
[40592.265915]  phl_disp_eng_send_msg+0x90/0x98 [8852au]
[40592.266443]  _phl_cmd_scan_req_acquired+0x108/0x164 [8852au]
[40592.266966]  register_cur_cmd_req+0x4c/0x7c [8852au]
[40592.267488]  dispr_process_token_req+0x64/0x108 [8852au]
[40592.268014]  _handle_token_op_info+0xac/0x1a8 [8852au]
[40592.268538]  token_op_hanler+0x54/0xb4 [8852au]
[40592.269067]  dispr_thread_loop_hdl+0x40/0x1f4 [8852au]
[40592.269596]  dispr_share_thread_loop_hdl+0x1c/0x28 [8852au]
[40592.270067]  share_thread_hdl+0x8c/0x144 [8852au]
[40592.270520]  kthread+0x12c/0x140
[40592.270534]  ret_from_fork+0x10/0x20
[40592.270633] ================================================================================
[40592.284708] ================================================================================
[40592.298752] UBSAN: array-index-out-of-bounds in /home/pi/rtl8852au/phl/phl_cmd_dispatcher.c:580:5
[40592.313226] index 16 is out of range for type 'u8 [16]'
[40592.321299] CPU: 0 PID: 15833 Comm: disp_eng_share_ Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40592.321316] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40592.321323] Call trace:
[40592.321327]  dump_backtrace+0x0/0x1f0
[40592.321345]  show_stack+0x24/0x30
[40592.321354]  dump_stack_lvl+0x8c/0xb8
[40592.321365]  dump_stack+0x18/0x34
[40592.321373]  ubsan_epilogue+0x10/0x54
[40592.321381]  __ubsan_handle_out_of_bounds+0x80/0x90
[40592.321393]  set_msg_custom_bitmap+0x1c8/0x200 [8852au]
[40592.322066]  dispr_send_msg+0x200/0x360 [8852au]
[40592.322547]  phl_disp_eng_send_msg+0x90/0x98 [8852au]
[40592.322998]  _phl_cmd_scan_req_acquired+0x108/0x164 [8852au]
[40592.323448]  register_cur_cmd_req+0x4c/0x7c [8852au]
[40592.323901]  dispr_process_token_req+0x64/0x108 [8852au]
[40592.324350]  _handle_token_op_info+0xac/0x1a8 [8852au]
[40592.324804]  token_op_hanler+0x54/0xb4 [8852au]
[40592.325270]  dispr_thread_loop_hdl+0x40/0x1f4 [8852au]
[40592.325733]  dispr_share_thread_loop_hdl+0x1c/0x28 [8852au]
[40592.326187]  share_thread_hdl+0x8c/0x144 [8852au]
[40592.326639]  kthread+0x12c/0x140
[40592.326654]  ret_from_fork+0x10/0x20
[40592.326777] ================================================================================
[40592.340853] ================================================================================
[40592.354871] UBSAN: array-index-out-of-bounds in /home/pi/rtl8852au/phl/phl_cmd_dispatcher.c:1304:7
[40592.369404] index 16 is out of range for type 'u8 [16]'
[40592.377396] CPU: 0 PID: 15833 Comm: disp_eng_share_ Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40592.377410] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40592.377417] Call trace:
[40592.377421]  dump_backtrace+0x0/0x1f0
[40592.377440]  show_stack+0x24/0x30
[40592.377448]  dump_stack_lvl+0x8c/0xb8
[40592.377460]  dump_stack+0x18/0x34
[40592.377467]  ubsan_epilogue+0x10/0x54
[40592.377475]  __ubsan_handle_out_of_bounds+0x80/0x90
[40592.377487]  get_module_by_id+0x158/0x198 [8852au]
[40592.378221]  dispr_send_msg+0x25c/0x360 [8852au]
[40592.378807]  phl_disp_eng_send_msg+0x90/0x98 [8852au]
[40592.379333]  _phl_cmd_scan_req_acquired+0x108/0x164 [8852au]
[40592.379859]  register_cur_cmd_req+0x4c/0x7c [8852au]
[40592.380382]  dispr_process_token_req+0x64/0x108 [8852au]
[40592.380899]  _handle_token_op_info+0xac/0x1a8 [8852au]
[40592.381422]  token_op_hanler+0x54/0xb4 [8852au]
[40592.381944]  dispr_thread_loop_hdl+0x40/0x1f4 [8852au]
[40592.382460]  dispr_share_thread_loop_hdl+0x1c/0x28 [8852au]
[40592.382971]  share_thread_hdl+0x8c/0x144 [8852au]
[40592.383483]  kthread+0x12c/0x140
[40592.383498]  ret_from_fork+0x10/0x20
[40592.383641] ================================================================================
[40598.643896] ================================================================================
[40598.660246] UBSAN: array-index-out-of-bounds in /home/pi/rtl8852au/phl/phl_msg_hub.c:136:6
[40598.674814] index 16 is out of range for type 'u8 [16]'
[40598.682925] CPU: 3 PID: 15834 Comm: msg_notify_thre Tainted: G        WC OE     5.15.0-1005-raspi #5-Ubuntu
[40598.682945] Hardware name: Raspberry Pi Zero 2 Rev 1.0 (DT)
[40598.682953] Call trace:
[40598.682957]  dump_backtrace+0x0/0x1f0
[40598.682976]  show_stack+0x24/0x30
[40598.682985]  dump_stack_lvl+0x8c/0xb8
[40598.682998]  dump_stack+0x18/0x34
[40598.683005]  ubsan_epilogue+0x10/0x54
[40598.683014]  __ubsan_handle_out_of_bounds+0x80/0x90
[40598.683026]  msg_forward+0xcc/0xf0 [8852au]
[40598.683681]  msg_thread_hdl+0x60/0x144 [8852au]
[40598.684229]  kthread+0x12c/0x140
[40598.684243]  ret_from_fork+0x10/0x20
[40598.684290] ================================================================================

Sometimes the adapter seems to work despite these errors. Other times, with no pattern I've discerned, establishing connections with nmcli times out, the entire device hangs and won't accept ssh session requests, the device will spontaneously stop working, or other odd problems. At one point the network interface name briefly became wlan1 though it's usually named wlx90de8025f143.
"journalctl -xe NM_CONNECTION=de57e244-3714-49e7-b209-f21ac57064f7 + NM_DEVICE=wlan1":

Jul 25 16:26:09 raspberrypi NetworkManager[694]: <warn>  [1658766369.8505] device (wlan1): Activation: (wifi) Hotspot network creation took too long, failing activation
Jul 25 16:26:09 raspberrypi NetworkManager[694]: <info>  [1658766369.8513] device (wlan1): state change: config -> failed (reason 'supplicant-timeout', sys-iface-state: 'managed')
Jul 25 16:26:09 raspberrypi NetworkManager[694]: <warn>  [1658766369.8556] device (wlan1): Activation: failed for connection 'axhotspot'
Jul 25 16:26:09 raspberrypi NetworkManager[694]: <info>  [1658766369.8584] device (wlan1): state change: failed -> disconnected (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3785] device (wlan1): Activation: starting connection 'axhotspot' (de57e244-3714-49e7-b209-f21ac57064f7)
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3796] device (wlan1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3848] device (wlan1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3865] device (wlan1): Activation: (wifi) access point 'axhotspot' has security, but secrets are required.
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3868] device (wlan1): state change: config -> need-auth (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3927] device (wlan1): state change: need-auth -> prepare (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3969] device (wlan1): state change: prepare -> config (reason 'none', sys-iface-state: 'managed')
Jul 25 16:26:10 raspberrypi NetworkManager[694]: <info>  [1658766370.3987] device (wlan1): Activation: (wifi) connection 'axhotspot' has security, and secrets exist.  No new secrets```