Cannot allocate memory

[Shulyaka]

Hi! I am unable to load the module.

[root@localhost rtl8852au]# insmod ./8852au.ko
insmod: ERROR: could not insert module ./8852au.ko: Cannot allocate memory
[root@localhost rtl8852au]# modprobe 8852au
modprobe: ERROR: could not insert '8852au': Cannot allocate memory
[root@localhost rtl8852au]# free -h
               total        used        free      shared  buff/cache   available
Mem:           2,0Gi       178Mi       716Mi       1,0Mi       1,1Gi       1,8Gi
Swap:          2,0Gi          0B       2,0Gi

dmesg:

[ 9755.528942] vmap allocation for size 13467648 failed: use vmalloc=<size> to increase size
[ 9755.528988] insmod: vmalloc error: size 13463552, vm_struct allocation failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
[ 9755.529047] CPU: 0 PID: 6148 Comm: insmod Tainted: G        WC O      5.18.18-200.fc36.armv7hl #1
[ 9755.529062] Hardware name: Allwinner sun7i (A20) Family
[ 9755.529083]  unwind_backtrace from show_stack+0x18/0x1c
[ 9755.529117]  show_stack from dump_stack_lvl+0x40/0x4c
[ 9755.529142]  dump_stack_lvl from warn_alloc+0xa8/0x134
[ 9755.529165]  warn_alloc from __vmalloc_node_range+0xd8/0x440
[ 9755.529187]  __vmalloc_node_range from module_alloc+0x44/0x54
[ 9755.529208]  module_alloc from load_module+0xc68/0x23d4
[ 9755.529234]  load_module from sys_finit_module+0xdc/0xe8
[ 9755.529257]  sys_finit_module from ret_fast_syscall+0x0/0x1c
[ 9755.529279] Exception stack(0xf1331fa8 to 0xf1331ff0)
[ 9755.529295] 1fa0:                   00000061 00000000 00000003 0043fbac 00000000 00000c10
[ 9755.529310] 1fc0: 00000061 00000000 00000c10 0000017b 008e8210 00440650 00000002 00452da4
[ 9755.529321] 1fe0: beac3e30 beac3e20 00438c08 b6aa5170
[ 9755.529468] Mem-Info:
[ 9755.529487] active_anon:235 inactive_anon:13932 isolated_anon:0
                active_file:119293 inactive_file:149575 isolated_file:0
                unevictable:0 dirty:70 writeback:0
                slab_reclaimable:13829 slab_unreclaimable:9639
                mapped:27187 shmem:300 pagetables:1084 bounce:0
                kernel_misc_reclaimable:0
                free:182546 free_pcp:787 free_cma:24090
[ 9755.529521] Node 0 active_anon:940kB inactive_anon:55728kB active_file:477172kB inactive_file:598300kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:108748kB dirty:280kB writeback:0kB shmem:1200kB writeback_tmp:0kB kernel_stack:1352kB pagetables:4336kB all_unreclaimable? no
[ 9755.529548] DMA free:473332kB boost:0kB min:3372kB low:4212kB high:5052kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:61972kB inactive_file:41908kB unevictable:0kB writepending:0kB present:786432kB managed:742972kB mlocked:0kB bounce:0kB free_pcp:2308kB local_pcp:796kB free_cma:96360kB
[ 9755.529578] lowmem_reserve[]: 0 0 1277 1277
[ 9755.529601] HighMem free:256852kB boost:0kB min:512kB low:2060kB high:3608kB reserved_highatomic:0KB active_anon:940kB inactive_anon:55728kB active_file:415200kB inactive_file:556392kB unevictable:0kB writepending:280kB present:1307648kB managed:1307648kB mlocked:0kB bounce:0kB free_pcp:840kB local_pcp:516kB free_cma:0kB
[ 9755.529628] lowmem_reserve[]: 0 0 0 0
[ 9755.529649] DMA: 7*4kB (MEC) 7*8kB (UME) 6*16kB (ME) 4*32kB (MC) 7*64kB (MEC) 4*128kB (UM) 4*256kB (UM) 4*512kB (ME) 2*1024kB (ME) 4*2048kB (MC) 112*4096kB (MC) = 473332kB
[ 9755.529751] HighMem: 3019*4kB (M) 901*8kB (UM) 22*16kB (M) 347*32kB (UM) 247*64kB (UM) 49*128kB (UM) 39*256kB (UM) 37*512kB (M) 77*1024kB (UM) 37*2048kB (M) 5*4096kB (M) = 256852kB
[ 9755.529854] 269168 total pagecache pages
[ 9755.529859] 0 pages in swap cache
[ 9755.529864] Swap cache stats: add 0, delete 0, find 0/0
[ 9755.529872] Free swap  = 2050044kB
[ 9755.529877] Total swap = 2050044kB
[ 9755.529883] 523520 pages RAM
[ 9755.529888] 326912 pages HighMem/MovableOnly
[ 9755.529892] 10865 pages reserved
[ 9755.529897] 24576 pages cma reserved

[lwfinger]

Yes, these Wifi 6 drivers get large. I looked at disabling MP, and turning off debug, but that only reduced the driver by a little over 200KB. My changes are shown in the attached file.
diffs.txt
Getting it into a system with only 2GB may be a problem, but every little thing will help. You may try eliminating features, but I'm not sure what you want in the final product. For example, do you want to use it as an AP, or do you want VHT or HT performance? That is for you to decide.

You will probably need to follow the kernel's advice to use a "vmalloc=" on the kernel's command line. To see the current values, use 'sudo cat /proc/meminfo | grep Vmalloc' to see what your system currently has. Try increasing the value shown under VmallocTotal to see if the system will still boot AND load the driver.