Illegal instruction error during CGO execution over faiss_Index_train() for an IVF100,SQ8 index

[abhinavdangeti]

Summary

Platform

OS: Ubuntu 20.04

Faiss version: v1.7.4

Installed from: compiled by self

Faiss compilation options:

Running on:

• CPU

Interface:

• C++

Reproduction instructions

Hello,

At Couchbase, we are exploring the possibility of embedding FAISS index' content into our Search indexes to support indexing and searching across vector content. Our Search service runs on GO, so we're using GO bindings (https://github.com/DataIntelligenceCrew/go-faiss) via CGO to talk to a pre-compiled FAISS C++ library.

The C++ library is compiled as an external library via cmake with instructions from Couchbase's CMakeLists, here's a flavor of that -

cmake -B build
      -DFAISS_ENABLE_GPU=OFF
      -DFAISS_ENABLE_PYTHON=OFF
      -DFAISS_ENABLE_C_API=ON
      -DBUILD_TESTING=OFF
      -DBUILD_SHARED_LIBS=ON
      -DCMAKE_PREFIX_PATH=${PROJECT_BINARY_DIR}/tlm/deps/openblas.exploded
      -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
      -DCMAKE_INSTALL_LIBDIR=lib

With this and a few other instructions - we've been able to compile all of Couchbase with FAISS successfully. However, on certain linux platforms - we see the search process panic-ing on SIGILLs, which seems somewhat indicative of something amiss while compiling FAISS ..

SIGILL: illegal instruction
PC=0x7fcb8a468b84 m=3 sigcode=2
signal arrived during cgo execution
instruction bytes: 0xc5 0xfa 0x10 0x0 0x48 0x8b 0x85 0x60 0xff 0xff 0xff 0x8b 0x0 0x4c 0x8b 0x45

goroutine 49 [syscall]:
runtime.cgocall(0x124f1f0, 0xc000815108)
        /home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.3/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0008150e0 sp=0xc0008150a8 pc=0x40b70b
github.com/blevesearch/go-faiss._Cfunc_faiss_Index_train(0x7fcb2560b720, 0x4afb, 0xc00fc08000)
        _cgo_gotypes.go:513 +0x4b fp=0xc000815108 sp=0xc0008150e0 pc=0x6e722b
github.com/blevesearch/go-faiss.(*faissIndex).Train.func1(0x7fcb2560b720?, 0x2?, {0xc00fc08000, 0x43dcbb?, 0x7fcb256554f0?})
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/[email protected]/index.go:97 +0x6a fp=0xc000815148 sp=0xc000815108 pc=0x6e7fea
github.com/blevesearch/go-faiss.(*faissIndex).Train(0xc000815378?, {0xc00fc08000, 0x257d80, 0x2d4800})
        /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/blevesearch/[email protected]/index.go:97 +0x59 fp=0xc000815188 sp=0xc000815148 pc=0x6e7f19

Can I get some guidance on or around this situation and if this is something that you've had to deal with before?

Related: DataIntelligenceCrew/go-faiss#23

root@se2207-ubu20:~# lscpu
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      46 bits physical, 48 bits virtual
CPU(s):                             4
On-line CPU(s) list:                0-3
Thread(s) per core:                 1
Core(s) per socket:                 1
Socket(s):                          4
NUMA node(s):                       1
Vendor ID:                          GenuineIntel
CPU family:                         6
Model:                              106
Model name:                         Intel(R) Xeon(R) Gold 6338N CPU @ 2.20GHz
Stepping:                           6
CPU MHz:                            2194.926
BogoMIPS:                           4389.85
Hypervisor vendor:                  Xen
Virtualization type:                full
L1d cache:                          192 KiB
L1i cache:                          128 KiB
L2 cache:                           5 MiB
L3 cache:                           192 MiB
NUMA node0 CPU(s):                  0-3
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:        KVM: Vulnerable
Vulnerability L1tf:                 Mitigation; PTE Inversion
Vulnerability Mds:                  Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIB
                                    RS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Mitigation; Clear CPU buffers; SMT Host state unknown
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx
                                    fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid pni pcl
                                    mulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave
                                     avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd
                                     ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx
                                    smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves umip pku ospke gfni vaes vpclm
                                    ulqdq rdpid md_clear flush_l1d

[alexanderguzhva]

Hi @abhinavdangeti , 1) is it possible to see the exact dataset size and the parameters for training an index? 2) can this issue be always reproduced or is there a probability? 3) can you try running in a debug mode, will the issue persist or not?
Thanks

[abhinavdangeti]

Hi @alexanderguzhva , so it seems we've figured out the root cause for the SIGILL - not sure we entirely understand it just yet, but the problem had to do with how we were building openBLAS which is a BLAS implementation that is a dependency for FAISS.

Now I know you recommend using Intel MKL's BLAS implementation for performance, but for a proof of concept we just used one that was already in our build eco system.

We changed our compile instructions to follow conda-forge's recipe for building openBLAS, assuming they'd know better about compiling things that'd work across different architectures - we updated the CPU Target to PRESCOTT and DYNAMIC_ARCH=1 as seen here - https://github.com/conda-forge/openblas-feedstock/blob/main/recipe/build.sh#L36...L39 and strangely enough that resolved this illegal instruction while executing the C++ dynamic library that we encountered.

[mdouze]

The test machine does not support avx2. Could that be the issue? Faiss needs that to be set at compile time (although it seems that your cmake command line does not set it).

[abhinavdangeti]

@mdouze Hmm, believe the test machine does support avx2.
Is it necessary to set -DFAISS_OPT_LEVEL=avx2 while building with cmake?