geninfo generates corrupted files

[matttbe]

Hello,

First, thank you for developing and maintaining this great tool!

I'm using it (v2.3.1) to generate code coverage for a part of the kernel.

The capture looks OK:

# lcov --capture --keep-going -j "${INPUT_CPUS}" \
     --rc geninfo_unexecuted_blocks=1 \
     --include '/net/mptcp/' \
     --function-coverage --branch-coverage \
     -b "${VIRTME_BUILD_DIR}" -o "${LCOV_FILE}"
  Auto-detecting gcov kernel support.
  Found upstream gcov kernel support at /sys/kernel/debug/gcov
  Copying data to temporary directory /tmp/zQjTEFYYXK
  Capturing coverage data from /tmp/zQjTEFYYXK
  geninfo cmd: '/usr/bin/geninfo /tmp/zQjTEFYYXK --toolname lcov --output-filename /home/runner/work/mptcp_net-next/mptcp_net-next/kernel.lcov --keep-going --base-directory /home/runner/work/mptcp_net-next/mptcp_net-next/.virtme/build --parallel 4 --branch-coverage --rc geninfo_unexecuted_blocks=1 --include /net/mptcp/'
  Found gcov version: 14.2.0
  Using intermediate gcov format
  Recording 'internal' directories:
  	/tmp/zQjTEFYYXK
  	/home/runner/work/mptcp_net-next/mptcp_net-next/.virtme/build
  Writing temporary data to /tmp/geninfo_datrewd
  Scanning /tmp/zQjTEFYYXK for .gcda files ...
  Found 20 data files in /tmp/zQjTEFYYXK
  using: chunkSize: 4, nchunks:5, intervalLength:1
  elapsed:0.0m: remaining:16 files 0.0m: 5.89 files/s 0.16 s/file (interval:5.89 f/s 0.16 s/f)
  elapsed:0.0m: remaining:12 files 0.0m: 11.60 files/s 0.16 s/file (interval:366.27 f/s 0.16 s/f)
  elapsed:0.0m: remaining:8 files 0.0m: 16.24 files/s 0.17 s/file (interval:81.78 f/s 0.17 s/f)
  elapsed:0.0m: remaining:4 files 0.0m: 17.03 files/s 0.18 s/file (interval:19.92 f/s 0.22 s/f)
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":600: mismatched exception tag for id 0, 0: '1' -> '0'
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":600: mismatched exception tag for id 0, 0: '1' -> '0'
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":659: mismatched exception tag for id 0, 0: '0' -> '1'
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 4, 4: '0' -> '1'
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 4, 4: '0' -> '1'
  lcov: ERROR: (mismatch) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":659: mismatched exception tag for id 0, 0: '0' -> '1'
  elapsed:0.0m: remaining:0 files 0.0m: 15.30 files/s 0.17 s/file (interval:10.89 f/s 0.14 s/f)
  Finished processing 20 GCDA files
  Apply filtering..
  Finished filter file processing
  Finished .info-file creation
  Summary coverage rate:
    source files: 22
    lines.......: 88.1% (7275 of 8260 lines)
    functions...: 92.4% (500 of 541 functions)
    branches....: 63.7% (3808 of 5975 branches)
  Message summary:
    6 error messages:
      mismatch: 6

Showing the stats no longer works:

$ lcov --branch-coverage --summary "${LCOV_FILE}"
 lcov: ERROR: (corrupt) unable to read trace file '/home/runner/work/mptcp_net-next/mptcp_net-next/kernel.lcov': lcov: ERROR: (inconsistent) "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.c":2382: line is hit but no branches on line have been evaluated.
Reading tracefile /home/runner/work/mptcp_net-next/mptcp_net-next/kernel.lcov.
	To skip consistency checks, see the 'check_data_consistency' section in man lcovrc(5).
	(use "lcov --ignore-errors inconsistent ..." to bypass this error)
	(use "lcov --ignore-errors corrupt ..." to bypass this error)

Are these errors expected?

[matttbe]

I forgot to add that it was working before. I suspect it stopped working when switching the base from Ubuntu 24.10 to 25.04, which means lcov switched from v2.1~beta2 to v2.3. I upgraded lcov to v2.3.1 and I have the same issue.

Note that I didn't notice this issue with the stats quickly, and I no longer have log from before the upgrade. All the builds after the upgrade had the issue, that's why I'm fairly confident the issue started with the last upgrade.

---

FYI, generating HTML files has errors, but be careful that this part is using lcov v2.0 here, not v2.3.1. If I remember well, these errors were there and OK before when the capture was done with lcov 2.1 and not 2.3. With the capture generated by lcov 2.3, the function coverage is no longer working when using genhtml 2.0:

genhtml: WARNING: Error: "lcov-btf-debug/kernel.lcov":3: unexpected .info file record 'FNL:0,102,107'
genhtml: WARNING: Error: "lcov-btf-debug/kernel.lcov":4: unexpected .info file record 'FNA:0,48,bpf_mptcp_sched_check_member'
	(use "genhtml --ignore-errors format ..." to bypass this error)
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 16, 16: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 24, 24: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 6, 6: '1' -> '0'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 16, 16: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 24, 24: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 6, 6: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 26, 26: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 6, 6: '1' -> '0'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 16, 16: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 24, 24: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 6, 6: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

genhtml: WARNING: Error: "/home/runner/work/mptcp_net-next/mptcp_net-next/net/mptcp/protocol.h":594: mismatched exception tag for id 26, 26: '0' -> '1'
	(use "genhtml --ignore-errors mismatch ..." to bypass this error)

Found 22 entries.
Found common filename prefix "/home/runner/work/mptcp_net-next/mptcp_net-next/net"
Generating output.
Overall coverage rate:
  lines......: 89.2% (7474 of 8378 lines)
  functions......: no data found
  branches......: 64.1% (4282 of 6681 branches)

(I will try to upgrade to v2.3 ASAP, but this piece is harder to change for the moment)

[henry2cox]

The issue appears to be that gcov is producing some inconsistent coverage data for the files you are looking at - and lcov has gotten a lot more picky about checking for such inconsistency.
We don't like to rely on inconistent/obviously wrong data for verification - so we want to flag them so we can either fix the tools in question or (manually) sign off on the issue.

In this case:

• during capture, geninfo complains about a number of branch expressions which are reported as related to exception handlers in some compilation units, but as not related to exception handlers in other compilation units. (This gets ignored/overwritten when you use the --ignore mismatch flag.)
• then when you attempt to summarize, lcov complains about inconsistent data - specifically, that there is a line which is not hit (count is zero) but which contains branches at least one of which is marked hit. It is clearly not possible to evaluate a branch on some line without evaluating the statement on that line.
  This issue can also be ignored: --ignore inconsistent, or you can turn off consistency checking - as described in the error message text.

Generally, newer versions of llvm/gcc have fewer bugs...but you appear to already be using gcc/14.2.0 so not much upgrading is possible. (Would be interesting - though maybe not possible - to run the same tests on an llvm build using both the gcov and profdata paths...and see how different each of the reports are. Experience says: quite different.)

[matttbe]

@henry2cox thank you for your reply, and the nice explanations! I better understand why lcov is complaining now, and the errors are initially due to gcov, ignored during the capture step.

I'm now using lcov --keep-going --summary <file> and the summary is back, thank you!

Generally, newer versions of llvm/gcc have fewer bugs...but you appear to already be using gcc/14.2.0 so not much upgrading is possible.

Good to know, thank you. I usually upgrade once every 6 months.

(Would be interesting - though maybe not possible - to run the same tests on an llvm build using both the gcov and profdata paths...and see how different each of the reports are. Experience says: quite different.)

Good to know, thank you! To motivate the switch, are the reports from llvm usually better?
It shouldn't be too difficult to switch, but there are probably some parts that will need to be adapted.

(I will try to upgrade to v2.3 ASAP, but this piece is harder to change for the moment)

FYI I managed to force the upgrade of genhtml to v2.3.1, and it fixes the issue with the function coverage (as visible here)

I guess we can close this ticket then! Thanks again!

[henry2cox]

Good to know, thank you! To motivate the switch, are the reports from llvm usually better?

Sadly, neither is consistently better than the other in all circumstances.
Branch coverage numbers seem a bit less noisy (need less filtering) in the LLVM data.
MC/DC data is cleaner/more actionable from GCC. (You are using gcc/14 - so could enable MC/DC.)
LLVM generates some odd statement coverage artifacts - some of which are relatively easily filtered.

[matttbe]

Looks like a funny situation where we cannot get the best from both :)

MC/DC data is cleaner/more actionable from GCC. (You are using gcc/14 - so could enable MC/DC.)

I didn't know, thank you for the tip! It looks like it is not enabled by default on the kernel side, and there are no easy option to enable that (we can set env vars). I will first try to investigate why.