Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload coredumps on the CI #11528

Closed
Akirathan opened this issue Nov 11, 2024 · 1 comment
Closed

Upload coredumps on the CI #11528

Akirathan opened this issue Nov 11, 2024 · 1 comment
Labels
-ci -compiler p-medium Should be completed in the next few sprints

Comments

@Akirathan
Copy link
Member

In case of segfaults, JVM generates a coredump. We would like to archive this coredump as an artifact on the CI.

Examples of segfaulting jobs and their output

https://github.com/enso-org/enso/actions/runs/11683349234/job/32532353827#step:7:1191 :

  INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # A fatal error has been detected by the Java Runtime Environment:
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ #  SIGSEGV (0xb) at pc=0x00007f6bb617ae66, pid=60384, tid=60548
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # JRE version: OpenJDK Runtime Environment GraalVM CE 21.0.2+13.1 (21.0.2+13) (build 21.0.2+13-jvmci-23.1-b30)
 INFO ide_ci::program::command: sbt ℹ️ # Java VM: OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (21.0.2+13-jvmci-23.1-b30, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
 INFO ide_ci::program::command: sbt ℹ️ # Problematic frame:
 INFO ide_ci::program::command: sbt ℹ️ # V  [libjvm.so+0x7a3e66]  G1ConcurrentMark::scan_root_region(MemRegion const*, unsigned int)+0x136
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /runner/_work/enso/enso/core.60384)
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # An error report file with more information is saved as:
 INFO ide_ci::program::command: sbt ℹ️ # /runner/_work/enso/enso/hs_err_pid60384.log
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # If you would like to submit a bug report, please visit:
 INFO ide_ci::program::command: sbt ℹ️ #   https://github.com/oracle/graal/issues
 INFO ide_ci::program::command: sbt ℹ️ #
TRACE ide_ci::program::command: sbt ℹ️ <ENDUT>
TRACE ide_ci::program::command: sbt ⚠️ <ENDUT>

https://github.com/enso-org/enso/actions/runs/11685885385/job/32540377401#step:7:1063 :

 INFO ide_ci::program::command: sbt ℹ️ [thread 1967627 also had an error][thread 1967608 also had an error]
 INFO ide_ci::program::command: sbt ℹ️ 
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # A fatal error has been detected by the Java Runtime Environment:
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ #  SIGSEGV (0xb) at pc=0x00007f74fc244fcb, pid=1967301, tid=1967618
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # JRE version: OpenJDK Runtime Environment GraalVM CE 21.0.2+13.1 (21.0.2+13) (build 21.0.2+13-jvmci-23.1-b30)
 INFO ide_ci::program::command: sbt ℹ️ # Java VM: OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (21.0.2+13-jvmci-23.1-b30, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
 INFO ide_ci::program::command: sbt ℹ️ # Problematic frame:
 INFO ide_ci::program::command: sbt ℹ️ # V  [libjvm.so+0xb5cfcb][thread 1967599 also had an error]
 INFO ide_ci::program::command: sbt ℹ️   LinkResolver::runtime_resolve_virtual_method(CallInfo&, methodHandle const&, Klass*, Handle, Klass*, bool, JavaThread*)+0xdb
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /runner/_work/enso/enso/core.1967301)
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # An error report file with more information is saved as:
 INFO ide_ci::program::command: sbt ℹ️ # /runner/_work/enso/enso/hs_err_pid1967301.log
Warning: _ci::program::command: sbt ℹ️ [43.083s][warning][os] Loading hsdis library failed
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # If you would like to submit a bug report, please visit:
 INFO ide_ci::program::command: sbt ℹ️ #   https://github.com/oracle/graal/issues
 INFO ide_ci::program::command: sbt ℹ️ #

https://github.com/enso-org/enso/actions/runs/11670158875/job/32493750427#step:7:1068 :

  INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # A fatal error has been detected by the Java Runtime Environment:
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ #  SIGSEGV (0xb) at pc=0x0000000000000000, pid=1302068, tid=1302212
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # JRE version: OpenJDK Runtime Environment GraalVM CE 21.0.2+13.1 (21.0.2+13) (build 21.0.2+13-jvmci-23.1-b30)
 INFO ide_ci::program::command: sbt ℹ️ # Java VM: OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (21.0.2+13-jvmci-23.1-b30, mixed mode, sharing, tiered, jvmci, jvmci compiler, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
 INFO ide_ci::program::command: sbt ℹ️ # Problematic frame:
 INFO ide_ci::program::command: sbt ℹ️ # V  [libjvm.so+0x7e4950][thread 1302201 also had an error]
 INFO ide_ci::program::command: sbt ℹ️   G1ParScanThreadState::handle_evacuation_failure_par(oopDesc*, markWord, unsigned long)+0x1f0
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /runner/_work/enso/enso/core.1302068)
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # An error report file with more information is saved as:
 INFO ide_ci::program::command: sbt ℹ️ # /runner/_work/enso/enso/hs_err_pid1302068.log
 INFO ide_ci::program::command: sbt ℹ️ #
 INFO ide_ci::program::command: sbt ℹ️ # If you would like to submit a bug report, please visit:
 INFO ide_ci::program::command: sbt ℹ️ #   https://github.com/oracle/graal/issues
 INFO ide_ci::program::command: sbt ℹ️ #

Reproduce segfault

Segfault should be reproducible by enabling parallel native image builds with:

diff --git a/build.sbt b/build.sbt
index 2a7909868b..a9045b5be0 100644
--- a/build.sbt
+++ b/build.sbt
@@ -109,7 +109,7 @@ Global / concurrentRestrictions += Tags.limit(simpleLibraryServerTag, 1)
 /** Tag limiting the concurrent spawning of `native-image` subprocess.
   */
 val nativeImageBuildTag = NativeImage.nativeImageBuildTag
-Global / concurrentRestrictions += Tags.limit(nativeImageBuildTag, 1)
+//Global / concurrentRestrictions += Tags.limit(nativeImageBuildTag, 1)
 
 lazy val gatherLicenses =
   taskKey[Unit](

with that, there should be multiple native-image subprocesses running at the same time on the machin, and that will cause out of memory, which results in segfaults.

@Akirathan Akirathan added p-medium Should be completed in the next few sprints -ci -compiler labels Nov 11, 2024
@JaroslavTulach
Copy link
Member

It was hardware issue, let's leave solution of this request for another time.

@JaroslavTulach JaroslavTulach closed this as not planned Won't fix, can't repro, duplicate, stale Jan 7, 2025
@github-project-automation github-project-automation bot moved this from ❓New to 🟢 Accepted in Issues Board Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
-ci -compiler p-medium Should be completed in the next few sprints
Projects
Status: 🟢 Accepted
Development

No branches or pull requests

2 participants