[Bug][move-compiler-v2] transactional test framework seems to have a race

[brmataptos]

🐛 Bug

Running

cargo test --locked --profile ci -p move-compiler-v2 -p move-compiler-v2-transactional-tests

twice on Macbook yields occasionally different outputs for some transactional tests.

For example, this test first failed, then succeeded:

brm@brms-mbp-2 aptos-core % git diff
diff --git a/third_party/move/move-compiler-v2/transactional-tests/tests/dependencies/transitive_deps.exp b/third_party/move/move-compiler-v2/transactional-tests/tests/dependencies/transitive_deps.exp
index 82abd5e5e9..15b4a32619 100644
--- a/third_party/move/move-compiler-v2/transactional-tests/tests/dependencies/transitive_deps.exp
+++ b/third_party/move/move-compiler-v2/transactional-tests/tests/dependencies/transitive_deps.exp
@@ -1,20 +1,3 @@
-comparison between v1 and v2 failed:
-- processed 3 tasks
-- 
-- task 2 'run'. lines 18-25:
-- Error: error[E03002]: unbound module
--    ┌─ TEMPFILE2:20:5
--    │
-- 20 │ use 0x42::Y;
--    │     ^^^^^^^ Invalid 'use'. Unbound module: '0x42::Y'
-- 
-- error[E03002]: unbound module
--    ┌─ TEMPFILE2:22:5
--    │
-- 22 │     Y::foo();
--    │     ^ Unbound module or type alias 'Y'
-- 
-- 
-- 
-+ processed 3 tasks
-+ 
+processed 3 tasks
+
+==> Compiler v2 delivered same results!

This is very hard to reproduce reliably enough for debugging output, however.

[brmataptos]

Further study reveals that the problem is actually in the V1 code, perhaps in the use of pre-compiled libraries.

[brmataptos]

Here is a stack dump at the point of the error, but it's not very informative:

+- task 1 'run'. lines 12-18:
+- Error: error[E03002]: unbound module
+-    ┌─ TEMPFILE1:14:9
+-    │
+- 14 │     use 0x42::test::two_args;
+-    │         ^^^^^^^^^^ Invalid 'use'. Unbound module: '0x42::test'
+- Backtrace: Backtrace [
+-     { fn: "std::backtrace_rs::backtrace::libunwind::trace", file: "/rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/libunwind.rs", line: 105 },
+-     { fn: "std::backtrace_rs::backtrace::trace_unsynchronized", file: "/rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/mod.rs", line: 66 },
+-     { fn: "std::backtrace::Backtrace::create", file: "/rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/backtrace.rs", line: 331 },
+-     { fn: "add_backtrace", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/diagnostics/mod.rs", line: 310 },
+-     { fn: "new<move_compiler::diagnostics::codes::NameResolution, alloc::string::String, alloc::string::String, core::iter::sources::empty::Empty<(move_ir_types::location::Loc, alloc::string::String)>, alloc::string::String, core::iter::sources::empty::Empty<alloc::string::String>>", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/diagnostics/mod.rs", line: 287 },
+-     { fn: "warn_about_unbound_module_use", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 1142 },
+-     { fn: "use_", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 1271 },
+-     { fn: "uses", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 1136 },
+-     { fn: "script_", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 628 },
+-     { fn: "script", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 612 },
+-     { fn: "definition", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 306 },
+-     { fn: "program", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/expansion/translate.rs", line: 211 },
+-     { fn: "move_compiler::command_line::compiler::run" },
+-     { fn: "run_impl<1, 8>", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/command_line/compiler.rs", line: 284 },
+-     { fn: "run<8>", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/command_line/compiler.rs", line: 341 },
+-     { fn: "{closure#0}<8>", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/command_line/compiler.rs", line: 227 },
+-     { fn: "and_then<(move_compiler::parser::ast::Program, alloc::collections::btree::map::BTreeMap<move_command_line_common::files::FileHash, alloc::collections::btree::map::BTreeMap<u32, alloc::string::String, alloc::alloc::Global>, alloc::alloc::Global>), move_compiler::diagnostics::Diagnostics, (alloc::collections::btree::map::BTreeMap<move_command_line_common::files::FileHash, alloc::collections::btree::map::BTreeMap<u32, alloc::string::String, alloc::alloc::Global>, alloc::alloc::Global>, move_compiler::command_line::compiler::SteppedCompiler<8>), move_compiler::command_line::compiler::{impl#0}::run::{closure_env#0}<8>>", file: "/rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs", line: 1321 },
+-     { fn: "run<8>", file: "/Users/brm/code/aptos-core/third_party/move/move-compiler/src/command_line/compiler.rs", line: 226 },
+-     { fn: "compile_source_unit", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 880 },
+-     { fn: "compile_script_default<move_transactional_test_runner::vm_test_harness::SimpleVMTestAdapter>", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 369 },
+-     { fn: "compile_script<move_transactional_test_runner::vm_test_harness::SimpleVMTestAdapter>", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 330 },
+-     { fn: "handle_command<move_transactional_test_runner::vm_test_harness::SimpleVMTestAdapter>", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 496 },
+-     { fn: "handle_known_task<move_transactional_test_runner::vm_test_harness::SimpleVMTestAdapter>", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 1115 },
+-     { fn: "run_test_impl<move_transactional_test_runner::vm_test_harness::SimpleVMTestAdapter>", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/framework.rs", line: 1053 },
+-     { fn: "run_test_with_config_and_exp_suffix", file: "/Users/brm/code/aptos-core/third_party/move/testing-infra/transactional-test-runner/src/vm_test_harness.rs", line: 568 },
+-     { fn: "run", file: "./tests/tests.rs", line: 140 },
+-     { fn: "{closure#0}", file: "./tests/tests.rs", line: 41 },
+-     { fn: "call_once<tests::TEST_CONFIGS::{closure_env#0}, (&std::path::Path)>", file: "/rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs", line: 250 },
+-     { fn: "core::ops::function::FnOnce::call_once{{vtable.shim}}" },

[brmataptos]

I've ruled out a "TempFile" race, as I currently have caught an instance of the bug in the debugger and see that the temp files are still there and have the expected contents for the test that has failed.

[brmataptos]

I can reliably reproduce on my MacBook M2 this by the following procedure on mainline, even with some interesting debugging output added:

c=0; while [ `git diff | wc -c` -eq 0 ]; do
UB=1 /usr/bin/time -h cargo test --locked -p move-compiler-transactional-tests -- --test-threads=1024 >& test.out31; echo ran $c; ((c = $c + 1))
done

However, it may take a long time, notably 477 runs last time I tried it (with extra debugging info added).

As it is so rare, and I'm not making progress, I'm timing out and moving on to other problems.