AssertionError: You must launch a compilation thread before attempting to use it

[rob-mccann]

Hey - we're seeing this appear intermittently when using happypack.

ERROR in ./components/ad-preview/ad-preview.scss
    Module build failed: ModuleBuildError: Module build failed: AssertionError: You must launch a compilation thread before attempting to use it!!!
        at Object.compile (/myproj/fe/node_modules/happypack/lib/HappyThread.js:138:7)
        at Object.compile (/myproj/fe/node_modules/happypack/lib/HappyThreadPool.js:82:14)
        at HappyPlugin.compile (/myproj/fe/node_modules/happypack/lib/HappyPlugin.js:192:14)
        at Object.HappyLoader (/myproj/fe/node_modules/happypack/lib/HappyLoader.js:31:15)
        at runLoaders (/myproj/fe/node_modules/webpack/lib/NormalModule.js:195:19)
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:364:11
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:230:18
        at runSyncOrAsync (/myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:143:3)
        at iterateNormalLoaders (/myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:229:2)
        at /myproj/fe/node_modules/loader-runner/lib/LoaderRunner.js:202:4
        at /myproj/fe/node_modules/enhanced-resolve/lib/CachedInputFileSystem.js:70:14
        at _combinedTickCallback (internal/process/next_tick.js:73:7)
        at process._tickCallback (internal/process/next_tick.js:104:9)

Webpack 3.6.0
Happypack 4.0.0

Our webpack config is a little too complex to share here, I'll update this when I have a condensed version.

I think this happens because we share a threadpool for multiple instances of webpack - I'm not sure if there's a nice way to work around it without spawning a new threadpool for each instance (and incurring the performance hit therein). I wonder if we could tell the threadpool to wait for X 'done' events before terminating the pool?

Current theory scratchpad

• running webpack in parallel
• SIGINT taking time to terminate the thread
• we transpile our webpack config; perhaps that's outputting something incorrect

[amireh]

Is it possible for you to stop the threadpool manually? As in, do you have the ability to tell when all your builds are finished?

If the answer is yes, we can add an option to simply not have happypack stop the pool by itself (in case it was shared.)

[rob-mccann]

That sounds like it might work - we can detect when it's done via webpacks callback.

[amireh]

Hmm, I just checked the code and it seems that we're doing reference-counting and only really stopping the threadpool once all active compilers have emitted the "done" event. This makes things more complicated, the fact that you're seeing this error.

Is there any chance you're using a plugin that is doing something finicky after the compiler is done? The error seems to indicate it's coming from a SASS loader, but I can't see how that could happen.

[rob-mccann]

Would you mind linking to where you see that?

[amireh]

1. First, HappyPlugin invokes threadPool.stop when the compiler emits "done" (are you using --bail?) in https://github.com/amireh/happypack/blob/master/lib/HappyPlugin.js#L107
2. Then, HappyThreadPool removes that compiler[1] from the list of active compilers in https://github.com/amireh/happypack/blob/master/lib/HappyThreadPool.js#L90
3. Then, finally, if no active compilers remain, a condition tested in https://github.com/amireh/happypack/blob/master/lib/HappyRPCHandler.js#L19, the threadpool is stopped: https://github.com/amireh/happypack/blob/master/lib/HappyThreadPool.js#L93.

[1] that also is kept by a reference-counting mechanism, so the compiler will only be removed when all the plugins for that compilation invoke stop

[rob-mccann]

Just thinking aloud below and trying to get a grasp from reading code by sight, apologies, but perhaps this flow will reproduce the issue?

const config = {}; // this has loads of stuff in it; cut it for now

const threadPool = HappyPack.ThreadPool();

const getConfig = () => (
  merge({
    plugins: [
      new HappyPack({
        id: 'babel',
        threadPool: threadPool,
        loaders: [ 'babel-loader' ]
      }),
    ],
  }, config);
);

await webpack(getConfig());
await webpack(getConfig());

I'll see if I can get a test to fail.

[rob-mccann]

I've done some investigation:

Array of configs

webpack([ { ...configA }, { ...configB }]).run(done)

Running webpack with an array of configurations work as you describe; the compilers are counted in and out again.

Sequential

webpack(configA);
compiler.run(() => { webpack(configB).run() })

Running webpack one after eachother causes the threadPool to terminate the threads after the first webpack and start them up again before the second. Whilst it's not ideal behaviour (performance-wise), it shouldn't error.

Parallel

webpack(configA).run();
webpack(configB).run();

Running initiating two compiler.run, without waiting for one to complete will choose one of the above behaviours, depending on where the thread is at.

Other ideas

Stepping through the code, I don't see how it's possible for us to get to the state where the above error shows. I'll keep digging.

Things I've tried already to get the test to fail:

• delaying the SIGINT using setTimeout; no effect.

[amireh]

Thanks for the great report.

Regarding the sequential and parallel examples you're providing - is there a legit use-case for those? Why would you do that over the first approach, feeding webpack with multiple configurations?

I believe that for those second and third cases it may actually be the best bet to provide you with a hook to opt-out of the thread pool shutting itself down, then you'll be responsible for doing that (once you know all compilations are done.)

[rob-mccann]

We have several, separate gulp tasks running doing various things in webpack. I'll see if we can somehow use gulp to compose a webpack config and then run it once. My hunch is that's not feasible, though.

I'm not sure if it's shutting down early or if open is not being called; can't yet reproduce the issue reliably :( Will keep investigating.

[rob-mccann]

I've added an updated list of theories to the opening post.

[fancy-rabbit]

same problem. invoke multiple compiler or pass multiple configs to compiler reproduces this.