Skip to content

Commit

Permalink
CliPeon: Fix deadlock on startup by eagerly creating ExecutorLifecycl…
Browse files Browse the repository at this point in the history
…e, ChatHandlerResource.

See stack traces here, from current master: https://gist.github.com/gianm/bd9a66c826995f97fc8f

1. The thread "qtp925672150-62" holds the lock on InternalInjectorCreator.class,
   used by Scopes.SINGLETON, and wants the lock on "handlers" in Lifecycle.addMaybeStartHandler
   called by DiscoveryModule.getServiceAnnouncer.
2. The main thread holds the lock on "handlers" in Lifecycle.addMaybeStartHandler, which it
   took because it's trying to add the ExecutorLifecycle to the lifecycle. main is trying
   to get the InternalInjectorCreator.class lock because it's running ExecutorLifecycle.start,
   which does some Jackson deserialization, and Jackson needs that lock in order to inject
   stuff into the Task it's deserializing.

This patch eagerly instantiates ChatHandlerResource (which I believe is what's trying to
create the ServiceAnnouncer in the qtp925672150-62 jetty thread) and the ExecutorLifecycle.
  • Loading branch information
gianm committed Mar 2, 2016
1 parent c82b680 commit 7557eb2
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ public class ChatHandlerServerModule implements Module
public void configure(Binder binder)
{
Jerseys.addResource(binder, ChatHandlerResource.class);
LifecycleModule.register(binder, ChatHandlerResource.class);

if (properties.containsKey(MAX_CHAT_REQUESTS_PROPERTY)) {
final int maxRequests = Integer.parseInt(properties.getProperty(MAX_CHAT_REQUESTS_PROPERTY));
Expand Down
2 changes: 1 addition & 1 deletion services/src/main/java/io/druid/cli/CliPeon.java
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,6 @@
import io.druid.segment.loading.SegmentLoaderConfig;
import io.druid.segment.loading.StorageLocationConfig;
import io.druid.segment.realtime.firehose.ChatHandlerProvider;
import io.druid.segment.realtime.firehose.ChatHandlerResource;
import io.druid.segment.realtime.firehose.NoopChatHandlerProvider;
import io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider;
import io.druid.segment.realtime.plumber.CoordinatorBasedSegmentHandoffNotifierConfig;
Expand Down Expand Up @@ -157,6 +156,7 @@ public void configure(Binder binder)
binder.bind(DataSegmentArchiver.class).to(OmniDataSegmentArchiver.class).in(LazySingleton.class);

binder.bind(ExecutorLifecycle.class).in(ManageLifecycle.class);
LifecycleModule.register(binder, ExecutorLifecycle.class);
binder.bind(ExecutorLifecycleConfig.class).toInstance(
new ExecutorLifecycleConfig()
.setTaskFile(new File(taskAndStatusFile.get(0)))
Expand Down

0 comments on commit 7557eb2

Please sign in to comment.