blockchain同步处理导致死锁

[bysomeone]

• blockchain事件处理限定1000容量
• 比特元节点断网恢复后大批量同步区块，由于区块处理逻辑互斥，区块处理数超过1000后，事件处理流程堵塞
• 获取到互斥锁的区块执行协程，在CheckBlock时依赖blockchain GetBlock事件处理（此时事件处理已经堵塞），导致循环依赖死锁

互斥锁1等待协程

937 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x15894cc 0x15894a3 0x15986f4 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#	0x468304	sync.runtime_SemacquireMutex+0x24					/usr/local/go/src/runtime/sema.go:71
#	0x15894cb	sync.(*RWMutex).RLock+0x4b						/usr/local/go/src/sync/rwmutex.go:63
#	0x15894a2	github.com/33cn/chain33/blockchain.(*OrphanPool).IsKnownOrphan+0x22	github.com/33cn/[email protected]/blockchain/orphanpool.go:52
#	0x15986f3	github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0x953	
github.com/33cn/[email protected]/blockchain/process.go:71
#	0x15ab147	github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147	
github.com/33cn/[email protected]/blockchain/query_block.go:255
#	0x158fb36	github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6		
github.com/33cn/[email protected]/blockchain/proc.go:188
#	0x159320a	github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca	
github.com/33cn/[email protected]/blockchain/proc.go:408

互斥锁2等待协程

62 @ 0x43b4b6 0x44c36c 0x44c346 0x468305 0x4828e5 0x483eb6 0x483e95 0x1598db8 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#	0x468304	sync.runtime_SemacquireMutex+0x24					/usr/local/go/src/runtime/sema.go:71
#	0x4828e4	sync.(*Mutex).lockSlow+0x164						/usr/local/go/src/sync/mutex.go:138
#	0x483eb5	sync.(*Mutex).Lock+0x35							/usr/local/go/src/sync/mutex.go:81
#	0x483e94	sync.(*RWMutex).Lock+0x14						/usr/local/go/src/sync/rwmutex.go:111
#	0x1598db7	github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x97	github.com/33cn/[email protected]/blockchain/process.go:108
#	0x1598c7b	github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb	
github.com/33cn/[email protected]/blockchain/process.go:103
#	0x15ab147	github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147	github.com/33cn/[email protected]/blockchain/query_block.go:255
#	0x158fb36	github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6		
github.com/33cn/[email protected]/blockchain/proc.go:188
#	0x159320a	github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca	
github.com/33cn/[email protected]/blockchain/proc.go:408

互斥锁占用协程，等待blockchain事件处理

1 @ 0x43b4b6 0x44b272 0x9c61a5 0x9c6525 0xb6dc02 0xb72965 0xb71019 0x159af53 0x159af20 0x1599bc9 0x15998c5 0x158b2b3 0x1598ff5 0x1598c7c 0x15ab148 0x158fb37 0x159320b 0x46c3a1
#	0x9c61a4	github.com/33cn/chain33/queue.(*client).WaitTimeout+0x184			
github.com/33cn/[email protected]/queue/client.go:155
#	0x9c6524	github.com/33cn/chain33/queue.(*client).Wait+0x24				
github.com/33cn/[email protected]/queue/client.go:170
#	0xb6dc01	github.com/33cn/chain33/util.CheckBlock+0x81					
github.com/33cn/[email protected]/util/exec.go:25
#	0xb72964	github.com/33cn/chain33/util.PreExecBlock+0x14a4				
github.com/33cn/[email protected]/util/util.go:448
#	0xb71018	github.com/33cn/chain33/util.ExecBlock+0x258					
github.com/33cn/[email protected]/util/util.go:273
#	0x159af52	github.com/33cn/chain33/blockchain.execBlock+0x172				
github.com/33cn/[email protected]/blockchain/exec.go:15
#	0x159af1f	github.com/33cn/chain33/blockchain.(*BlockChain).connectBlock+0x13f		github.com/33cn/[email protected]/blockchain/process.go:288
#	0x1599bc8	github.com/33cn/chain33/blockchain.(*BlockChain).connectBestChain+0x128		github.com/33cn/[email protected]/blockchain/process.go:206
#	0x15998c4	github.com/33cn/chain33/blockchain.(*BlockChain).maybeAcceptBlock+0x544		github.com/33cn/[email protected]/blockchain/process.go:187
#	0x158b2b2	github.com/33cn/chain33/blockchain.(*OrphanPool).ProcessOrphans+0x6b2		github.com/33cn/[email protected]/blockchain/orphanpool.go:235
#	0x1598ff4	github.com/33cn/chain33/blockchain.(*BlockChain).maybeAddBestChain+0x2d4	github.com/33cn/[email protected]/blockchain/process.go:123
#	0x1598c7b	github.com/33cn/chain33/blockchain.(*BlockChain).ProcessBlock+0xedb		github.com/33cn/[email protected]/blockchain/process.go:103
#	0x15ab147	github.com/33cn/chain33/blockchain.(*BlockChain).ProcAddBlockMsg+0x147		github.com/33cn/[email protected]/blockchain/query_block.go:255
#	0x158fb36	github.com/33cn/chain33/blockchain.(*BlockChain).addBlock+0x2b6			github.com/33cn/[email protected]/blockchain/proc.go:188
#	0x159320a	github.com/33cn/chain33/blockchain.(*BlockChain).processMsg+0xca		github.com/33cn/[email protected]/blockchain/proc.go:408