Leaking HscEnvs

[mniip]

Working on a project with ~100 components I've been struggling with #2151 for a while. The specific symptom was that HLS would boot using a couple GBs of RAM depending on the components loaded (not a problem), and then as files were edited, the memory usage would just grow and grow to several times the original size, even though the set of loaded components didn't change (problem).

My setup is:

• Arch Linux
• GHC 9.6.7 from GHCup
• Cabal multi-component cradle
• Code OSS + vscode-haskell
• HLS 2.11.0.0, but also reproducible on git HEAD
• No hie.yaml

The typical memory usage pattern looks something like this:

It is most severe when I'm moving stuff between modules and fixing up downstream imports afterwards. This is a proprietary codebase so unfortunately I can't share an eventlog file.

However with a lot of help from @fendor I was able to narrow it down to a specific issue that is reproducible even in a small codebase if you know where to look. According to him, HLS creates an HscEnv per module, but according to the "Detailed" tab there's easily 30x too many HscEnvs and their number just keeps growing all the time.

Here's an eventlog from a much smaller project with only 10 modules: https://cdn.discordapp.com/attachments/1393902060830986250/1410195005364113478/haskell-language-server-9.6.7.eventlog.gz?ex=68b021f1&is=68aed071&hm=dd4b8701031bae67c5b0959443fd90bc5109739d6f29dd8d8c3b5036fd2c7010&

This does not look like much but HscEnv bytes have gone up from 69768 to 80920:

On GHC 9.6.7, an HscEnv is 136 bytes, so we went from 513 HscEnvs to 595. Why are there so many in a 10-module project? Where did extra 82 come from?

With the help of ghc-debug we can see that the retainers of a typical HscEnv look like this:

───────────────────────────────────────────Retainers──────────────────────────────────────────
├─┐ 0   0x428938f1a8   HscEnv 0x428813b000 0x7f91d668f5d0 0x7f91d668f4e0 0x7f91d668f3f0       
│ │                    0x4282d04a50 0x420830a7a0 0x420830a7b0 0x7f91d668f300 0x420830a7c0     
│ │                    0x7f91d6517718 0x428938f6d0 0x420830a800 0x42895de680 0x420830a838     
│ │                    0x420830a860 114                                                       
│ ├─┄ Field 0   0x428813b000   DynFlags 0x7f91d65462a8 0x7f91d6545608 0x7f91d64d3a48          
│ │                            0x7f91d6793850 0x7f91d6ac7d78 0x420830aac0 0x420830aad8        
│ │                            0x7f91cd18a0e8 0x420830aaf0 0x420830ab08 0x42881fae28          
│ │                            0x7f91cc46b110 0x7f91cc46b110 0x7f91cc46b110 0x7f91cc46b118    
│ │                            0x7f91cc46b118 0x7f91cc46b110 0x7f91cc46b110 0x420830ab88      
│ │                            0x420830aba8 0x7f91cc46b178 0x42881faeb0 0x42881faed0          
│ │                            0x42881faef0 0x42881faf08 0x42881faf20 0x42881faf40            
│ │                            0x42881faf58 0x42881faf78 0x42881faf98 0x42881fafb8            
│ │                            0x42881fafd8 0x42881f9400 0x42881f9420 0x42881f9438            
│ │                            0x42881f9450 0x420830ac80 0x7f91cc46b178 0x420830ac98          
│ │                            0x420830acb0 0x7f91cc46b178 0x420830acc8 0x420830ace8          
│ │                            0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178    
│ │                            0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178    
│ │                            0x42881f94d0 0x42881f94e8 0x7f91cc46b118 0x7f91cc46b110        
│ │                            0x42881f9508 0x42881f9528 0x42881f9540 0x7f91cc5a7748          
│ │                            0x7f91cc5a7748 0x7f91cc5a7748 0x7f91cc5a7768 0x7f91cc5a7788    
│ │                            0x7f91cd18a0e8 0x7f91cc46b178 0x7f91d6536d58 0x7f91cc46b110    
│ │                            0x7f91cd18a0e8 0x7f91d65334c8 0x42881f9558 0x42881f9570        
│ │                            0x42881f9588 0x7f91cc5a7788 0x7f91cc5a7928 0x7f91cc5a7d88      
│ │                            0x7f91d6533588 0x7f91d65334f0 0x7f91cc5a7778 0x7f91d6533560    
│ │                            0x7f91d6533588 0x7f91d65337a8 0x7f91d65338d8 0x7f91d65338d8    
│ │                            0x7f91cc46b110 0x7f91cd18a0e8 0x7f91cc5a7888 0x42881f95a0      
│ │                            0x7f91d629cff8 0x7f91cd18a0e8 0x7f91d6533638 0x7f91d6533648    
│ │                            0x428814f060 0x7f91cd18a0e8 0x7f91cc46b178 0x428409ec30        
│ │                            0x7f91cd18a0e8 0x7f91ce189a00 0x7f91ce189a00 0x7f91ce189a00    
│ │                            0x7f91cd18a0e8 0x420830ad70 0x7f91cd18a0e8 0x420830ad80        
│ │                            0x42881f95c8 0x42881f95d8 0x42881f95e8 0x7f91d650e8b8          
│ │                            0x7f91d650e588 0x7f91d6533658 0x7f91d6533680 0x7f91d65336a8    
│ │                            0x7f91d65336d0 0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cd18a0e8    
│ │                            0x7f91cd18a0e8 0x7f91d6542d00 0x7f91cc46b110 0x7f91d65336f8    
│ │                            0x7f91cd18a0e8 0x420830adc0 0x428474d408 0x7f91cc46b178        
│ │                            0x7f91cc46b178 0x7f91cc46b178 0x7f91cd18a0e8 0x7f91d6542bc0    
│ │                            0x7f91cc46b118 0x7f91d6533748 0x7f91cc46b178 0x7f91cc46b178    
│ │                            0x7f91cc46b178 0x7f91cc46b178 0x7f91d6533770 0x7f91cc46b110    
│ │                            0x7f91cc46b110 0x7f91cc46b178 0x7f91cc46b178 0x428474d428      
│ │                            0x7f91cc46b178 0x420830ae18 0x7f91cc46b178 0x7f91cc46b178      
│ │                            0x7f91cd18a0e8 0x7f91ce1821d8 0x428261c160 0x4283bfaad8        
│ │                            0x7f91ce1821d8 0x420830ae80 0x7f91d6956ad0 0x7f91cc46b118      
│ │                            0x7f91cc46b110 0x7f91d695dfb8 0x7f91d695dfb8 0x7f91d695dfb8    
│ │                            0x7f91d695dfb8 0x7f91d695dfb8 0x7f91d695dfb8 0x7f91d695dfb8    
│ │                            0x7f91d695dfb8 0x7f91d695dfb8 0x420830ae90 0x42881f9658        
│ │                            0x7f91d64b1690 0x7f91cc5a77e8 0x7f91cc5a7a68 0x7f91d651ebf0    
│ │                            0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cc46b178 0x7f91cc5a7798    
│ │                            0x7f91cc5a7d88 0x7f91cc46b118 0x7f91d64b9670 0x7f91d6813958    
│ │                            0x7f91d69a2588 0x7f91d69542e0 0x7f91cc46b178 0x7f91cd18a0e8    
│ │                            0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cc46b110 0x7f91cc46b110    
│ │                            0x7f91cc46b110 0x7f91cc46b110 0x7f91cc46b110 0x7f91cc46b110    
│ │                            0x428474d468 0x42881f9690 0x42881f96a0 0x7f91cc5a7f48          
│ │                            0x7f91cc5a7948 0x7f91cc5a7948 0x7f91cc46b110 0x7f91cd18a0e8    
│ │                            0x7f91d6536d38 0x7f91cc5a7758 0x7f91d63bbe20 3                 
│ ├─┄ Field 1   0x7f91d668f5d0   _thunk{ 0x7f91d668f5a8 }()                                   
│ ├─┄ Field 2   0x7f91d668f4e0   _thunk{ 0x7f91d668f4b8 }()                                   
│ ├─┄ Field 3   0x7f91d668f3f0   _thunk{ 0x7f91d668f3c8 }()                                   
│ ├─┄ Field 4   0x4282d04a50   MVar 0x4289dadd50                                              
│ ├─┄ Field 5   0x420830a7a0   STRef 0x4282d62640                                             
│ ├─┄ Field 6   0x420830a7b0   STRef 0x4282d62650                                             
│ ├─┄ Field 7   0x7f91d668f300   _thunk{ 0x7f91d668f2d8 }()                                   
│ ├─┄ Field 8   0x420830a7c0   Just 0x4282d62660                                              
│ ├─┄ Field 9   0x7f91d6517718   Plugins 0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178         
│ │                              0x7f91d65176f8                                               
│ ├─┄ Field 10   0x428938f6d0   UnitEnv 0x4282d62678 0x428814f060 0x428938fc80 0x4282d626b8   
│ │                             0x428813c168                                                  
│ ├─┄ Field 11   0x420830a800   Logger 0x7f91cc46b178 0x7f91cc46b178 0x7f91cc46b178           
│ │                             0x4282d62738 0x7f91d64fd528 0x4282d62748                      
│ ├─┄ Field 12   0x42895de680   Hooks 0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cd18a0e8            
│ │                             0x7f91cd18a0e8 0x42895de9b0 0x7f91cd18a0e8 0x7f91cd18a0e8     
│ │                             0x42895de9c0 0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cd18a0e8     
│ │                             0x7f91cd18a0e8 0x7f91cd18a0e8 0x7f91cd18a0e8                  
│ ├─┄ Field 13   0x420830a838   TmpFs 0x4282d627c0 0x4282d627d0 0x4282d627e0 0x4282d627f0     
│ ├─┄ Field 14   0x420830a860   _thunk{ 0x7f91d64fd570 } 0x4282d62800                         
│ ├─┄ 0   0x428938ecf0   Env 0x428938f1a8 0x428938f230 0x428938f248 97                        
│ ├─┄ 1   0x428923f8d8   _fun{ 0x7f91d68fb1d0 }{0x42871ccd68,0x42871ccd78,0x428938ecf0}       
│ ├─┄ 2   0x428923e7a0   _thunk 0x428923f8d8                                                  
│ ├─┄ 3   0x42871d91d0   Tip 0x428923e7a0 8214565720323798282                                 
│ ├─┄ 4   0x42871d9078   Bin 0x42871d91d0 0x42871d91e8 8214565720323798282 1                  
│ ├─┄ 5   0x42871d0f68   Bin 0x42871d9050 0x42871d9078 8214565720323798280 2                  
│ ├─┄ 6   0x4284448c48   Bin 0x42871d0f68 0x42871d0f90 8214565720323798280 4                  
│ ├─┄ 7   0x428678a6e0   Bin 0x4284448c20 0x4284448c48 8214565720323798272 8                  
│ ├─┄ 8   0x42867895a0   Bin 0x428678a6e0 0x428678a708 8214565720323798272 16                 
│ ├─┄ 9   0x42867b7900   Bin 0x42867895a0 0x42867895c8 8214565720323798272 32                 
│ ├─┄ 10   0x4286786ec0   Bin 0x42867b7900 0x42867b7928 8214565720323798272 64                
│ ├─┄ 11   0x42867b73b0   Bin 0x4286786ec0 0x4286786ee8 8214565720323798272 128               
│ ├─┄ 12   0x4286d5f140   Bin 0x42867b7388 0x42867b73b0 8214565720323798016 256               
│ ├─┄ 13   0x4286d5e390   Bin 0x4286d5f140 0x4286d5f168 8214565720323798016 512               
│ ├─┄ 14   0x42831d6b10   Bin 0x4286d5e368 0x4286d5e390 8214565720323796992 1024              
│ ├─┄ 15   0x42831d5a98   Bin 0x42831d6b10 0x42831d6b38 8214565720323796992 2048              
│ ├─┄ 16   0x42831d5000   Bin 0x42831d5a70 0x42831d5a98 8214565720323792896 4096              
│ ├─┄ 17   0x42895d9878   Bin 0x42831d43c8 0x42831d5000 8214565720323784704 8192              
│ ├─┄ 18   0x4282543b00   Bin 0x42895d9878 0x42895d98a0 8214565720323784704 16384             
│ ├─┄ 19   0x4282542ab0   Bin 0x4282543b00 0x4282543b28 8214565720323784704 32768             
│ ├─┄ 20   0x4282541b78   Bin 0x4282542a88 0x4282542ab0 6917529027641081856                   
│ │                       1152921504606846976                                                 
│ ├─┄ 21   0x4283779910   Bin 0x4282541b50 0x4282541b78 0 4611686018427387904                 
│ ├─┄ 22   0x42885a2790   EPS 0x7f91ce1865d0 0x42837798e0 0x7f91ce1865d0 0x4283779910         
│ │                       0x4283779938 0x4283779958 0x4287b2b1e0 0x4283779970 0x4283779988    
│ │                       0x42837799a0 0x42837799d0                                           
│ ├─┄ 23   0x4282d62678   _mutVar 0x42885a2790                                                
│ ├─┄ 24   0x428813c000   UnitEnv 0x4282d62678 0x428814f060 0x428813c138 0x4282d626b8         
│ │                       0x428813c168                                                        
│ ├─┄ 25   0x4282d62818   HscEnv 0x420830a0d8 0x7f91cc46b178 0x428813bfa0 0x420830a730        
│ │                       0x4282d04a50 0x428813bfb8 0x428813bfc8 0x7f91d64dd1a0 0x420830a7c0  
│ │                       0x7f91d6517718 0x428813c000 0x420830a800 0x7f91d64fa118 0x420830a838
│ │                       0x420830a860 114                                                    
│ ├─┄ 26   0x4287840530   HscEnvEq 0x42879a0878 0x4282d62818 0x4286fa0eb0 0x4286fa0ec8        

We see that the HscEnv is retained by a thunk stored deep inside a different HscEnv which is in turn stored in HscEnvEq.

HscEnvEq is an HLS datatype and is a wrapper via which HLS actually manipulates HscEnvs. Coming back to the "Detailed" tab we see that these oscillate between 480 and 640 bytes (12-16 closures) and don't actually leak over time. So there is a large amount of HscEnvs that HLS doesn't "know" about.

Decoding the pointer chain from the outer HscEnv @0x4282d62818 to the inner HscEnv @0x428938f1a8, we see that it's retained by:
-> hsc_unit_env (UnitEnv) -> ue_eps (ExternalUnitCache) -> euc_eps (ExternalPackageState) -> eps_PTE (PackageTypeEnv) -> [value at some key in NameEnv] (TyThing) -> THUNK @0x428923f8d8 -> FUN @0x7f91d68fb1d0

Recompiling HLS (and GHC) with -finfo-table-map (quite a challenge I must say), I found that the FUN points to GHC.Tc.Utils.Monad.forkM.

There's other kinds of retainers, sometimes the HscEnv appears inside a thunk inside a DataCon; but they all share the property that there's an HscEnv inside a forkM thunk inside the ExternalPackageState of an HscEnv managed by HLS. I haven't seen any retainer chains longer than 2 HscEnvs.

This is where I hit a wall. I don't know who called forkM or why, nor what is supposed to happen instead of a thunk.

There is however a memory invariant here that can be easily reproduced, and hopefully someone else can take it from here.

Note also that after #4596 interface files are rebuilt much less frequently, which makes it leak slower, but it still leaks. I'm also not completely convinced that HscEnvs are created only when these "regenerate interface" traces occur.

[soulomoon]

@mniip Great findings!
forkM is called because when we rehydrate an interface, we are doing so lazily, so that we can tie the knot etc. in ghc.

There's other kinds of retainers, sometimes the HscEnv appears inside a thunk inside a DataCon; but they all share the property that there's an HscEnv inside a forkM thunk inside the ExternalPackageState of an HscEnv managed by HLS. I haven't seen any retainer chains longer than 2 HscEnvs.

This is odd, ExternalPackageState should contain rehydrated version of interfaces. When we rehydrate an ModIface to ModDetail and distribute the fields into EPS, an HscEnv is not likely to be introduced during it's proccess.
Only two possibilities we reference to an hscEnv in this proccess:

1. Through knotvars, hsc_type_env_vars etc..
2. The ModIface itself reference to another HscEnv. (Unlikely)

[mniip]

Yeah I quickly found that this appears to be in part intentional design. The IORef EPS are all shared between HscEnvs and there seems to be only 2 EPS in existence at all times (1 per component?).

The EPS is actually kind enough to contain EpsStats that tells us how many declarations were thunked and how many of those thunks were evaluated.

To answer your question of why HscEnv is retained, it's very simple: forkM operates in the IfL = IOEnv (Env IfGblEnv IfLclEnv) monad. To do unsafeInterleaveIO in such a monad you must remember the current state, i.e. the Env that contains env_top :: HscEnv, thus you create a thunk that retains HscEnv.

The thing I don't quite understand in this setup is what happens to old versions of files? If a file hasn't changed and we reload its iface, presumably we're going to get all the same uniques and completely overwrite the correct leaves of the NameEnv. But what if the uniques changed? How are the environments in the EPS cleared from stale data?

[soulomoon]

Right, I forget the env_top.

The shared NameCache should ensure we reuse the same uniques @mniip , for HLS, it is in the shakeExtra's ideNc

I don't think we remove stuff from EPS. Modules from home package store it's modDetails into HPT(Home package table). You might want to take a look at this as well.