Skip to content

Tags: TrustLLMeu/Megatron-LM

Tags

trustllm-v0.3.6

Toggle trustllm-v0.3.6's commit message
Add safety check

trustllm-v0.3.5

Toggle trustllm-v0.3.5's commit message
Do not include `mmap`s in `_IndexReader` pickle

Shouldn't affect `IndexedDataset`, since that, in turn, doesn't include
the `_IndexReader` in its pickle, but this way it's just safer and more
future-proof.

trustllm-v0.3.4

Toggle trustllm-v0.3.4's commit message
Do not include `mmap`ed objects in `pickle`ing

We rewrite the `pickle` protocol for the datasets because this avoids
running out of memory on some machines.