Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Jan 16, 2023

• add docs on multinode training to main README too

  karpathy

  committed
  7f74652

  Copy full SHA for 7f74652

• small tweaks to docs and variable names stylistically

  karpathy

  committed
  46ce997

  Copy full SHA for 46ce997

• clarify that these should be run on two separate machines

  karpathy

  committed
  684800d

  Copy full SHA for 684800d

• docs for multinode ddp

  karpathy

  committed
  9352df2

  Copy full SHA for 9352df2

• get rid of gpu_id, the world is more complicated than that when world_size > 8

  karpathy

  committed
  c3dddbf

  Copy full SHA for c3dddbf

• local rank -> rank

  karpathy

  committed
  f5e6ac8

  Copy full SHA for f5e6ac8

Commits on Jan 15, 2023

• add gradient accumulation support to simulate larger batch sizes. ty @VHellendoorn for original PR

  karpathy

  committed
  cf99914

  Copy full SHA for cf99914

Commits on Jan 14, 2023

• add note of caution for the produced warning, investigate later

  karpathy

  committed
  89da79e

  Copy full SHA for 89da79e

• a bit better settings... for a single gpu at least. these settings would fry a simple cpu though i think

  karpathy

  committed
  7d7ded2

  Copy full SHA for 7d7ded2

• fix bug... if topk > vocab_size, torch.topk will throw error

  karpathy

  committed
  91d0251

  Copy full SHA for 91d0251

• correctly propagate the vocab_size from the rendered dataset into the model args

  karpathy

  committed
  57735f5

  Copy full SHA for 57735f5

• reverse the order, making sure that the final layer init is preserved, and becomes the token embedding instead of the other way around. otherwise the loss can be all messed up from a bad init

  karpathy

  committed
  43b37fd

  Copy full SHA for 43b37fd

• tie the weights of lm_head.weight and transformer.wte.weight, i.e. the last linear layer of decoder and the token embeddings.

  karpathy

  committed
  7c82885

  Copy full SHA for 7c82885

Commits on Jan 13, 2023

• it's true

  karpathy

  committed
  32b4f08

  Copy full SHA for 32b4f08

• more scaling laws, clarification, and add simple interpolation of Approach 2

  karpathy

  committed
  3e0fd42

  Copy full SHA for 3e0fd42

Commits on Jan 12, 2023

• inference time mini-optimization low-hanging fruit ty @jxtps for raising: when we are running inference we can apply lm_head on only the very last token

  karpathy

  committed
  8f85b83

  Copy full SHA for 8f85b83

• meant to set always_save_checkpoint to False instead, so we only write when val improves

  karpathy

  committed
  e21cbf8

  Copy full SHA for e21cbf8

• including transformers as a dependency of the repo as well

  karpathy

  committed
  c1ac2d5

  Copy full SHA for c1ac2d5

• add note about windows and pytorch 2.0 and torch compile in general

  karpathy

  committed
  7f51d17

  Copy full SHA for 7f51d17

Commits on Jan 11, 2023

• oh no nanoGPT is trending quickly explain the character-level functionality I added late last night

  karpathy

  committed
  bb49751

  Copy full SHA for bb49751

• add support for character-level language models, a new character-level shakespeare dataset, a new config file that shows how to train a character-level baby GPT on it, and adjust the sample functio…

  Show description for d17350a
  karpathy

  committed
  d17350a

  Copy full SHA for d17350a

• guess the config from globals() and log all of it with wandb

  karpathy

  committed
  c2a402f

  Copy full SHA for c2a402f

Commits on Jan 8, 2023

• adjust the readme to reflect changes in the autocast branch

  karpathy

  committed
  8b2e622

  Copy full SHA for 8b2e622

• copy pasting what seems to work to bench,sample as well. ty @lantiga

  karpathy

  committed
  b77c2e8

  Copy full SHA for b77c2e8

• add device and dtype support to train.py args

  karpathy

  committed
  a855d31

  Copy full SHA for a855d31

• Merge pull request karpathy#20 from lantiga/wandb-optional-import

  Show description for e7cd674
  karpathy

  authored
  e7cd674

  Copy full SHA for e7cd674

• Move conditional import

  lantiga

  committed
  09f1f45

  Copy full SHA for 09f1f45

• Make wandb import conditioned to wandb_log=True

  lantiga

  committed
  aba47f0

  Copy full SHA for aba47f0

• ran readme through spellchecker heh

  karpathy

  committed
  e53b9d2

  Copy full SHA for e53b9d2

• tune the readme with new header image and the loss curve for 124M

  karpathy

  committed
  df3b8a5

  Copy full SHA for df3b8a5

Commits on Jan 7, 2023

• progress! based on chinchilla author correspondence

  karpathy

  committed
  d56bdf0

  Copy full SHA for d56bdf0

Commits on Jan 6, 2023

• small tweaks to notebook

  karpathy

  committed
  27fc6a4

  Copy full SHA for 27fc6a4

• update scaling laws. basically i can't reproduce any of params, flops, or scaling laws of the Chinchilla paper atm...

  karpathy

  committed
  69d1a5f

  Copy full SHA for 69d1a5f

Commits on Jan 5, 2023

• minor args re-arranging and removing some spurious ones like wandb entity ty @tcapelle

  karpathy

  committed
  9629093

  Copy full SHA for 9629093

• Merge pull request karpathy#19 from nat/patch-1

  Show description for 529c967
  karpathy

  authored
  529c967

  Copy full SHA for 529c967

Pagination

Previous

Next