Commits

Breadcrumbs

History for

onmaster

User selector

All users

Datepicker

All time

Commit History

Commits on Feb 7, 2023

• oops i should not be needing or multiplying by world_size to calculate mfu

  karpathy

  committed
  e58f0cf

  Copy full SHA for e58f0cf

Commits on Feb 6, 2023

• small tweaks, make default WD be 0.1 as is often cited, and remove spurious init of LayerNorm, which is already initialized at 1,0

  karpathy

  committed
  8b1e432

  Copy full SHA for 8b1e432

• bugfix we have to call the raw_model's estimate_mfu ty @jprobichaud for original PR

  karpathy

  committed
  ab21d6c

  Copy full SHA for ab21d6c

Commits on Feb 5, 2023

• add the estimation of model flops utilization (MFU), a very commonly looked at metric that estimates the token throughput in units of A100 bfloat16 peak flops (312 TFLOPS). this gives us a sense of…

  Show description for ab0718a
  karpathy

  committed
  ab0718a

  Copy full SHA for ab0718a

Commits on Feb 4, 2023

• clean up TODOs a bit, they are stale

  karpathy

  committed
  a74e836

  Copy full SHA for a74e836

• mildly dramatic refactor for handing all these usage cases across all possible supported and unsupported devices for all the possible switches and flags

  karpathy

  committed
  25d95db

  Copy full SHA for 25d95db

• very slight refactor, bit cleaner

  karpathy

  committed
  e108ffb

  Copy full SHA for e108ffb

• Pin memory only when training on GPU

  nynyg

  committed
  b8286f3

  Copy full SHA for b8286f3

• padding 50257 -> 50304 vocab_size, the nerest multiple of 64. the biggest deal smallest optimization i've made in recent past, about 25% faster. this is because the last layer is a major latency bo…

  Show description for 77e7e04
  karpathy

  committed
  77e7e04

  Copy full SHA for 77e7e04

• slight tweak compressing LOC

  karpathy

  committed
  b3c17c6

  Copy full SHA for b3c17c6

• Explicitly set ddp device

  ramtingh

  authored
  9da1627

  Copy full SHA for 9da1627

• who needs a dataloader? overlap the prefetching of the next batch with GPU compute, ehiding the data loading latency entirely. this saves about 1ms lol

  karpathy

  committed
  3fd4c0c

  Copy full SHA for 3fd4c0c

Commits on Feb 3, 2023

• Merge pull request karpathy#106 from YassineYousfi/master

  Show description for 7d44bdf
  karpathy

  authored
  7d44bdf

  Copy full SHA for 7d44bdf

Commits on Feb 2, 2023

• change grad accum to default off because i think it just confuses everyone

  karpathy

  committed
  d8b1a94

  Copy full SHA for d8b1a94

Commits on Feb 1, 2023

• use the enabled arg in GradScaler

  YassineYousfi

  committed
  40f4d6f

  Copy full SHA for 40f4d6f

Commits on Jan 31, 2023

• rename iter to it, because iter is a concrete Python builtin

  karpathy

  committed
  038ce89

  Copy full SHA for 038ce89

Commits on Jan 30, 2023

• merge, make cleaner, careful with gradient clipping when using grad scaler fp16 training

  karpathy

  committed
  924a087

  Copy full SHA for 924a087

• based on my experiments these biases are indeed not needed. code runs faster, identical results. keeping the option just because it deviates from the gpt-2 setup

  karpathy

  committed
  0e90ee9

  Copy full SHA for 0e90ee9

Commits on Jan 27, 2023

• stay true to the README file and set grad accum to 5, so the default batch size is about 0.5M and is reproducing gpt2

  karpathy

  committed
  001c1e7

  Copy full SHA for 001c1e7

• let me set bias=True until I validate it properly, but this should be ok to merge to master for now, is equivalent to previous functionality

  karpathy

  committed
  79dbe00

  Copy full SHA for 79dbe00

• bunch of plumbing of bias all around. measuring bias=False to be about 6% faster

  karpathy

  committed
  e808a67

  Copy full SHA for e808a67

• grad clipping seems to slightly speed up training in the beginning but i can't see a big difference later in the training. it costs non-negligeable compute to clip. adding it for now because it is …

  Show description for 3cb3fc0
  karpathy

  committed
  3cb3fc0

  Copy full SHA for 3cb3fc0

Commits on Jan 24, 2023

• use GradScaler in model only if dtype is float16

  johnwildauer

  committed
  e0e94a1

  Copy full SHA for e0e94a1

Commits on Jan 20, 2023

• Merge pull request karpathy#71 from cchan/patch-1

  Show description for 3611338
  karpathy

  authored
  3611338

  Copy full SHA for 3611338

• make mentions of mps in docs. ty good people in issue karpathy#28

  karpathy

  committed
  1f77d03

  Copy full SHA for 1f77d03

• Zero-grad more aggressively to save memory

  cchan

  authored
  6716607

  Copy full SHA for 6716607

Commits on Jan 16, 2023

• small tweaks to docs and variable names stylistically

  karpathy

  committed
  46ce997

  Copy full SHA for 46ce997

• clarify that these should be run on two separate machines

  karpathy

  committed
  684800d

  Copy full SHA for 684800d

• docs for multinode ddp

  karpathy

  committed
  9352df2

  Copy full SHA for 9352df2

• get rid of gpu_id, the world is more complicated than that when world_size > 8

  karpathy

  committed
  c3dddbf

  Copy full SHA for c3dddbf

• local rank -> rank

  karpathy

  committed
  f5e6ac8

  Copy full SHA for f5e6ac8

Commits on Jan 15, 2023

• add gradient accumulation support to simulate larger batch sizes. ty @VHellendoorn for original PR

  karpathy

  committed
  cf99914

  Copy full SHA for cf99914

Commits on Jan 14, 2023

• correctly propagate the vocab_size from the rendered dataset into the model args

  karpathy

  committed
  57735f5

  Copy full SHA for 57735f5

Commits on Jan 12, 2023

• inference time mini-optimization low-hanging fruit ty @jxtps for raising: when we are running inference we can apply lm_head on only the very last token

  karpathy

  committed
  8f85b83

  Copy full SHA for 8f85b83

Commits on Jan 11, 2023

• add support for character-level language models, a new character-level shakespeare dataset, a new config file that shows how to train a character-level baby GPT on it, and adjust the sample functio…

  Show description for d17350a
  karpathy

  committed
  d17350a

  Copy full SHA for d17350a

Pagination

Previous

Next