Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Mar 7, 2023

• Merge pull request karpathy#195 from drisspg/enable_sdpa_with_nonzero_dropout

  Show description for 0d8fbd1
  karpathy

  authored
  0d8fbd1

  Copy full SHA for 0d8fbd1

Commits on Mar 5, 2023

• enable sdpa for nonzero dropout

  drisspg

  committed
  6170531

  Copy full SHA for 6170531

Commits on Feb 15, 2023

• Merge pull request karpathy#145 from otaviogood/gradAccumStability

  Show description for ae3a8d5
  karpathy

  authored
  ae3a8d5

  Copy full SHA for ae3a8d5

Commits on Feb 13, 2023

• fix for training stability on single GPU

  otaviogood

  committed
  086ebe1

  Copy full SHA for 086ebe1

Commits on Feb 10, 2023

• fix misinformation in readme

  karpathy

  committed
  55c5069

  Copy full SHA for 55c5069

Commits on Feb 7, 2023

• oops i should not be needing or multiplying by world_size to calculate mfu

  karpathy

  committed
  e58f0cf

  Copy full SHA for e58f0cf

Commits on Feb 6, 2023

• small tweaks, make default WD be 0.1 as is often cited, and remove spurious init of LayerNorm, which is already initialized at 1,0

  karpathy

  committed
  8b1e432

  Copy full SHA for 8b1e432

• bugfix we have to call the raw_model's estimate_mfu ty @jprobichaud for original PR

  karpathy

  committed
  ab21d6c

  Copy full SHA for ab21d6c

Commits on Feb 5, 2023

• also add a sampling/inference section

  karpathy

  committed
  f83dd03

  Copy full SHA for f83dd03

• revamp the readme file to be a bit better and more accessible, i hope

  karpathy

  committed
  23a8e70

  Copy full SHA for 23a8e70

• tune the hyperparams a bit, in configs

  karpathy

  committed
  fce706c

  Copy full SHA for fce706c

• add the estimation of model flops utilization (MFU), a very commonly looked at metric that estimates the token throughput in units of A100 bfloat16 peak flops (312 TFLOPS). this gives us a sense of…

  Show description for ab0718a
  karpathy

  committed
  ab0718a

  Copy full SHA for ab0718a

• oops optimizer now demands to know device_type

  karpathy

  committed
  5809026

  Copy full SHA for 5809026

Commits on Feb 4, 2023

• make more accurate the way in which we count parameters. previous count incorrectly included the positional encoding params, when typically only the number of weight parameters is reported for thes…

  Show description for 34720df
  karpathy

  committed
  34720df

  Copy full SHA for 34720df

• oops forgot to subtract embedding params, which don't enter the 6ND equation

  karpathy

  committed
  3341b4c

  Copy full SHA for 3341b4c

• fix silly error, i don't want to confuse a future GPT training on this notebook in the future

  karpathy

  committed
  5a162bc

  Copy full SHA for 5a162bc

• add reference for 6ND to notebook too

  karpathy

  committed
  0bb96d3

  Copy full SHA for 0bb96d3

• new notebook with a bunch of calculations related to flops and memory of Transformer

  karpathy

  committed
  eae986c

  Copy full SHA for eae986c

• clean up TODOs a bit, they are stale

  karpathy

  committed
  a74e836

  Copy full SHA for a74e836

• mildly dramatic refactor for handing all these usage cases across all possible supported and unsupported devices for all the possible switches and flags

  karpathy

  committed
  25d95db

  Copy full SHA for 25d95db

• very slight refactor, bit cleaner

  karpathy

  committed
  e108ffb

  Copy full SHA for e108ffb

• Merge pull request karpathy#120 from nynyg/remove_cpu_pin_mem

  Show description for dc14989
  karpathy

  authored
  dc14989

  Copy full SHA for dc14989

• Pin memory only when training on GPU

  nynyg

  committed
  b8286f3

  Copy full SHA for b8286f3

• padding 50257 -> 50304 vocab_size, the nerest multiple of 64. the biggest deal smallest optimization i've made in recent past, about 25% faster. this is because the last layer is a major latency bo…

  Show description for 77e7e04
  karpathy

  committed
  77e7e04

  Copy full SHA for 77e7e04

• slight tweak compressing LOC

  karpathy

  committed
  b3c17c6

  Copy full SHA for b3c17c6

• Merge pull request karpathy#116 from ramtingh/master

  Show description for 53d56b8
  karpathy

  authored
  53d56b8

  Copy full SHA for 53d56b8

• Explicitly set ddp device

  ramtingh

  authored
  9da1627

  Copy full SHA for 9da1627

• who needs a dataloader? overlap the prefetching of the next batch with GPU compute, ehiding the data loading latency entirely. this saves about 1ms lol

  karpathy

  committed
  3fd4c0c

  Copy full SHA for 3fd4c0c

• Merge pull request karpathy#115 from akashmjn/akashmjn/fix-notebook-stats

  Show description for 46428d3
  karpathy

  authored
  46428d3

  Copy full SHA for 46428d3

Commits on Feb 3, 2023

• keep only what's needed

  akashmjn

  authored
  d9a7337

  Copy full SHA for d9a7337

• include launch command too. anyone should be able to do this now

  karpathy

  committed
  3969860

  Copy full SHA for 3969860

• add gpt2 training config

  karpathy

  committed
  f9348f3

  Copy full SHA for f9348f3

• add template .gitattributes that fixes language stats

  akashmjn

  authored
  0e2c12b

  Copy full SHA for 0e2c12b

• use the new fused AdamW from pytorch nightly, if available

  karpathy

  committed
  e170e40

  Copy full SHA for e170e40

• Merge pull request karpathy#106 from YassineYousfi/master

  Show description for 7d44bdf
  karpathy

  authored
  7d44bdf

  Copy full SHA for 7d44bdf

Pagination

Previous

Next