Code

Arch/mixtral#169

Code

Open

cm2435 wants to merge 17 commits intounslothai:mainunslothai/unsloth:mainfrom cm2435:arch/mixtralcm2435/unsloth:arch/mixtralCopy head branch name to clipboard

+1,170-12

Commits

Commits on Jan 18, 2024

• Added kernel for layerNORMS from triton tutorials[https://triton-lang.org/main/getting-started/tutorials/05-layer-norm.html]

  cm2435

  committed
  1a21471

Commits on Jan 22, 2024

• formatting and typo fix

  cm2435

  committed
  b60c138

• starting to structure out test suite

  cm2435

  committed
  598dda3

• Contributed relu triton kernl, removed a small amount of boilerplate from the layernorm, added test suite for kernls.

  cm2435

  committed
  3954f16

Commits on Jan 24, 2024

• update test to use common torch.allclose function util

  cm2435

  committed
  48eb887

• impliment forward and backward kernels for GeLU

  cm2435

  committed
  4a7c20c

• fixed incorrect bwd pass for GeLU, wrapped functions in pytorch convience handlers

  cm2435

  committed
  b6d224b

Commits on Feb 4, 2024

• added partial scaling to rope emebeddings

  cm2435

  committed
  e520e9a

• added new seeded dropout kernel

  cm2435

  committed
  cbd31c3

• updated tests

  cm2435

  committed
  b62c886

• formatting

  cm2435

  committed
  e3e41a7

Commits on Feb 6, 2024

• uncommented out init.py

  cm2435

  committed
  440ef5d

• doing work to impliment model

  cm2435

  committed
  a1e2b0d

• updated pre_patch for phi2

  cm2435

  committed
  f2112b1

Commits on Feb 12, 2024

• wrote decoder model fwd for phi2

  cm2435

  committed
  0386c96

• Added torch profiler to look at slowest torch modules

  cm2435

  committed
  11aa5df

• adding tools to be able to profile model fwds to see what to turn into kernels

  cm2435

  committed
  6db5b12