Commits

Branch selector

 master

User selector

All users

Datepicker

All time

Commit History

Commits on Sep 7, 2021

• tweak eps for causal linear attn

  lucidrains

  committed
  24ecf20

  Copy full SHA for 24ecf20

Commits on Aug 21, 2021

• add token shifting feature

  lucidrains

  committed
  734e304

  Copy full SHA for 734e304

Commits on Apr 14, 2021

• 0.18.1

  lucidrains

  authored
  dd3f7b6

  Copy full SHA for dd3f7b6

• fix stability issues

  lucidrains

  authored
  a03db94

  Copy full SHA for a03db94

Commits on Apr 3, 2021

• add rotary embeddings

  lucidrains

  committed
  6dfc9a1

  Copy full SHA for 6dfc9a1

Commits on Mar 30, 2021

• fix bug with context and D_inv

  lucidrains

  committed
  dc2a661

  Copy full SHA for dc2a661

• remove one kv head, and make sure causal linear attention is normalized by D_inv

  lucidrains

  committed
  c12825c

  Copy full SHA for c12825c

Commits on Mar 29, 2021

• fix leakage bug

  lucidrains

  committed
  a0db8ba

  Copy full SHA for a0db8ba

• make sure no blindspot is supported

  lucidrains

  committed
  843403d

  Copy full SHA for 843403d

• fix leakage when blindspot size is 1

  lucidrains

  committed
  5c1ecfe

  Copy full SHA for 5c1ecfe

Commits on Mar 25, 2021

• scale after norm

  lucidrains

  committed
  354f610

  Copy full SHA for 354f610

Commits on Jan 2, 2021

• update gcd to be used from math package, for python 3.9 compatibility

  lucidrains

  committed
  b169815

  Copy full SHA for b169815

Commits on Dec 20, 2020

• numerical stability for causal linear attn

  lucidrains

  committed
  c6eeb1b

  Copy full SHA for c6eeb1b

Commits on Dec 12, 2020

• fix axial positional embedding if max sequence length is smaller than local attention window size, also allow one to use absolute positional embedding with flag

  lucidrains

  committed
  f6ec175

  Copy full SHA for f6ec175

Commits on Oct 29, 2020

• add a final norm after attention layers

  lucidrains

  committed
  405721a

  Copy full SHA for 405721a

Commits on Oct 20, 2020

• fix autopadder for linear attention transformer (not LM only)

  lucidrains

  committed
  9957a7b

  Copy full SHA for 9957a7b

Commits on Oct 9, 2020

• recent linear papers seem to suggest that softmax normalization for keys is enough

  lucidrains

  committed
  8cdcce4

  Copy full SHA for 8cdcce4

Commits on Oct 5, 2020

• allow one to turn off normalization of queries for image linear attention, given reported better results from https://openreview.net/forum?id=KiFeuZu24k

  lucidrains

  committed
  1033093

  Copy full SHA for 1033093

Commits on Aug 5, 2020

• do not assert that dimension is divisible by number of heads if dim_head is supplied

  lucidrains

  committed
  b8014a5

  Copy full SHA for b8014a5

Commits on Jul 13, 2020

• update readme

  lucidrains

  committed
  0cca90f

  Copy full SHA for 0cca90f

Commits on Jul 12, 2020

• be able to set the dimension of each head in multi-head attention. new paper shows that performance can scale with number of heads as long as dimension is fixed and decoupled from embedding dimension

  lucidrains

  committed
  5f89ed6

  Copy full SHA for 5f89ed6

Commits on Jul 8, 2020

• mask values as well, just to be safe

  lucidrains

  committed
  5190577

  Copy full SHA for 5190577

Commits on Jul 5, 2020

• move local attention to separate library

  lucidrains

  committed
  31ec103

  Copy full SHA for 31ec103

Commits on Jun 30, 2020

• update citation

  lucidrains

  committed
  bf36cc0

  Copy full SHA for bf36cc0

Commits on Jun 29, 2020

• make sure to multiply by scale sqrt(dim) for query and keys

  lucidrains

  committed
  e1dc86f

  Copy full SHA for e1dc86f

• hide one key/value head setting from readme for now, due to reported (and confirmed) degraded results

  lucidrains

  committed
  c25d45a

  Copy full SHA for c25d45a

• make sure, when using one key/value head, that local and global attention gets their own separate k/v head

  lucidrains

  committed
  75a6cef

  Copy full SHA for 75a6cef

• make sure linformer self attention receives dropout

  lucidrains

  committed
  97afab2

  Copy full SHA for 97afab2

• make sure linformer is added to requirements

  lucidrains

  committed
  98efbd6

  Copy full SHA for 98efbd6

Commits on Jun 28, 2020

• update readme

  lucidrains

  committed
  500c31c

  Copy full SHA for 500c31c

• add linformer

  lucidrains

  committed
  665309a

  Copy full SHA for 665309a

Commits on Jun 21, 2020

• Create python-publish.yml

  lucidrains

  authored
  ad6d795

  Copy full SHA for ad6d795

• add all dropouts except for q(kv) attention dropout

  lucidrains

  committed
  42af16e

  Copy full SHA for 42af16e

Commits on Jun 12, 2020

• add ability for ImageLinearAttention to accept contextual key values

  lucidrains

  committed
  f04e007

  Copy full SHA for f04e007

Commits on Jun 11, 2020

• make ImageLinearAttention behave more like conv class

  lucidrains

  committed
  c129867

  Copy full SHA for c129867

Pagination

Previous

Next