Commits

Branch selector

 main

User selector

n1t0

Datepicker

All time

Commit History

Commits on Feb 10, 2021

• Fix tokenizers training in notebook (huggingface#10110)

  n1t0

  authored
  1fbaa3c

  Copy full SHA for 1fbaa3c

Commits on Feb 8, 2021

• Update tokenizers requirement (huggingface#10077)

  n1t0

  authored
  f285e4c

  Copy full SHA for f285e4c

Commits on Jan 18, 2021

• Remove duplicated extra["retrieval"] (huggingface#9621)

  n1t0

  authored
  72fc9ab

  Copy full SHA for 72fc9ab

Commits on Oct 23, 2020

• Fix BatchEncoding.word_to_tokens for removed tokens (huggingface#7939)

  n1t0

  authored
  5e32301

  Copy full SHA for 5e32301

Commits on Jul 6, 2020

• Various tokenizers fixes (huggingface#5558)

  Show description for 5787e4c
  n1t0

  authored
  5787e4c

  Copy full SHA for 5787e4c

Commits on Jun 25, 2020

• Fix convert_graph_to_onnx (huggingface#5230)

  n1t0

  authored
  0e1fce3

  Copy full SHA for 0e1fce3

Commits on Jun 15, 2020

• [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (huggingface#4510)

  Show description for 3643422
  

  3 peopleauthored
  3643422

  Copy full SHA for 3643422

Commits on May 28, 2020

• Fix add_special_tokens on fast tokenizers (huggingface#4531)

  n1t0

  authored
  5e73701

  Copy full SHA for 5e73701

Commits on May 22, 2020

• Fix convert_token_type_ids_from_sequences for fast tokenizers (huggingface#4503)

  n1t0

  authored
  35df911

  Copy full SHA for 35df911

Commits on Apr 22, 2020

• Bump tokenizers version to final 0.7.0 (huggingface#3898)

  n1t0

  authored
  13dd2ac

  Copy full SHA for 13dd2ac

Commits on Apr 10, 2020

• Update tokenizers to 0.7.0-rc5 (huggingface#3705)

  n1t0

  authored
  b7cf9f4

  Copy full SHA for b7cf9f4

Commits on Dec 27, 2019

• Hotfix tokenizers version for sdist installs

  n1t0

  committed
  bfe870b

  Copy full SHA for bfe870b

Commits on Dec 26, 2019

• Better added_tokens handling

  n1t0

  committed
  e6ec24f

  Copy full SHA for e6ec24f

• Code style update

  n1t0

  committed
  599db13

  Copy full SHA for 599db13

• Handle unk_token

  Show description for 835b76a
  n1t0

  committed
  835b76a

  Copy full SHA for 835b76a

• FastPreTrainedTokenizer => PreTrainedTokenizerFast

  n1t0

  committed
  7ead04c

  Copy full SHA for 7ead04c

• Update for changes in tokenizers API

  n1t0

  committed
  1f82a5d

  Copy full SHA for 1f82a5d

Commits on Dec 24, 2019

• tokenizers is now a real dependency

  n1t0

  committed
  734d29b

  Copy full SHA for 734d29b

• Add tests for fast tokenizers

  n1t0

  committed
  2818e50

  Copy full SHA for 2818e50

• Fix style

  n1t0

  committed
  31c56f2

  Copy full SHA for 31c56f2

• BertTokenizerFast

  n1t0

  committed
  951ae99

  Copy full SHA for 951ae99

• GPT2TokenizerFast

  n1t0

  committed
  041eac2

  Copy full SHA for 041eac2

• FastPreTrainedTokenizer

  n1t0

  committed
  3471ff0

  Copy full SHA for 3471ff0

Commits on Jul 31, 2019

• Fix small typos

  n1t0

  committed
  f2a3eb9

  Copy full SHA for f2a3eb9