Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add architectures where the transformer is used only for embeddings #163

Draft
wants to merge 26 commits into
base: master
Choose a base branch
from
Draft

Add architectures where the transformer is used only for embeddings #163

wants to merge 26 commits into from

Conversation

lfoppiano
Copy link
Collaborator

@lfoppiano lfoppiano commented Aug 21, 2023 •
edited
Loading

This PR is still work in progress

The goal of this PR is add additional architectures where the Transformer layer is frozen and used only to generate embedding representation.
At the moment the last 4 layers are concatenated together and passed to a Bidirectional LSTM with a dense layer on top. Once the implementation is confirmed, we can experiment with different options.

Currently:

  • when enabling the character channel, the ChainCRF architecture does not work
  • the results are much lower than expected and there are some behaviours that need a review

lfoppiano added 23 commits July 12, 2023 14:41
@lfoppiano
add BERT+BidLSTM and BERT+BidLSTM+CRF base models
d238ac5
@lfoppiano
typo
f634d24
@lfoppiano
Fix LSTM size, add Dense layer for ChainCRF
19493c5
@lfoppiano
add crf related flags in configuration
fcb764a
@lfoppiano
Merge branch 'master' into bert-bidlstm-models
09af3d1
@lfoppiano
reduce the size of the LSTM to try avoiding OOM
536e0de
@lfoppiano
Merge branch 'master' into bert-bidlstm-models
24dcfed
@lfoppiano
freze bert and concatenate embeddings
78c8054
@lfoppiano
freze bert layer
3cd4810
@lfoppiano
add frozen bert to bert_lstm_crf
40f8648
@lfoppiano
update bert_bidlstm_chaincrf
86cddbf
@lfoppiano
update tensorflow addons
f12b8b3
@lfoppiano
reverted wrong committed file
33eed9b
@lfoppiano
hacky solution for selecting the default config values
22240ad
@lfoppiano
fix the learning rate for the hacky solution
51a0d56
@lfoppiano
add examples superconductors
359446c
@lfoppiano
use the same method everywhere to know if a model is using transformers
3e07c97
@lfoppiano
tag startwith
fb509d1
@lfoppiano
remove unused fields
679b9a4
@lfoppiano
add character embedding channel
8976db3
@lfoppiano
LSTM output the same size of a single embedding
5e95960
@lfoppiano
revert change
87144fc
@lfoppiano
update
9ecc400
@lfoppiano lfoppiano requested review from pjox and kermitt2 August 21, 2023 05:26
@lfoppiano lfoppiano added the enhancement New feature or request label Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pjox pjox Awaiting requested review from pjox

@kermitt2 kermitt2 Awaiting requested review from kermitt2

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lfoppiano