Skip to content

Commit

Permalink
update performance
Browse files Browse the repository at this point in the history
  • Loading branch information
@AlexYangLi
AlexYangLi committed Jan 30, 2019
1 parent 8159ce6 commit 61880ec
Showing 1 changed file with 29 additions and 28 deletions.
57 changes: 29 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -143,39 +143,40 @@ python3 train.py
| < aspect_text_char_len = 0.99 | 16 |

### Performance
Note: results in the parenthese is the performance of models with word embeddings fixed (aspect embeddings fine tuned)
Note: results in the `()` are the performances of models with word embeddings fixed but aspect embeddings fine tuned; results in the `[]`
are the performances of models with noth word embeddings and aspect embeddings fixed

- Accuracy

| model | laptop(paper) | laptop | restaurant(paper) | restaurant | twitter(paper) | twitter |
|---------|---------------|-----------------|-------------------|----------------|----------------|------------------|
|td_lstm | | 0.69122(0.7225)| | 0.7732(0.7875)| 0.708 | 0.69508(0.7182) |
|tc_lstm | | 0.68652(0.6833)| | 0.7642(0.7687)| 0.715 | 0.70379(0.72398)|
|ae_lstm | 0.689 | 0.67398(0.6834)| 0.766 | 0.7598(0.7571)| | 0.6878(0.6936) |
|at_lstm | | 0.68181(0.7179)| | 0.7669(0.7696)| | 0.6575(0.70520) |
|atae_lstm| 0.687 | 0.68025(0.6849)| 0.772 | 0.7598(0.7607)| | 0.68061(0.69508)|
|memnet | 0.7237 | 0.5329(0.52978)| 0.8095 | 0.6508(0.6508)| | 0.57803(0.5606) |
|ram | 0.7449 | 0.7021(0.7210) | 0.8023 | 0.7866(0.7946)| 0.6936 | 0.69653(0.71242)|
|ian | 0.721 | 0.6912(0.6927) | 0.786 | 0.7758(0.7892)| | 0.6835(0.71242) |
|cabasc | 0.7507 | | 0.8089 | | 0.7153 | |
| model | laptop(paper) | laptop | restaurant(paper) | restaurant | twitter(paper) | twitter |
|---------|---------------|---------------------------|-------------------|--------------------------|----------------|--------------------------|
|td_lstm | | 0.69905(0.7194) | | 0.7732(0.8008) | 0.708 | 0.69219(0.7109) |
|tc_lstm | | 0.69122(0.6912)[0.7178] | | 0.7642(0.7696)[0.79375] | 0.715 | 0.71387(0.7124)[0.72543]|
|ae_lstm | 0.689 | 0.69122(0.6974)[0.69749] | 0.766 | 0.7651(0.7625)[0.78125] | | 0.68497(0.68641)[0.6820]|
|at_lstm | | 0.69122(0.6880)[0.70689] | | 0.7678(0.7812)[0.78571] | | 0.67196(0.7052)[0.70086]|
|atae_lstm| 0.687 | 0.69749(0.6927)[0.70376] | 0.772 | 0.7732(0.7732)[0.78482] | | 0.66907(0.6965)[0.6921] |
|memnet | 0.7237 | 0.52821(0.5266)[0.53448] | 0.8095 | 0.6580(0.65)[0.65803] | | 0.57369(0.5751)[0.5780] |
|ram | 0.7449 | 0.70376(0.7225)[0.72727] | 0.8023 | 0.7937(0.8071)[0.81607] | 0.6936 | 0.69653(0.71387)[0.6979]|
|ian | 0.721 | 0.6865(0.7100) | 0.786 | 0.7732(0.7821) | | 0.68208(0.71965) |
|cabasc | 0.7507 | 0.6974(0.6990)[0.67554] | 0.8089 | 0.7919(0.8080)[0.80357] | 0.7153 | 0.69219(0.69508)[0.6690]|

- Macro-F1

| model | laptop(paper) | laptop | restaurant(paper) | restaurant | twitter(paper) | twitter |
|---------|---------------|-----------------|-------------------|----------------|----------------|------------------|
|td_lstm | | 0.62223(0.6667)| | 0.6623(0.6836)| 0.690 | 0.6783(0.70238) |
|tc_lstm | | 0.62287(0.6223)| | 0.6022(0.6651)| 0.695 | 0.6797(0.70639) |
|ae_lstm | | 0.60334(0.6159)| | 0.6365(0.6300)| | 0.6638(0.66873) |
|at_lstm | | 0.61957(0.6564)| | 0.6630(0.6451)| | 0.6553(0.67674) |
|atae_lstm| | 0.6172(0.63431)| | 0.6096(0.6430)| | 0.6629(0.67799) |
|memnet | | 0.40214(0.3538)| | 0.3339(0.3011)| | 0.5096(0.49457) |
|ram | 0.7135 | 0.6474(0.6794) | 0.7080 | 0.6855(0.6915)| 0.6730 | 0.6769(0.6873) |
|ian | | 0.62409(0.6306)| | 0.6675(0.6800)| | 0.65373(0.70094)|
|cabasc | | | | | | |
| model | laptop(paper) | laptop | restaurant(paper) | restaurant | twitter(paper) | twitter |
|---------|---------------|---------------------------|-------------------|--------------------------|----------------|--------------------------|
|td_lstm | | 0.64172(0.6636) | | 0.6653(0.6986) | 0.690 | 0.6746(0.6898) |
|tc_lstm | | 0.62847(0.6335)[0.6666] | | 0.6651(0.6492)[0.70170] | 0.695 | 0.6983(0.6848)[0.7122] |
|ae_lstm | | 0.62486(0.6435)[0.6355] | | 0.6419(0.6370)[0.68319] | | 0.66644(0.6670)[0.65437]|
|at_lstm | | 0.61267(0.6259)[0.65907] | | 0.6630(0.6689)[0.6676] | | 0.6553(0.68194)[0.6823] |
|atae_lstm| | 0.6246(0.63185)[0.6539] | | 0.6505(0.6657)[0.68006] | | 0.6651(0.67400)[0.67085]|
|memnet | | 0.38574(0.4118)[0.2322] | | 0.3547(0.4040)[0.3395] | | 0.48982(0.49209)[0.4989]|
|ram | 0.7135 | 0.6474(0.6747)[0.66476] | 0.7080 | 0.6887(0.7035)[0.72720] | 0.6730 | 0.66536(0.69679)[0.6685]|
|ian | | 0.62007(0.6604) | | 0.6550(0.6768) | | 0.65071(0.69550) |
|cabasc | | 0.64601(0.6356)[0.5886] | | 0.6815(0.7201)[0.69357] | | 0.66790(0.67948)[0.6610]|

- Personal conclusion
1. I found `AT-LSTM` is always better than `AE-LSTM` & `ATAE-LSTM`. Actually it's not just on SemEval14 & twitter data, but also many other sentiment analysis data.
2. Surprisingly, I failed to achieved similar performance as stated in the parper of `Memnet`. Or maybe there are bugs in the code?
3. `TD-LSTM` performs unexpectedly well.
4. Models with fixed word embeddings are generally better than those with fine-tuned word embeddings, which is consistent with the paper of `RAM`.
5. L2 regularization didn't help.

1. Surprisingly, I failed to achieved similar performance as stated in the parper of `Memnet`. Or maybe there are bugs in the code?
2. `TD-LSTM` performs unexpectedly well.
3. Models with fixed embeddings are generally better than those with fine-tuned embeddings, which is consistent with the paper of `RAM`.
4. L2 regularization didn't help.

0 comments on commit 61880ec

Please sign in to comment.