This repository houses an EMG gesture recognition tutorial by Evan Campbell. In this tutorial, both handcrafted and deep learning approaches for gesture recognition are covered in Python. A provided dataset can be downloaded following the instructions found in Data/Raw_Data/instructions.txt.
In order to follow the tutorial, first the main_construct_dataset.py function must be run to prepare the recorded gestures (.csv files) into windowed segments (.npy files). This script also performs filtering typically used in EMG gesture recognition. A notch filter was used to remove power line interference (60 Hz noise), and a bandpass filter was used with a pass band between 20-450 Hz to remove motion artefact contaminants and electrical component noise while retaining the majority of the EMG signal energy. With the current specified values, windows were formed using 250 ms length and 100 ms increment (150 ms overlap).
For the handcrafted features, a number of common feature sets from EMG literature were used. The Hudgins time domain feature set was used, which contains 4 features per channel (mean absolute value, zero crossings, slope sign change, and waveform length). The time domain power spectral descriptors (Khushaba's feature set) was also included which contains 6 features per channel (normalized 0th order moment, normalized 2nd order moment, normalized 4th order moment, sparsity, irregularity factor, and waveform length ratio). The low sampling frequency 4 feature set was also used which contains (l-score, maximum fractal length, mean squared ratio, and willison's amplitude). The low sampling frequency 9 feature set extends the previous feature set by adding 5 more features per channel (zero crossings, root mean square, integral of EMG, DASDV, and variance).
For the deep learning pipelines, a simple architecture was used which has 3 convolutional blocks followed by a linear layer. The convolutional blocks all contained a convolutional layer, batch normalization, ReLU activation function, and dropout. Negative log likelihood loss was used alongside the Adam optimizer during training. The learning rate was initialized to 0.005 and a scheduler adapted the learning rate by monitoring the validation loss throughout over the epochs.
-
Within Subject Handcrafted Feature Pipeline (main_withinsubject_handcrafted.py)
-
Within Subject Deep learning Pipeline Using Convolutional Neural Networks (main_withinsubject_deeplearning.py)
-
Between Subject Handcrafted Feature Pipelin (main_betweensubject_handcrafted.py)
-
Between Subject Handcrafted Features Pipeline with Projection Techniques (main_betweensubject_handcraftedcca.py - in progress): Prior to 2020, the state-of-the-art technique for achieving high performance for between subject gesture recognition relied on canonical correlation analysis (See Khushaba et al 2015).
-
Between Subject Deep Learning Pipeline Using Convolutional Neural Networks
- Using a single subject to train the deep learning model for another subject (main_betweensubject_deeplearning.py)
- Using all other subjects to train the deep learning model for a particular subject (main_pooledsubject_deeplearning.py)
-
Subject-Independent Adaptive Domain Adversarial Neural Network (main_subjectindependent_ADANN.py - in progress) This technique builds a model that is well suited for many subjects. This was a breakthrough towards between subject gesture recognition, but at this point a full acquisition protocol was still required by all end users (Cote-Allard et al 2020).
-
Between Subject Adaptive Domain Adversarial Neural Network (main_betweensubject_ADANN.py - in progress) This technique builds off the subject-independent adaptive domain adversarial neural network but has a mechanism to adapt to a subject that only provides a single repetition (Campbell et al 2021).
Subject | TD | TDPSD | LSF4 | LSF9 | CNN |
---|---|---|---|---|---|
S0 | 96.28059405065319 | 98.56654908559705 | 98.35276652018267 | 98.7307863078328 | 99.41452297815479 |
S1 | 92.51591477500472 | 95.14179058412597 | 95.20592566635236 | 96.12663088881233 | 91.98451775994974 |
S2 | 79.98360633675084 | 87.80965845505101 | 87.40853755760294 | 88.49376938758729 | 82.84690424656084 |
S3 | 94.27212213215147 | 97.81476234326948 | 96.9501681444503 | 97.38578874842342 | 98.34340933983604 |
S4 | 90.2387268463126 | 92.31312443007117 | 92.97773680298462 | 94.35339496396251 | 92.3039876748737 |
S5 | 84.74000178134908 | 88.52049651157299 | 88.1928249760174 | 88.39310135008016 | 88.94057710914986 |
S6 | 68.76059863812824 | 73.57149850658146 | 71.9219018079592 | 74.72747812834588 | 71.05533519526878 |
S7 | 82.91067080347233 | 88.69695863046176 | 87.75371866558746 | 87.96901335158972 | 89.80586879149514 |
S8 | 89.0038670320066 | 92.16268090014776 | 92.7841280938596 | 93.19802624276724 | 90.88853727483391 |
S9 | 92.24744750289952 | 95.7731074458428 | 95.40857665655058 | 96.16485755785428 | 95.6602616884948 |
Mean | 87.09535498987285 | 91.03706268927213 | 90.69562848915473 | 91.55428469272556 | 90.12439220586175 |
STD | 7.842983256183912 | 6.870777987868703 | 7.266719880825716 | 6.734721042939276 | 7.81916163111671 |
TIME (ms) | 0.03975517799803106 | 0.07131806242423407 | 1.3518062178069783 | 1.4118450304628045 | -- |
train \ test | S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
S0 | NA | 42.89077548690876 | 43.151809292698594 | 20.682759605770602 | 47.20638540478905 | 21.256417569880206 | 19.85719385933595 | 15.434658278524449 | 25.162534828891907 | 17.212354661530778 | 28.094987665370027 |
S1 | 35.48594262879977 | NA | 15.544928984369424 | 48.17883159548636 | 53.35661345496009 | 50.54192812321734 | 30.396287040342735 | 18.66599942808121 | 43.43787954561692 | 36.586061773307655 | 36.910496952686835 |
S2 | 44.79092336235194 | 4.80131269173147 | NA | 17.75460648478789 | 6.021949828962372 | 12.69965772960639 | 15.158871831488755 | 16.764369459536745 | 18.91833964420947 | 18.011270418717455 | 17.213477939043614 |
S3 | 30.569430569430565 | 51.58022401369765 | 13.039754478623939 | NA | 35.81812998859749 | 56.26069594980034 | 39.564441270974655 | 27.6880183014012 | 51.23955133242838 | 38.005563877594696 | 38.19620108694988 |
S4 | 57.00727843584986 | 49.611186416494256 | 22.7535507815288 | 17.040422796743325 | NA | 23.7592698231603 | 22.14923241699393 | 15.885044323706033 | 27.541616060584413 | 14.423282687780869 | 27.796764860315754 |
S5 | 31.53275296132439 | 68.1743597060712 | 11.869245592748555 | 62.49107270389944 | 52.12371721778791 | NA | 45.091038914673334 | 28.610237346296824 | 67.00007144388083 | 31.86389899422213 | 44.306266097878286 |
S6 | 18.90252604538319 | 69.4513804665763 | 3.1261151952037687 | 56.599057277531784 | 45.70980615735461 | 61.986594409583574 | NA | 34.265084358021156 | 54.94034435950561 | 40.89450032099294 | 42.87504539890588 |
S7 | 23.026973026973028 | 33.07412427766284 | 9.692384554992506 | 52.77103270961291 | 27.99315849486887 | 48.60239589275528 | 42.056408425562296 | NA | 31.813960134314495 | 53.370425850631285 | 35.82231815193039 |
S8 | 48.865420293991725 | 71.74859099664694 | 19.27057312111912 | 45.100699900014284 | 42.524230330672744 | 56.538790644609236 | 34.45912174223491 | 16.84300829282242 | NA | 20.572080747556885 | 39.54694622996314 |
S9 | 30.091337234194377 | 48.86209602625384 | 5.609877953036899 | 38.90158548778746 | 22.776510832383124 | 33.620935539075866 | 32.666904676901105 | 29.64684014869888 | 25.562620561548904 | NA | 29.74874538443116 |
Mean | 29.723319435821086 | 38.0860959860633 | 38.64562271890412 | 33.37624007692886 | 27.136524113546834 | 30.444536169390215 | 37.123084462149464 | 36.39959669719822 | 39.69403216267247 | 29.882197944800403 |
train \ test | S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
S0 | NA | 33.13833202539773 | 41.01777175076726 | 23.49664333666619 | 48.19697833523375 | 29.834569309754706 | 23.455908604069975 | 13.382899628252787 | 26.705722654854615 | 15.086668093301947 | 28.257277082033212 |
S1 | 39.61038961038961 | NA | 19.498965098850903 | 73.32523925153549 | 55.865165336374 | 59.72618368511123 | 40.01428061406641 | 29.217901058049755 | 59.46274201614632 | 42.17133889721093 | 46.54357839641496 |
S2 | 53.51791066076781 | 1.462509809517015 | NA | 10.65562062562491 | 18.635974914481185 | 4.171420422133486 | 8.211353088182792 | 1.8944809837003145 | 8.294634564549547 | 3.2455952635708685 | 12.232166703614212 |
S3 | 26.77322677322677 | 58.57173432260826 | 5.659838698165727 | NA | 21.38683010262258 | 51.88248716486024 | 34.75901463762942 | 42.82956820131542 | 46.75287561620347 | 41.57928525572437 | 36.68831786359514 |
S4 | 61.20308263165406 | 76.87807662124563 | 17.82171151238313 | 32.78103128124554 | NA | 52.14632059326868 | 39.12888254194931 | 21.10380325993709 | 48.910480817317996 | 25.058848705328487 | 41.67024866270332 |
S5 | 50.706436420722135 | 74.11714346864522 | 15.059596031689388 | 67.21896871875447 | 56.13597491448119 | NA | 53.07390217779364 | 42.193308550185876 | 66.32135457598056 | 39.18253798416435 | 51.556580315824085 |
S6 | 40.730697873555016 | 72.0054219875865 | 3.8041538790949967 | 58.98443079560063 | 50.726909920182436 | 64.71762692527096 | NA | 46.43980554761224 | 49.22483389297707 | 40.972965261430915 | 47.51187178703453 |
S7 | 21.22877122877123 | 32.98137975315688 | 6.259367639711655 | 57.2132552492501 | 27.25912200684151 | 44.63776383342841 | 38.007854337736525 | NA | 25.662641994713155 | 55.153719951494395 | 34.26709733278932 |
S8 | 54.90937633794777 | 76.94228436898052 | 22.767825280137036 | 52.91386944722183 | 36.488027366020525 | 65.28807758128922 | 46.53338093538022 | 30.683442951100943 | NA | 32.16349240316713 | 46.52108629680502 |
S9 | 29.42057942057942 | 58.13654847684953 | 8.857326386410676 | 48.907298957291815 | 26.403933865450398 | 41.6001140901312 | 40.2713316672617 | 44.63111238204175 | 24.640994498821176 | NA | 35.87435997164863 |
Mean | 32.685101174338996 | 44.44253304875892 | 39.970699071660995 | 39.34283156616534 | 32.10469523257713 | 34.5374176492318 | 38.94633484364067 | 42.1545470032467 | 44.30512893098316 | 32.633295891858744 |
train \ test | S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
S0 | NA | 36.37012199472069 | 41.28184997501963 | 25.874875017854592 | 46.507981755986314 | 27.54563605248146 | 19.371652981078185 | 13.72605090077209 | 38.14388797599486 | 13.075112347528353 | 29.09968544460402 |
S1 | 46.567717996289424 | NA | 4.653486546285062 | 64.19797171832596 | 58.715792474344354 | 50.90559041642898 | 40.79971438771867 | 29.246496997426362 | 61.984711009502035 | 44.996076752978105 | 44.67417314436655 |
S2 | 48.53004138718424 | 1.6194620817578653 | NA | 16.547636051992573 | 11.402508551881414 | 2.167712492869367 | 8.168511245983577 | 3.2098941950243063 | 5.815531899692791 | 16.042513731364576 | 12.611534626416747 |
S3 | 34.37990580847723 | 67.97460226867375 | 11.077010919991436 | NA | 37.39310148232612 | 60.61751283513976 | 41.977865048197074 | 38.13983414355162 | 54.625991283846545 | 45.402667807974886 | 43.5098323997976 |
S4 | 54.202939917225635 | 68.14582292930014 | 12.968381985582756 | 21.554063705184973 | NA | 42.35596120935539 | 41.11388789717958 | 22.304832713754646 | 39.78709723512181 | 18.196733005207218 | 35.625524510879124 |
S5 | 33.68060510917654 | 62.609688235713776 | 4.8890157733209625 | 61.7340379945722 | 49.18757126567845 | NA | 49.460906818993216 | 41.70717758078353 | 65.16396370650854 | 35.07382837577573 | 44.834088317835885 |
S6 | 33.21678321678322 | 73.11835628165798 | 1.7843123260295481 | 51.73546636194829 | 53.70581527936146 | 63.92612664004563 | NA | 47.926794395195884 | 47.77452311209545 | 40.60917326485484 | 45.97748343088582 |
S7 | 35.014985014985015 | 26.096882357137762 | 13.98900863607166 | 47.75032138265962 | 33.55188141391106 | 26.87535653166001 | 38.421992145662266 | NA | 26.227048653282846 | 52.792638561951634 | 33.41334607748021 |
S8 | 51.855287569573285 | 71.3776128986231 | 15.252301762900577 | 54.69218683045279 | 37.09378563283923 | 60.48203080433543 | 45.233845055337376 | 26.6871604232199 | NA | 28.404308438547687 | 43.45316882398104 |
S9 | 27.486798915370343 | 52.621816365841475 | 18.78524016843908 | 55.24925010712756 | 30.05273660205245 | 26.689960068454077 | 40.96394144948233 | 44.17357735201601 | 28.120311495320426 | NA | 36.01595916934486 |
Mean | 32.148818398966604 | 43.152763238638585 | 40.143479850422025 | 38.19563125054392 | 30.4659095964697 | 33.92991590506485 | 35.76492035606229 | 41.92089214108315 | 41.58564113354625 | 31.906824074794486 |
train \ test | S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
S0 | NA | 33.637725618891345 | 41.6244379416173 | 20.2756749035852 | 50.8551881413911 | 26.846833998859097 | 19.62156372724027 | 13.390048613096942 | 33.528613274273056 | 13.64576645980455 | 28.158428075417653 |
S1 | 51.113172541743964 | NA | 17.678966526300762 | 61.09841451221254 | 62.07953249714937 | 55.355105533371365 | 40.04998214923241 | 30.55476122390621 | 60.648710437950996 | 44.92474498894358 | 47.05593226786791 |
S2 | 44.61253032681604 | 1.3412285082399942 | NA | 11.769747178974432 | 13.968072976054732 | 1.5188248716486026 | 7.197429489468048 | 2.4306548470117244 | 4.801028791883975 | 7.789428632570083 | 10.603216180296405 |
S3 | 26.873126873126875 | 55.57537276164657 | 18.685318678181428 | NA | 26.432440136830103 | 59.697661152310324 | 38.214923241699395 | 38.80468973405776 | 41.88754733157105 | 46.31571438761681 | 39.165199366337816 |
S4 | 57.57100042814328 | 61.26132553328102 | 14.609949325529941 | 20.761319811455508 | NA | 35.35367940673132 | 34.64476972509818 | 22.562196168144126 | 37.872401228834754 | 24.395463299807403 | 34.336900547447286 |
S5 | 31.94662480376766 | 61.06870229007634 | 8.857326386410676 | 60.89130124267962 | 53.46351197263398 | NA | 49.82506247768654 | 41.38547326279669 | 68.97192255483317 | 28.646836436265065 | 45.00630682523885 |
S6 | 28.492935635792776 | 69.10180495113077 | 10.905716936692599 | 56.96329095843451 | 45.48888255416191 | 56.26069594980034 | NA | 45.13869030597655 | 45.98128170322212 | 44.73214922605036 | 44.78504980236244 |
S7 | 11.567004424147282 | 26.867375329956484 | 3.033330954250232 | 45.87201828310241 | 13.654503990877991 | 41.436109526525954 | 38.25776508389861 | NA | 29.677788097449454 | 52.478778800199734 | 29.20496383226757 |
S8 | 56.10817753674896 | 70.69986445031033 | 23.25315823281707 | 51.606913298100274 | 38.05587229190422 | 58.58528237307473 | 44.88397001071046 | 24.06348298541607 | NA | 31.435908410014978 | 44.29918106545523 |
S9 | 34.90794919366348 | 51.1521723621317 | 26.543430162015557 | 54.606484787887446 | 24.451254275940705 | 37.52139189960069 | 39.82149232416994 | 43.49442379182156 | 21.454597413731513 | NA | 37.105910690106946 |
Mean | 31.216667573206166 | 42.86520436301945 | 40.623952595776764 | 36.39354995554014 | 29.870048401437273 | 31.27033293487077 | 36.98032109034862 | 38.2744118558436 | 40.51536733329962 | 31.711232549455715 |
train \ test | S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
S0 | NA | 0.4709995006064065 | 0.3976161587324245 | 0.20761319811455506 | 0.6163055872291904 | 0.3152452937820879 | 0.25548018564798286 | 0.13482985416070917 | 0.3619347002929199 | 0.23239888722448107 | 0.33249148508786186 |
S1 | 0.47245611531325815 | NA | 0.17471986296481337 | 0.6399085844879303 | 0.626354047890536 | 0.6567313177410155 | 0.5366654766154945 | 0.3902630826422648 | 0.6235621918982639 | 0.39104073043726373 | 0.5013001566656489 |
S2 | 0.4757385471671186 | 0.07569380038524649 | NA | 0.20632766747607484 | 0.12143671607753706 | 0.06988020536223617 | 0.08589789360942521 | 0.13533028309979983 | 0.03507894548831893 | 0.15871317497681717 | 0.15156635929361936 |
S3 | 0.33723419437705154 | 0.707926089748163 | 0.179287702519449 | NA | 0.5030644241733181 | 0.6723474044495151 | 0.4936808282756158 | 0.45145839290820705 | 0.5924841037365149 | 0.5023896140951566 | 0.4933191949203323 |
S4 | 0.655273298130441 | 0.5582506955839338 | 0.23353079723074727 | 0.33716611912583916 | NA | 0.5016400456360525 | 0.3985005355230275 | 0.24857020303116958 | 0.49074801743230695 | 0.31614237820101293 | 0.41553578776605893 |
S5 | 0.3512202083630655 | 0.6268103017764144 | 0.11441010634501463 | 0.728253106699043 | 0.4923745724059293 | NA | 0.4720456979650125 | 0.46418358593079784 | 0.7317282274773166 | 0.4326271488693915 | 0.49040588398133167 |
S6 | 0.19823034108748394 | 0.6666191053720483 | 0.04111055599172079 | 0.643622339665762 | 0.34927309007981755 | 0.6177980604677695 | NA | 0.5090077209036317 | 0.5596199185539759 | 0.4247093230615593 | 0.44555449502041866 |
S7 | 0.22077922077922077 | 0.3499322251551687 | 0.08307758189993576 | 0.5451364090844165 | 0.30252280501710377 | 0.3594552196235026 | 0.37750803284541234 | NA | 0.29163392155461887 | 0.5251444468221699 | 0.339465540309061 |
S8 | 0.30676466390752105 | 0.7181993293857459 | 0.13475126686175148 | 0.4164405084987859 | 0.3390820980615735 | 0.5148317170564746 | 0.3965726526240628 | 0.22326279668287102 | NA | 0.3808402881803267 | 0.38119392458434587 |
S9 | 0.31589838732695874 | 0.591710066348006 | 0.08043679965741203 | 0.6103413798028853 | 0.23603192702394526 | 0.4028094694808899 | 0.42541949303820065 | 0.451172433514441 | 0.27034364506680003 | NA | 0.3760181779177265 |
Mean | 0.3900310143844728 | 0.45496174865603756 | 0.41122341816316843 | 0.4170204119568795 | 0.3543469720380221 | 0.3262890176701671 | 0.3919252615068052 | 0.4054740868681917 | 0.43714675821237514 | 0.3384323160902861 |
| S0 | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | S9 | Mean | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0.6659768802625945 | 0.7210530070628522 | 0.2673613589322675 | 0.6891872589630053 | 0.6061858608893956 | 0.7387335995436395 | 0.5402356301320956 | 0.4952101801544181 | 0.5840537257983853 | 0.5451886725158713 | 0.5853186174254525 |