Train loss doesn't decrease

[kmttsh]

Thank you for publishing source code.
With reference to "Readme", I first tried out "train_svhn.py", but the train loss does not go down well.
Could you check whether something is wrong?

1. download "svhn_datasets_and_models.zip"
2. edit generated/easy/train.csv
   insert "4 4" to line1
3. prepare the curriculum specification

[
    {
        "train": "<any directory>/svhn_dataset_and_models/generated/easy/train.csv",
        "validation": "<any directory>/svhn_dataset_and_models/generated/easy/valid.csv"
    }
]

4. execute train script

$ python train_svhn.py specification.json <log_dir> --blank-label 10 --char-map ../datasets/svhn/svhn_char_map.json -b 64 -g 0
<dir>/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py:131: UserWarning: optimizer.eps is changed to 1e-08 by MultiprocessParallelUpdater for new batch size.
  format(optimizer.eps))
epoch       iteration   main/loss   main/accuracy  lr          fast_validation/main/loss  fast_validation/main/accuracy  validation/main/loss  validation/main/accuracy
0           100         7.50537     0.00480469     0.00308566                                       
0           200         6.50787     0.0170703      0.00425853                                       
0           300         6.56413     0.0196875      0.00509208                                       
0           400         7.11518     0.0114844      0.00574294                                       
0           500         7.72572     0              0.00627392                                       
0           600         7.69402     0              0.00671828                                       
0           700         7.39172     0.00714844     0.0070964                                        
0           800         6.21048     0.0192187      0.00742193                                       
0           900         6.74564     0.0199219      0.00770463                                       
0           1000        6.19807     0.0205859      0.00795176  6.12725                    0.0204327 
0           1100        6.18472     0.0188672      0.00816892                                       
0           1200        6.18693     0.0186328      0.00836054                                       
0           1300        6.69872     0.0203516      0.00853021                                       
0           1400        6.78628     0.0203125      0.00868087                                       
0           1500        6.78714     0.0190234      0.00881497                                       
1           1600        6.71787     0.02           0.00893457                                       
1           1700        6.40516     0.019375       0.00904141                                       
1           1800        6.9848      0.0145703      0.00913701                                       
1           1900        6.7551      0.0185156      0.00922265                                       
1           2000        6.21905     0.0180078      0.00929946  6.17801                    0.0204327 
1           2100        6.07871     0.0210938      0.00936842                                       
1           2200        6.2583      0.0188281      0.00943037                                       
1           2300        6.10771     0.0191406      0.00948608                                       
1           2400        6.42393     0.0198438      0.0095362                                        
1           2500        6.79451     0.0196484      0.00958132                                       
1           2600        6.73464     0.0185547      0.00962197                                       
1           2700        6.38567     0.0184375      0.0096586                                        
1           2800        6.06156     0.0200391      0.00969162                                       
1           2900        6.13261     0.0175781      0.0097214                                        
1           3000        6.06467     0.0183203      0.00974827  6.04983                    0.0204327 
1           3100        6.09032     0.0203906      0.00977252                                       
2           3200        6.0301      0.0217969      0.0097944                                        
2           3300        6.14339     0.0198828      0.00981416                                       
2           3400        6.28762     0.0178125      0.00983201                                       
2           3500        6.20747     0.0170313      0.00984812                                       
2           3600        6.68129     0.0208984      0.00986268                                       
2           3700        6.14636     0.0198828      0.00987584                                       
2           3800        6.11925     0.0185156      0.00988773                                       
2           3900        6.01957     0.0199609      0.00989847                                       
2           4000        6.00062     0.0197656      0.00990818  6.0088                     0.0204327 
2           4100        6.01043     0.0193359      0.00991696                                       
2           4200        6.02558     0.0192187      0.0099249                                        
2           4300        6.60501     0.0197266      0.00993207                                       
2           4400        6.14031     0.0174609      0.00993856                                       
2           4500        6.09017     0.0189844      0.00994443                                       
2           4600        6.07965     0.0204688      0.00994973                                       
3           4700        7.72351     0.00117187     0.00995453                                       
3           4800        7.69427     0              0.00995887                                       
3           4900        7.6744      0              0.00996279                                       
3           5000        7.648       0.00160156     0.00996634  10.9449                    0         
3           5100        7.67579     0              0.00996955                                       
3           5200        7.33484     0.00734375     0.00997245                                       
3           5300        7.32619     0.00722656     0.00997508                                       
3           5400        7.13675     0.0139844      0.00997745                                       
3           5500        7.6732      0              0.0099796                                        
3           5600        7.67175     0              0.00998155                                       
3           5700        7.68901     3.90625e-05    0.0099833                                        
3           5800        7.66603     0              0.00998489                                       
3           5900        7.52425     0.00371094     0.00998633                                       
3           6000        7.47132     0.00558594     0.00998764  10.7991                    0         
3           6100        7.66778     0              0.00998881                                       
3           6200        7.6659      7.8125e-05     0.00998988                                       
4           6300        7.65129     0              0.00999084                                       
4           6400        7.67035     0              0.00999172                                       
4           6500        7.67038     0              0.0099925                                        
4           6600        7.65043     0              0.00999322                                       
4           6700        7.65205     0              0.00999386                                       
4           6800        7.66811     3.90625e-05    0.00999445                                       
4           6900        7.6578      0              0.00999498                                       
4           7000        7.65613     0.000117187    0.00999546  7.66984                    0         
4           7100        7.56788     0.00222656     0.00999589                                       
4           7200        7.66979     0              0.00999628                                       
4           7300        7.5385      0.00460937     0.00999663                                       
4           7400        7.65839     0.000664063    0.00999695                                       
4           7500        7.66035     0              0.00999724                                       
4           7600        7.67189     0              0.00999751                                       
4           7700        7.66347     0.00015625     0.00999774                                       
4           7800        7.64676     0              0.00999796                                       
5           7900        7.66213     0.000234375    0.00999815                                       
5           8000        7.64856     0.00015625     0.00999833  7.66626                    0         
5           8100        7.66833     0              0.00999849                                       
5           8200        7.65137     0              0.00999863                                       
5           8300        7.41944     0.0117578      0.00999876                                       
5           8400        7.25518     0.0126172      0.00999888                                       
5           8500        7.39664     0.015625       0.00999899                                       
5           8600        7.61295     0.00492188     0.00999908                                       
5           8700        7.87834     0.00515625     0.00999917                                       
5           8800        7.74946     0.0028125      0.00999925                                       
5           8900        7.71717     0              0.00999932                                       
5           9000        7.67788     0              0.00999939  9.00867                    0         
5           9100        7.68219     7.8125e-05     0.00999944                                       
5           9200        7.67184     0              0.0099995                                        
5           9300        7.66276     0              0.00999954                                       
6           9400        7.70615     0.0021875      0.00999959                                       
6           9500        7.65457     0.00164062     0.00999963                                       
6           9600        7.68174     0              0.00999966                                       
6           9700        7.6786      0              0.0099997                                        
6           9800        7.66229     0              0.00999972                                       
6           9900        7.69749     0.000195313    0.00999975                                       
6           10000       7.67097     0              0.00999977  7.68619                    0         
6           10100       7.69214     0              0.0099998                                        
6           10200       7.63691     0.00105469     0.00999982                                       
6           10300       7.72377     3.90625e-05    0.00999983                                       
6           10400       7.64548     0              0.00999985                                       
6           10500       7.68391     0              0.00999986                                       
6           10600       7.66396     0              0.00999988                                       
6           10700       7.64902     0              0.00999989                                       
6           10800       7.67067     0              0.0099999                                        
6           10900       7.64861     0              0.00999991                                       
7           11000       7.66595     0              0.00999992  7.67054                    0         
7           11100       7.66005     0              0.00999992                                       
7           11200       7.6663      0              0.00999993                                       
7           11300       7.67519     0              0.00999994                                       
7           11400       7.66069     0              0.00999994                                       
7           11500       7.67084     0              0.00999995                                       
7           11600       7.67509     0              0.00999995                                       
7           11700       7.66532     0              0.00999996                                       
7           11800       7.48501     0.00386719     0.00999996                                       
7           11900       7.65868     0              0.00999997                                       
7           12000       7.66759     0              0.00999997  7.66575                    0         
7           12100       7.64796     0              0.00999997                                       
7           12200       7.642       0              0.00999998                                       
7           12300       7.64562     0              0.00999998                                       
7           12400       7.65264     0              0.00999998                                       
8           12500       7.67088     0              0.00999998                                       
8           12600       7.65142     0              0.00999998                                       
8           12700       7.65538     0              0.00999998                                       
8           12800       7.65558     0              0.00999999                                       
8           12900       7.66965     0              0.00999999                                       
8           13000       7.65551     0              0.00999999  7.77481                    0         
8           13100       7.57688     0.000195313    0.00999999                                       
8           13200       7.6336      0              0.00999999                                       
8           13300       6.82549     0.0161719      0.00999999                                       
8           13400       6.67704     0.0145703      0.00999999                                       
8           13500       7.42664     0.0046875      0.00999999                                       
8           13600       7.64274     0              0.00999999                                       
8           13700       7.43383     0.00761719     0.00999999                                       
8           13800       7.60467     0.000390625    0.00999999                                       
8           13900       6.867       0.0198438      0.01                                             
8           14000       7.47555     0.00910156     0.01        7.68502                    0         
9           14100       7.65677     0              0.01                                             
9           14200       7.5657      0.000273438    0.01                                             
9           14300       7.61998     0              0.01                                             
9           14400       7.61674     0.00382812     0.01                                             
9           14500       7.68163     0              0.01                                             
9           14600       7.65586     0              0.01                                             
9           14700       7.41383     0.00523438     0.01                                             
9           14800       7.31547     0.0123438      0.01                                             
9           14900       7.66587     0              0.01                                             
9           15000       7.65826     0.000625       0.01        8.35651                    0         
9           15100       7.65988     0              0.01                                             
9           15200       7.65109     0              0.01                                             
9           15300       7.63758     0              0.01                                             
9           15400       7.64964     0              0.01                                             
9           15500       7.63695     0              0.01                                             
9           15600       7.64448     0              0.01                     

Even if I use images generated by ”create_svhn_dataset_4_images.py”, the same result will be obtained.

[Bartzi]

The learning rate is too high!
You can check this by having a look at all images in the folder <log_dir>/bboxes. You should see that the predicted bboxes jump around a lot!

I suggest setting the learning rate to 1e-4 or 1e-5 with the command line switch -lr or --learning-rate

[kmttsh]

Thank you for your reply!
As you suggested, the accuracy has improved, but it is not enough.

python train_svhn.py <dir>/svhn_dataset_and_models/specification.json ./work --gpus 3 --blank-label 10 --char-map ../datasets/svhn/svhn_char_map.json -b  64 --learning-rate 0.0001 --epoch 200
<dir>/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py:131: UserWarning: optimizer.eps is changed to 1e-08 by MultiprocessParallelUpdater for new batch size.
  format(optimizer.eps))
epoch       iteration   main/loss   main/accuracy  lr          fast_validation/main/loss  fast_validation/main/accuracy  validation/main/loss  validation/main/accuracy
0           100         7.30914     0.00507813     3.08566e-05                                                                                                            
0           200         6.04959     0.0198828      4.25853e-05                                                                                                            
0           300         5.97783     0.0191797      5.09208e-05                                                                                                            
0           400         5.97065     0.0211719      5.74294e-05                                                                                                            
0           500         5.94329     0.0212109      6.27392e-05                                                                                                            
0           600         5.94245     0.0192969      6.71828e-05                                                                                                            
0           700         5.95647     0.020625       7.0964e-05                                                                                                            
0           800         5.89843     0.0215625      7.42193e-05                                                                                                            
0           900         5.91704     0.0216406      7.70463e-05                                                                                                            
0           1000        5.92901     0.0198828      7.95176e-05  6.10376                    0.0174028                                                                      
0           1100        5.92836     0.0207422      8.16892e-05                                                                                                            
0           1200        5.92269     0.0225781      8.36054e-05                                                                                                            
0           1300        5.89819     0.0221875      8.53021e-05                                                                                                            
0           1400        5.89788     0.0219922      8.68087e-05                                                                                                            
0           1500        5.88448     0.0226562      8.81497e-05                                                                                                            
1           1600        5.89985     0.0208203      8.93457e-05                                                                                                            
1           1700        5.87362     0.0215625      9.04141e-05                                                                                                            
1           1800        5.87873     0.0211719      9.13701e-05                                                                                                            
1           1900        5.88609     0.0219922      9.22265e-05                                                                                                            
1           2000        5.86692     0.0218359      9.29946e-05  5.91479                    0.021234                                                                       
1           2100        5.87499     0.0230469      9.36842e-05                                                                                                            
1           2200        5.90617     0.0214844      9.43037e-05                                                                                                            
1           2300        5.88547     0.0224219      9.48608e-05                                                                                                            
1           2400        5.88797     0.02125        9.5362e-05                                                                                                            
1           2500        5.87188     0.0216797      9.58132e-05                                                                                                            
1           2600        5.86528     0.0226953      9.62197e-05                                                                                                            
1           2700        5.85535     0.0210156      9.6586e-05                                                                                                            
1           2800        5.85524     0.0235156      9.69162e-05                                                                                                            
1           2900        5.84669     0.0228906      9.7214e-05                                                                                                            
1           3000        5.87788     0.0221484      9.74827e-05  5.86614                    0.0217849                                                                      
1           3100        5.86994     0.0215625      9.77252e-05                                                                                                            
2           3200        5.84037     0.0232422      9.7944e-05                                                                                                            
2           3300        5.83626     0.0228516      9.81416e-05                                                                                                            
2           3400        5.79569     0.0234766      9.83201e-05                                                                                                            
2           3500        5.78465     0.0230859      9.84812e-05                                                                                                            
2           3600        5.74576     0.0256641      9.86268e-05                                                                                                            
2           3700        5.76113     0.0221094      9.87584e-05                                                                                                            
2           3800        5.73173     0.0233203      9.88773e-05                                                                                                            
2           3900        5.731       0.02375        9.89847e-05                                                                                                            
2           4000        5.71808     0.0241406      9.90818e-05  5.77324                    0.0233624                                                                      
2           4100        5.72374     0.0244141      9.91696e-05                                                                                                            
2           4200        5.7117      0.0257031      9.9249e-05                                                                                                            
2           4300        5.71056     0.0239453      9.93207e-05                                                                                                            
2           4400        5.71849     0.0242188      9.93856e-05                                                                                                            
2           4500        5.70581     0.02625        9.94443e-05                                                                                                            
2           4600        5.68513     0.0243359      9.94973e-05                                                                                                            
3           4700        5.70342     0.0253125      9.95453e-05                                                                                                            
3           4800        5.67535     0.0263672      9.95887e-05                                                                                                            
3           4900        5.67486     0.0269922      9.96279e-05                                                                                                            
3           5000        5.64855     0.0248828      9.96634e-05  5.97818                    0.0250651                                                                      
3           5100        5.67777     0.0243359      9.96955e-05                                                                                                            
3           5200        5.66923     0.0266016      9.97245e-05                                                                                                            
3           5300        5.69099     0.0239063      9.97508e-05             


~~~~~~~~~~~~~~~~~~~~~

198         310400      3.60274     0.178672       0.0001                                                                                                                
198         310500      3.57621     0.180156       0.0001                                                                                                                
198         310600      3.53797     0.181992       0.0001                                                                                                                
198         310700      3.63558     0.180273       0.0001                                                                                                                
198         310800      3.59109     0.179375       0.0001                                                                                                                
198         310900      3.61663     0.176719       0.0001                                                                                                                
199         311000      3.5567      0.180586       0.0001      6.24907                    0.0733674                                                                      
199         311100      3.60563     0.178945       0.0001                                                                                                                
199         311200      3.66259     0.179727       0.0001                                                                                                                
199         311300      3.59662     0.185078       0.0001                                                                                                                
199         311400      3.53541     0.189766       0.0001                                                                                                                
199         311500      3.58667     0.182734       0.0001                                                                                                                
199         311600      3.52114     0.195586       0.0001                                                                                                                
199         311700      3.58835     0.180156       0.0001                                                                                                                
199         311800      3.5372      0.185391       0.0001                                                                                                                
199         311900      3.57168     0.183438       0.0001                                                                                                                
199         312000      3.47688     0.195195       0.0001      5.776                      0.0995843                                                                      
199         312100      3.48593     0.189102       0.0001                                                                                                                
199         312200      3.55354     0.18082        0.0001                                                                                                                
199         312300      3.48362     0.189219       0.0001                                                                                                                
199         312400      3.52996     0.185586       0.0001                                                                                                                
200         312500      3.53738     0.182422       0.0001        

So, I have two questions.

1. what accuracy is obtained by this learning?

2. how many epochs does this network need to learn?

[Bartzi]

Do you mean what kind of accuracy is obtained?
If so it is word level accuracy.

The network does not need many epochs to learn normally. 10 should be enough to get a first decent version, but then you should restart the training with randomly initialized recognition net, but using the trained weights for the localziation net.

Did you have a look at the pictures in <log_dir>/bboxes? How do the predicted bboxes look like? These images are there to give you a feeling of the performance of the network and if you set all hyperparameters correctly.

[kmttsh]

Sorry, what I wanted to know is how much "main / accuracy" is improved when this script is executed.
And I tried training with another parameter again, the accuracy has improved.

I think that I could not set an appropriate learning rate for batch size.

 $ python train_svhn.py <dir>/svhn_dataset_and_models/specification.json ./work --gpus 0 1 2 3 --blank-label 10 --char-map ../datasets/svhn/svhn_char_map.json -b 16 --learning-rate 0.0008 --epoch 100
<dir>/lib/python3.5/site-packages/chainer/training/updaters/multiprocess_parallel_updater.py:131: UserWarning: optimizer.eps is changed to 4e-08 by MultiprocessParallelUpdater for new batch size.
  format(optimizer.eps))
epoch       iteration   main/loss   main/accuracy  lr          fast_validation/main/loss  fast_validation/main/accuracy  validation/main/loss  validation/main/accuracy
0           100         7.08026     0.00921875     0.000246853                                                                                                            
0           200         6.48398     0.0171875      0.000340683                                                                                                            
0           300         6.3467      0.0167188      0.000407367                                                                                                            
0           400         6.07347     0.0220312      0.000459436                                                                                                            
0           500         6.14111     0.0207813      0.000501914                                                                                                            
0           600         6.07865     0.019375       0.000537463                                                                                                            
0           700         6.07493     0.0190625      0.000567712                                                                                                            
0           800         6.04775     0.0148437      0.000593755                                                                                                            
0           900         5.98705     0.0223437      0.00061637                                                                                                            
0           1000        5.96897     0.0203125      0.000636141  5.9492                     0.0199219                                                                      
0           1100        6.01165     0.0189062      0.000653513                                                                                                            
0           1200        5.96329     0.02375        0.000668843                                                                 

98          153400      0.903547    0.760156       0.0008                                                                                                                
98          153500      0.897576    0.7625         0.0008                                                                                                                
98          153600      0.916847    0.756719       0.0008                                                                                                                
98          153700      0.984916    0.741094       0.0008                                                                                                                
98          153800      0.970778    0.742188       0.0008                                                                                                                
98          153900      0.985056    0.745938       0.0008                                                                                                                
98          154000      0.990586    0.739531       0.0008      1.20713                    0.706172                                                                       
98          154100      0.939159    0.7525         0.0008                                                                                                                
98          154200      0.932501    0.754062       0.0008                                                                                                                
98          154300      1.02911     0.727812       0.0008                                                                                                                
98          154400      0.991427    0.747344       0.0008                                                                                                                
98          154500      1.03597     0.730156       0.0008                                                                                                                
98          154600      0.975343    0.747031       0.0008                                                                                                                
99          154700      0.984289    0.744844       0.0008                                                                                                                
99          154800      0.911613    0.760312       0.0008                                                                                                                
99          154900      0.865186    0.767969       0.0008                                                                                                                
99          155000      0.953758    0.743125       0.0008      1.42776                    0.676875                                                                       
99          155100      0.94563     0.751875       0.0008                                                                                                                
99          155200      0.951638    0.746406       0.0008                                                                                                                
99          155300      0.908591    0.756875       0.0008                                                                                                                
99          155400      0.926625    0.750469       0.0008                                                                                                                
99          155500      0.969775    0.748125       0.0008                                                                                                                
99          155600      0.981316    0.75           0.0008                                                                                                                
99          155700      0.93322     0.755625       0.0008                                                                                                                
99          155800      0.999408    0.736719       0.0008                                                                                                                
99          155900      1.00439     0.739844       0.0008                                                                                                                
99          156000      0.944262    0.749375       0.0008      1.29065                    0.695078                                                                       
99          156100      0.968931    0.749687       0.0008                                                                                                                
99          156200      0.955661    0.747344       0.0008                                                   

However, it seems that localization is not working well.
If so, could you tell me a good way to improve?

[Bartzi]

Hmm, two things:

1. it is not necessarily good the set the learning rate to a higher value than 1e-4, you are using 8e-4 right now.
2. In order to increase localization performance, you can try the following: Locate the trained model (.npz file) and restart the training, using this model (-r or --resume + path to model). You should also make the network load only the trained weights of the localization network and not the weights of the recognition network (use --load-localization for that). This should help.