BERTのファインチューニングのハイパーパラメータ調整

yoheikikuta/bert-japaneseをlivedoorコーパスでファインチューニングする

code:txt

[{'epoch': 1,

'Training Loss': 0.4150947373544638,

'Valid. Loss': 0.2322307937854045,

'Valid. Accur.': 0.9540540540540541,

'Training Time': '0:06:18',

'Validation Time': '0:00:13'},

{'epoch': 2,

'Training Loss': 0.12052251508210048,

'Valid. Loss': 0.18612784920512018,

'Valid. Accur.': 0.9608108108108108,

'Training Time': '0:06:20',

'Validation Time': '0:00:13'},

{'epoch': 3,

'Training Loss': 0.04736771415671911,

'Valid. Loss': 0.22951744343783403,

'Valid. Accur.': 0.9608108108108108,

'Training Time': '0:06:22',

'Validation Time': '0:00:13'},

{'epoch': 4,

'Training Loss': 0.016379446042892584,

'Valid. Loss': 0.19560098776946197,

'Valid. Accur.': 0.9702702702702702,

'Training Time': '0:06:22',

'Validation Time': '0:00:13'}]

適当にepoch10にしてみた

code:txt

[{'epoch': 1,

'Training Loss': 0.08382027277181461,

'Valid. Loss': 0.2743670979061642,

'Valid. Accur.': 0.9635135135135136,

'Training Time': '0:06:16',

'Validation Time': '0:00:13'},

{'epoch': 2,

'Training Loss': 0.040853016051509844,

'Valid. Loss': 0.2656957007743217,

'Valid. Accur.': 0.9635135135135136,

'Training Time': '0:06:21',

'Validation Time': '0:00:13'},

{'epoch': 3,

'Training Loss': 0.04085431272215665,

'Valid. Loss': 0.3141520078117783,

'Valid. Accur.': 0.9621621621621622,

'Training Time': '0:06:23',

'Validation Time': '0:00:13'},

{'epoch': 4,

'Training Loss': 0.020573266848148856,

'Valid. Loss': 0.3591364287041329,

'Valid. Accur.': 0.9581081081081081,

'Training Time': '0:06:26',

'Validation Time': '0:00:13'},

{'epoch': 5,

'Training Loss': 0.012506778622708936,

'Valid. Loss': 0.37596769848385375,

'Valid. Accur.': 0.9635135135135136,

'Training Time': '0:06:27',

'Validation Time': '0:00:13'},

{'epoch': 6,

'Training Loss': 0.01071635927596052,

'Valid. Loss': 0.26787628386471724,

'Valid. Accur.': 0.9675675675675676,

'Training Time': '0:06:28',

'Validation Time': '0:00:13'},

{'epoch': 7,

'Training Loss': 0.003255896027614469,

'Valid. Loss': 0.3228567664687698,

'Valid. Accur.': 0.9594594594594594,

'Training Time': '0:06:29',

'Validation Time': '0:00:13'},

{'epoch': 8,

'Training Loss': 0.0016660764364051588,

'Valid. Loss': 0.3710041522979736,

'Valid. Accur.': 0.9648648648648649,

'Training Time': '0:06:29',

'Validation Time': '0:00:14'},

{'epoch': 9,

'Training Loss': 9.848930579761832e-06,

'Valid. Loss': 0.35061086835087957,

'Valid. Accur.': 0.9662162162162162,

'Training Time': '0:06:29',

'Validation Time': '0:00:13'},

{'epoch': 10,

'Training Loss': 7.127967760847618e-06,

'Valid. Loss': 0.3456918832418081,

'Valid. Accur.': 0.9662162162162162,

'Training Time': '0:06:30',

'Validation Time': '0:00:13'}]

もしかしたら上のepoch4のやつから追い学習してしまったかも？やりなおす

学習に必要なリソース及び結果の適切なmemoizeしたい

任意のpythonオブジェクトはpickleでシリアライズは可能っぽい

やりなおした

code:txt

[{'epoch': 1,

'Training Loss': 0.4247249074738621,

'Valid. Loss': 0.18359624824008425,

'Valid. Accur.': 0.9621621621621622,

'Training Time': '0:06:18',

'Validation Time': '0:00:13'},

{'epoch': 2,

'Training Loss': 0.134765110559429,

'Valid. Loss': 0.16350458505991342,

'Valid. Accur.': 0.9743243243243244,

'Training Time': '0:06:21',

'Validation Time': '0:00:13'},

{'epoch': 3,

'Training Loss': 0.06271529341491773,

'Valid. Loss': 0.19041847344991322,

'Valid. Accur.': 0.9675675675675676,

'Training Time': '0:06:24',

'Validation Time': '0:00:13'},

{'epoch': 4,

'Training Loss': 0.04136088375447024,

'Valid. Loss': 0.18287213525256596,

'Valid. Accur.': 0.972972972972973,

'Training Time': '0:06:27',

'Validation Time': '0:00:13'},

{'epoch': 5,

'Training Loss': 0.017369677347498436,

'Valid. Loss': 0.26986650389594,

'Valid. Accur.': 0.9648648648648649,

'Training Time': '0:06:28',

'Validation Time': '0:00:13'},

{'epoch': 6,

'Training Loss': 0.014320675584749375,

'Valid. Loss': 0.14448782495550208,

'Valid. Accur.': 0.981081081081081,

'Training Time': '0:06:28',

'Validation Time': '0:00:13'},

{'epoch': 7,

'Training Loss': 0.0052554685402261625,

'Valid. Loss': 0.1969128737578521,

'Valid. Accur.': 0.972972972972973,

'Training Time': '0:06:29',

'Validation Time': '0:00:14'},

{'epoch': 8,

'Training Loss': 0.003742730157366708,

'Valid. Loss': 0.19850413638192255,

'Valid. Accur.': 0.9756756756756757,

'Training Time': '0:06:29',

'Validation Time': '0:00:13'},

{'epoch': 9,

'Training Loss': 0.0009715964049280625,

'Valid. Loss': 0.19306631088256837,

'Valid. Accur.': 0.9756756756756757,

'Training Time': '0:06:28',

'Validation Time': '0:00:13'},

{'epoch': 10,

'Training Loss': 0.0010697050750327485,

'Valid. Loss': 0.19987142086029053,

'Valid. Accur.': 0.9756756756756757,

'Training Time': '0:06:29',

'Validation Time': '0:00:13'}]