Loss and Accuracy Get Stuck in Keras

 

Created on: 9/30/19
Last update: 9/30/19

Problem Description

When training a prediction model using Keras, the loss and accuracy sometimes get stuck. For example:

Epoch 1/10
2499007/2499007 [==============================] - 2s - loss: 1.9112 - acc: 0.5325
Epoch 2/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 3/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 4/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 5/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 6/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 7/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 8/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 9/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378
Epoch 10/10
2499007/2499007 [==============================] - 0s - loss: 1.9105 - acc: 0.5378

This issue could be caused by several reasons. Here we summarize some solutions to fix it.

Solutions

Changing Optimizer

A list of optimizers in Keras: https://keras.io/optimizers/

Adjusting Learning Rate

Tune the network by adjusting learning rate of the optimizer. For example, increase learning from 0.00001 to 0.1:

sgd = optimizers.SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy',
                optimizer=sgd,
                metrics=['accuracy'])

Normalizing Activations

Add BatchNormalization after each layer:

...
model.add(Dense(20, activation='relu'))
model.add(BatchNormalization())
...

Reference

Loss not changing when training

Keras accuracy does not change

Why my training and validation loss is not changing?

What is the “dying ReLU” problem in neural networks?