CIFAR-10 CNN Progression: Keras 3#
CIFAR-10 holds 60,000 colour images of 32×32 pixels in ten classes. Objects vary in pose, scale, and background, which makes the task much harder than MNIST. This notebook trains two networks on the same split: a small network for 5 epochs and a deeper one with more regularization for 25 epochs. The better of the two on validation accuracy is then evaluated once on the test set.
Keras and PyTorch#
This notebook repeats the PyTorch version with the Keras 3 API, running on
the same PyTorch backend. compile() sets the optimizer and loss, fit()
runs the loop over batches and epochs, and callbacks such as early stopping
replace hand-written control flow. The data split, model, and evaluation
match the PyTorch notebook.
Setup#
The libraries used below.
Keras 3.15.1, backend torch, PyTorch 2.13.0
Load CIFAR-10 and split it#
The 50,000 official training images are split, with a fixed seed and stratified by class, into 45,000 for training and 5,000 for validation. The 10,000 official test images are used once, at the end.
SEED = 42
EPOCHS = 5 # Reduce to 1 or 2 for a quicker run.
BATCH_SIZE = 64
CLASS_NAMES = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']
keras.utils.set_random_seed(SEED)
torch.use_deterministic_algorithms(True, warn_only=True)
(x_development, y_development), (x_test, y_test) = keras.datasets.cifar10.load_data()
y_development = y_development.reshape(-1)
y_test = y_test.reshape(-1)
all_indices = np.arange(len(y_development))
train_indices, validation_indices = train_test_split(
all_indices,
test_size=5000,
random_state=SEED,
stratify=y_development,
)
split_signature = hashlib.sha256(
validation_indices.astype("<i8").tobytes()
).hexdigest()[:16]
# Scale pixel values from 0-255 to 0-1.
x_train = x_development[train_indices].astype("float32") / 255.0
y_train = y_development[train_indices].astype(np.int64)
x_validation = x_development[validation_indices].astype("float32") / 255.0
y_validation = y_development[validation_indices].astype(np.int64)
x_test = x_test.astype("float32") / 255.0
y_test = y_test.astype(np.int64)
assert set(train_indices).isdisjoint(validation_indices)
print(
f"train={len(y_train):,}, validation={len(y_validation):,}, "
f"test={len(y_test):,}, image shape={x_train.shape[1:]}"
)
train=45,000, validation=5,000, test=10,000, image shape=(32, 32, 3)
Look at the data#
One example per class and the number of training images in each. The classes are balanced, so accuracy is a fair summary of performance.
def as_image(sample):
# Convert one stored sample to (height, width[, channel]) for plotting.
if sample.ndim == 1:
sample = sample.reshape(28, 28)
if sample.ndim == 3 and sample.shape[0] in (1, 3):
sample = np.transpose(sample, (1, 2, 0))
return sample.squeeze()
fig = plt.figure(figsize=(12, 3.6))
grid = fig.add_gridspec(2, 10, height_ratios=[1, 1.2])
for label in range(10):
axis = fig.add_subplot(grid[0, label])
image = as_image(x_train[np.flatnonzero(y_train == label)[0]])
axis.imshow(image, cmap="gray" if image.ndim == 2 else None)
axis.set_title(CLASS_NAMES[label], fontsize=9)
axis.axis("off")
counts_axis = fig.add_subplot(grid[1, :])
counts_axis.bar(CLASS_NAMES, np.bincount(y_train, minlength=10), color="0.45")
counts_axis.set(ylabel="Training images")
plt.tight_layout()
plt.show()
Training loop#
fit() below compiles the model with the Adam optimizer and the
cross-entropy loss, trains it, and uses a callback to restore the weights
from the epoch with the best validation accuracy.
def fit(model, epochs, learning_rate=1e-3, train=None, validation=None, verbose=True):
# Train with Adam and restore the epoch with the best validation accuracy.
model.compile(
optimizer=keras.optimizers.Adam(learning_rate),
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=["accuracy"],
)
keep_best = keras.callbacks.EarlyStopping(
monitor="val_accuracy", mode="max", patience=epochs, restore_best_weights=True
)
history = model.fit(
*(train or (x_train, y_train)),
validation_data=validation or (x_validation, y_validation),
epochs=epochs,
batch_size=BATCH_SIZE,
callbacks=[keep_best],
verbose=2 if verbose else 0,
).history
if verbose:
best_epoch = int(np.argmax(history["val_accuracy"]))
print(
f"restored epoch {best_epoch + 1} "
f"(validation accuracy {history['val_accuracy'][best_epoch]:.4f})"
)
return history
def predict_classes(model, images):
return model.predict(images, batch_size=BATCH_SIZE, verbose=0).argmax(axis=1)
def plot_history(history, title=None):
epochs = np.arange(1, len(history["loss"]) + 1)
best = epochs[int(np.argmax(history["val_accuracy"]))]
fig, axes = plt.subplots(1, 2, figsize=(10, 3.4))
for axis, key, label in zip(axes, ("loss", "accuracy"), ("Cross-entropy loss", "Accuracy")):
axis.plot(epochs, history[key], marker="o", label="training")
axis.plot(epochs, history[f"val_{key}"], marker="o", label="validation")
axis.axvline(best, color="0.6", linestyle=":", label="restored epoch")
axis.set(title=label, xlabel="Epoch")
axis.grid(alpha=0.25)
axes[0].legend()
if title:
fig.suptitle(title)
plt.tight_layout()
plt.show()
Small network, 5 epochs#
Two convolution layers, the second with stride 2 to halve the resolution, each followed by batch normalization and a leaky ReLU. Batch normalization rescales each layer’s outputs with batch statistics, which stabilizes and speeds up training. One dense layer of 100 units and dropout precede the output.
model = keras.Sequential(
[
keras.Input(shape=(32, 32, 3)),
layers.Conv2D(16, 3),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Conv2D(32, 3, strides=2),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Flatten(),
layers.Dense(100),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Dropout(0.5),
layers.Dense(10),
],
name="cifar10_small",
)
model.summary()
Model: "cifar10_small"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d (Conv2D) │ (None, 30, 30, 16) │ 448 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization │ (None, 30, 30, 16) │ 64 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu (LeakyReLU) │ (None, 30, 30, 16) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_1 (Conv2D) │ (None, 14, 14, 32) │ 4,640 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_1 │ (None, 14, 14, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_1 (LeakyReLU) │ (None, 14, 14, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten (Flatten) │ (None, 6272) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense (Dense) │ (None, 100) │ 627,300 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_2 │ (None, 100) │ 400 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_2 (LeakyReLU) │ (None, 100) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout (Dropout) │ (None, 100) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_1 (Dense) │ (None, 10) │ 1,010 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 633,990 (2.42 MB)
Trainable params: 633,694 (2.42 MB)
Non-trainable params: 296 (1.16 KB)
history = fit(model, EPOCHS)
plot_history(history, 'Small network')
small_model, small_history = model, history
small_validation = max(history['val_accuracy'])
Epoch 1/5
704/704 - 41s - 58ms/step - accuracy: 0.4449 - loss: 1.5637 - val_accuracy: 0.4496 - val_loss: 1.6329
Epoch 2/5
704/704 - 31s - 44ms/step - accuracy: 0.5617 - loss: 1.2368 - val_accuracy: 0.5648 - val_loss: 1.1915
Epoch 3/5
704/704 - 15s - 21ms/step - accuracy: 0.6078 - loss: 1.1013 - val_accuracy: 0.5500 - val_loss: 1.3677
Epoch 4/5
704/704 - 15s - 21ms/step - accuracy: 0.6495 - loss: 0.9937 - val_accuracy: 0.6408 - val_loss: 1.0136
Epoch 5/5
704/704 - 14s - 20ms/step - accuracy: 0.6760 - loss: 0.9131 - val_accuracy: 0.6568 - val_loss: 0.9964
restored epoch 5 (validation accuracy 0.6568)
Deeper network, 25 epochs#
Three convolution stages and dense layers of 600 and 150 units, with dropout after the convolutions and between the dense layers. The extra capacity lets the network learn richer features, but also makes it easier to memorize the training images; the gap between training and validation accuracy shows when that begins.
EPOCHS = 25 # Reduce to 1 or 2 for a quicker run.
model = keras.Sequential(
[
keras.Input(shape=(32, 32, 3)),
layers.Conv2D(32, 3),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Conv2D(64, 3, strides=2),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Conv2D(128, 3, strides=2),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Dropout(0.2),
layers.Flatten(),
layers.Dense(600),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Dropout(0.25),
layers.Dense(150),
layers.BatchNormalization(),
layers.LeakyReLU(negative_slope=0.1),
layers.Dropout(0.5),
layers.Dense(10),
],
name="cifar10_deeper",
)
model.summary()
Model: "cifar10_deeper"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Layer (type) ┃ Output Shape ┃ Param # ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ conv2d_2 (Conv2D) │ (None, 30, 30, 32) │ 896 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_3 │ (None, 30, 30, 32) │ 128 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_3 (LeakyReLU) │ (None, 30, 30, 32) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_3 (Conv2D) │ (None, 14, 14, 64) │ 18,496 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_4 │ (None, 14, 14, 64) │ 256 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_4 (LeakyReLU) │ (None, 14, 14, 64) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ conv2d_4 (Conv2D) │ (None, 6, 6, 128) │ 73,856 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_5 │ (None, 6, 6, 128) │ 512 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_5 (LeakyReLU) │ (None, 6, 6, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_1 (Dropout) │ (None, 6, 6, 128) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ flatten_1 (Flatten) │ (None, 4608) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_2 (Dense) │ (None, 600) │ 2,765,400 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_6 │ (None, 600) │ 2,400 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_6 (LeakyReLU) │ (None, 600) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_2 (Dropout) │ (None, 600) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_3 (Dense) │ (None, 150) │ 90,150 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ batch_normalization_7 │ (None, 150) │ 600 │ │ (BatchNormalization) │ │ │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ leaky_re_lu_7 (LeakyReLU) │ (None, 150) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dropout_3 (Dropout) │ (None, 150) │ 0 │ ├─────────────────────────────────┼────────────────────────┼───────────────┤ │ dense_4 (Dense) │ (None, 10) │ 1,510 │ └─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 2,954,204 (11.27 MB)
Trainable params: 2,952,256 (11.26 MB)
Non-trainable params: 1,948 (7.61 KB)
history = fit(model, EPOCHS)
plot_history(history, 'Deeper network')
deeper_model, deeper_history = model, history
deeper_validation = max(history['val_accuracy'])
Epoch 1/25
704/704 - 21s - 30ms/step - accuracy: 0.4528 - loss: 1.5413 - val_accuracy: 0.5426 - val_loss: 1.2935
Epoch 2/25
704/704 - 21s - 29ms/step - accuracy: 0.5998 - loss: 1.1385 - val_accuracy: 0.4714 - val_loss: 1.6647
Epoch 3/25
704/704 - 20s - 29ms/step - accuracy: 0.6580 - loss: 0.9768 - val_accuracy: 0.6566 - val_loss: 0.9744
Epoch 4/25
704/704 - 21s - 29ms/step - accuracy: 0.6952 - loss: 0.8674 - val_accuracy: 0.4680 - val_loss: 1.6799
Epoch 5/25
704/704 - 21s - 29ms/step - accuracy: 0.7263 - loss: 0.7834 - val_accuracy: 0.6372 - val_loss: 1.0879
Epoch 6/25
704/704 - 20s - 29ms/step - accuracy: 0.7525 - loss: 0.7011 - val_accuracy: 0.6936 - val_loss: 0.8807
Epoch 7/25
704/704 - 20s - 29ms/step - accuracy: 0.7755 - loss: 0.6329 - val_accuracy: 0.6248 - val_loss: 1.0990
Epoch 8/25
704/704 - 21s - 29ms/step - accuracy: 0.7986 - loss: 0.5789 - val_accuracy: 0.7284 - val_loss: 0.7838
Epoch 9/25
704/704 - 21s - 29ms/step - accuracy: 0.8195 - loss: 0.5121 - val_accuracy: 0.7234 - val_loss: 0.8097
Epoch 10/25
704/704 - 20s - 29ms/step - accuracy: 0.8343 - loss: 0.4699 - val_accuracy: 0.7082 - val_loss: 0.9456
Epoch 11/25
704/704 - 21s - 29ms/step - accuracy: 0.8518 - loss: 0.4170 - val_accuracy: 0.7324 - val_loss: 0.8074
Epoch 12/25
704/704 - 21s - 29ms/step - accuracy: 0.8661 - loss: 0.3798 - val_accuracy: 0.6546 - val_loss: 1.3052
Epoch 13/25
704/704 - 20s - 29ms/step - accuracy: 0.8794 - loss: 0.3426 - val_accuracy: 0.7352 - val_loss: 0.9200
Epoch 14/25
704/704 - 20s - 29ms/step - accuracy: 0.8912 - loss: 0.3096 - val_accuracy: 0.6426 - val_loss: 1.3035
Epoch 15/25
704/704 - 21s - 29ms/step - accuracy: 0.8993 - loss: 0.2888 - val_accuracy: 0.7208 - val_loss: 0.9450
Epoch 16/25
704/704 - 20s - 29ms/step - accuracy: 0.9090 - loss: 0.2592 - val_accuracy: 0.7094 - val_loss: 1.0277
Epoch 17/25
704/704 - 20s - 29ms/step - accuracy: 0.9114 - loss: 0.2510 - val_accuracy: 0.7524 - val_loss: 0.8752
Epoch 18/25
704/704 - 20s - 29ms/step - accuracy: 0.9205 - loss: 0.2284 - val_accuracy: 0.7346 - val_loss: 0.9405
Epoch 19/25
704/704 - 20s - 29ms/step - accuracy: 0.9251 - loss: 0.2158 - val_accuracy: 0.7364 - val_loss: 0.9730
Epoch 20/25
704/704 - 20s - 29ms/step - accuracy: 0.9315 - loss: 0.1973 - val_accuracy: 0.7230 - val_loss: 1.0843
Epoch 21/25
704/704 - 20s - 29ms/step - accuracy: 0.9346 - loss: 0.1887 - val_accuracy: 0.7210 - val_loss: 1.1215
Epoch 22/25
704/704 - 20s - 29ms/step - accuracy: 0.9409 - loss: 0.1691 - val_accuracy: 0.6968 - val_loss: 1.3247
Epoch 23/25
704/704 - 20s - 29ms/step - accuracy: 0.9429 - loss: 0.1646 - val_accuracy: 0.7390 - val_loss: 1.0593
Epoch 24/25
704/704 - 20s - 29ms/step - accuracy: 0.9465 - loss: 0.1575 - val_accuracy: 0.7292 - val_loss: 1.0413
Epoch 25/25
704/704 - 20s - 29ms/step - accuracy: 0.9425 - loss: 0.1674 - val_accuracy: 0.7450 - val_loss: 1.0578
restored epoch 17 (validation accuracy 0.7524)
Choose on validation accuracy#
The choice between the two networks uses the validation set only, so the test score of the chosen network remains an unbiased estimate.
if deeper_validation > small_validation:
selected_name, model = "deeper network", deeper_model
else:
selected_name, model = "small network", small_model
fig, ax = plt.subplots(figsize=(7, 3.4))
ax.plot(np.arange(1, len(small_history["val_accuracy"]) + 1), small_history["val_accuracy"], marker="o", label="small (5 epochs)")
ax.plot(np.arange(1, len(deeper_history["val_accuracy"]) + 1), deeper_history["val_accuracy"], marker="o", label="deeper (25 epochs)")
ax.set(title="Validation accuracy", xlabel="Epoch", ylabel="Accuracy")
ax.legend()
ax.grid(alpha=0.25)
plt.show()
print(
f"best validation accuracy: small {small_validation:.4f}, deeper {deeper_validation:.4f}; "
f"selected: {selected_name}"
)
best validation accuracy: small 0.6568, deeper 0.7524; selected: deeper network
Evaluate the chosen network on the test set#
The confusion matrix shows which classes are mistaken for which.
test_predictions = predict_classes(model, x_test)
test_accuracy = accuracy_score(y_test, test_predictions)
cm = confusion_matrix(y_test, test_predictions, labels=np.arange(10))
print(f"test accuracy: {test_accuracy:.4f}")
print(
classification_report(
y_test, test_predictions, labels=np.arange(10),
target_names=CLASS_NAMES, digits=3, zero_division=0,
)
)
fig, ax = plt.subplots(figsize=(7, 6))
ConfusionMatrixDisplay(cm, display_labels=CLASS_NAMES).plot(
ax=ax, colorbar=False, values_format="d", xticks_rotation=45
)
ax.set_title("Test confusion matrix (rows: true class)")
plt.show()
mistakes = np.flatnonzero(test_predictions != y_test)[:12]
fig, axes = plt.subplots(2, 6, figsize=(12, 4.6))
for axis in axes.flat:
axis.axis("off")
for axis, index in zip(axes.flat, mistakes):
image = as_image(x_test[index])
axis.imshow(image, cmap="gray" if image.ndim == 2 else None)
axis.set_title(f"{CLASS_NAMES[y_test[index]]} as {CLASS_NAMES[test_predictions[index]]}", fontsize=9)
fig.suptitle("First twelve test mistakes (true as predicted)")
plt.tight_layout()
plt.show()
test accuracy: 0.7473
precision recall f1-score support
airplane 0.790 0.801 0.795 1000
automobile 0.835 0.874 0.854 1000
bird 0.719 0.604 0.657 1000
cat 0.537 0.557 0.547 1000
deer 0.674 0.719 0.696 1000
dog 0.629 0.682 0.655 1000
frog 0.813 0.827 0.820 1000
horse 0.803 0.764 0.783 1000
ship 0.874 0.816 0.844 1000
truck 0.822 0.829 0.826 1000
accuracy 0.747 10000
macro avg 0.750 0.747 0.748 10000
weighted avg 0.750 0.747 0.748 10000
What the results show#
In both saved runs, validation selects the deeper network. Training accuracy continues to improve while validation accuracy levels off, and the best checkpoint occurs before the final epoch. Cats have the lowest test recall. The models also receive different training budgets (5 versus 25 epochs), so the comparison does not isolate network depth. Try training-image flips and crops as a next step, keeping validation and test images unchanged.
Try it yourself in Keras#
Change one choice at a time and keep the data split and evaluation unchanged:
add random horizontal flips to the training images of the deeper network and compare the learning curves.
change the dropout rates of the deeper network and see where overfitting starts.
train the small network for 25 epochs to separate the effect of capacity from that of training time.