Constructing and Training Multi-Layer Perceptrons with Different Activation Functions and Optimizers

Introduction & Overview

Read summaries of the section's main ideas at different levels of detail.

Audio Book

Dive deep into the subject with an immersive audiobook experience.

Key Concepts

• Multi-Layer Perceptron (MLP): Understanding input, hidden, and output layers.

• Dense Layer: The fundamental building block of MLPs in Keras.

• Activation Functions: Sigmoid, ReLU, Softmax – their roles and where they are typically used.

• Loss Functions: e.g., sparse_categorical_crossentropy for multi-class classification with integer labels.

• Optimizers: Stochastic Gradient Descent (SGD), Adam, RMSprop – how they update weights.

• Metrics: e.g., accuracy for classification tasks.

• Forward Propagation: Implicitly understood as the model makes predictions.

• Backpropagation: Implicitly handled by the optimizer and loss function during training.

• Epochs and Batch Size: Hyperparameters controlling the training process.

• Lab Setup:

• You will typically work in a Python environment, likely using a Jupyter Notebook or Google Colab, which provides an interactive coding experience.

• Prerequisites:

• Basic understanding of Python programming.

• Familiarity with NumPy for array manipulation.

• Conceptual understanding of neural networks (from the lecture).

• TensorFlow and Keras installed (if not using Colab, which has them pre-installed).

• To install if not already present

• pip install tensorflow matplotlib scikit-learn

• Dataset:

• We will typically use a well-known, relatively simple dataset for classification, such as:

• MNIST Handwritten Digits Dataset: A classic dataset of 28x28 grayscale images of handwritten digits (0-9).

• Fashion MNIST Dataset: Similar to MNIST but with images of clothing items.

• Iris Dataset: A very simple, small tabular dataset for multi-class classification (though less commonly used for deep learning intros due to its simplicity).

• For this lab, let's assume we are using the Fashion MNIST dataset, which provides a good balance of complexity for an introductory deep learning lab.

• Lab Tasks:

• Task 1: Data Preparation

• Load the Dataset: Load the Fashion MNIST dataset using Keras's built-in utility.

• Separate into training and testing sets (X_train, y_train, X_test, y_test).

• Preprocess the Data:

• Normalize Pixel Values: Image pixel values are typically in the range 0-255. Scale them to 0-1 by dividing by 255.0.

• Flatten Images: MLPs expect a 1D input vector. Flatten the 28x28 images into a 784-element vector.

• One-Hot Encode Labels (Optional but good practice for categorical_crossentropy): If using categorical_crossentropy loss, convert integer labels (0-9) to one-hot encoded vectors. If using sparse_categorical_crossentropy, this step is not needed as it expects integer labels.

• Task 2: Building and Training MLPs with Different Activation Functions

• In this task, you will build three separate MLP models, each using a different activation function for its hidden layers (and the same output activation for multi-class classification).

• Model Configuration (Example):

• Input Layer: 784 neurons (for flattened 28x28 images).

• Hidden Layers: Two hidden layers, e.g., 128 neurons in the first, 64 neurons in the second.

• Output Layer: 10 neurons (for 10 classes in Fashion MNIST). Softmax activation must be used here for multi-class classification to output probabilities.

• Optimizer: Use 'Adam' for all models in this task to isolate the effect of activation functions.

• Loss Function: sparse_categorical_crossentropy (if labels are integers) or categorical_crossentropy (if labels are one-hot encoded).

• Metrics: accuracy.

• Epochs: 10-20 (a reasonable starting point).

• Batch Size: 32 or 64.

• Model 1: ReLU Activation

• Construct an MLP using relu activation for both hidden layers.

• Compile the model.

• Train the model using model.fit().

• Evaluate the model on the test set using model.evaluate().

• Store training history (loss and accuracy over epochs).

• Model 2: Sigmoid Activation

• Construct an MLP using sigmoid activation for both hidden layers.

• Compile the model.

• Train the model.

• Evaluate the model.

• Store training history.

• Model 3: Experiment with other activations (e.g., Tanh, if time permits)

• (Optional) Construct an MLP using another activation function (e.g., tanh).

• Compile, train, evaluate, and store history.

• Analysis:

• Plot the training and validation accuracy/loss for each model over epochs.

• Compare the convergence speed and final accuracy of models trained with ReLU vs. Sigmoid (and others).

• Discuss observations, e.g., which converged faster, which achieved higher accuracy, and why (referencing vanishing gradients, dying ReLUs, etc.).

• Task 3: Building and Training MLPs with Different Optimizers

• In this task, you will build three separate MLP models, keeping the architecture and activation functions (e.g., relu for hidden, softmax for output) consistent, but using different optimizers.

• Model Configuration (Example):

• Input Layer: 784 neurons.

• Hidden Layers: Two hidden layers, 128 and 64 neurons, both using relu activation.

• Output Layer: 10 neurons with softmax activation.

• Loss Function: sparse_categorical_crossentropy.

• Metrics: accuracy.

• Epochs: 10-20.

• Batch Size: 32 or 64.

• Model 1: SGD Optimizer

• Construct the MLP.

• Compile the model using optimizer='sgd'. You can also use tf.keras.optimizers.SGD(learning_rate=0.01) for more control.

• Train the model.

• Evaluate the model.

• Store training history.

• Model 2: Adam Optimizer

• Construct the MLP.

• Compile the model using optimizer='adam'.

• Train the model.

• Evaluate the model.

• Store training history.

• Model 3: RMSprop Optimizer

• Construct the MLP.

• Compile the model using optimizer='rmsprop'.

• Train the model.

• Evaluate the model.

• Store training history.

• Analysis:

• Plot the training and validation accuracy/loss for each model over epochs.

• Compare the convergence speed, stability of training (less oscillations), and final accuracy of models trained with SGD, Adam, and RMSprop.

• Discuss observations, e.g., which optimizer performed best, which exhibited more fluctuations, and why (referencing adaptive learning rates, momentum, etc.).

• Task 4: (Optional) Hyperparameter Tuning and Further Experimentation

• Try changing the number of hidden layers or neurons per layer.

• Experiment with different learning rates for the optimizers.

• Observe the effect of batch_size on training dynamics.

• Try using different datasets.

• Deliverables:

• A Jupyter Notebook or Python script containing all the code for data preparation, model construction, training, and evaluation for all tasks.

• Plots showing training and validation loss/accuracy curves for each experiment.

• A brief summary of your observations and conclusions for each task, discussing the impact of different activation functions and optimizers on model performance, convergence, and training stability.

• This lab will provide invaluable practical experience, allowing you to bridge the gap between theoretical knowledge and the hands-on application of deep learning concepts.

• ---

• Narrative Content Sessions

• Session 1: Setting up the Lab Environment and Data

• Context: Getting ready to build our first MLP, focusing on data loading and preprocessing.

• Narrative Content:

• Teacher: "Alright everyone, welcome to our first Deep Learning lab\! Today, we're going to get our hands dirty building and training MLPs. First things first, what's the very first step when working with any machine learning model?"

• Student\_1: "Load the data\!"

• Teacher: "Exactly\! For this lab, we'll use the Fashion MNIST dataset, which is conveniently built into Keras. After loading, what's a crucial preprocessing step for image data like pixel values, which are usually 0-255?"

• Student\_2: "Normalize them, divide by 255 to get them between 0 and 1."

• Teacher: "Perfect\! Why do we do that?"

• Student\_3: "To help the neural network learn better, maybe prevent large numbers from dominating?"

• Teacher: "Precisely, it helps with numerical stability and faster convergence. And since MLPs expect flat inputs, what do we need to do to our 28x28 images?"

• Student\_4: "Flatten them into a single long vector."

• Teacher: "Excellent\! So, for a 28x28 image, how long will that vector be?"

• Student\_1: "784\!"

• Teacher: "You got it\! So, for Task 1, focus on getting your data loaded, normalized, and flattened. This foundation is key before we even think about building the network."

• Session 2: Experimenting with Activation Functions

• Context: Building MLPs and observing the impact of ReLU vs. Sigmoid.

• Narrative Content:

• Teacher: "Now that our data is ready, let's build our first MLP. Remember, an MLP has input, hidden, and output layers. What kind of Keras layer do we use for these fully connected layers?"

• Student\_2: "A Dense layer\!"

• Teacher: "Correct\! For our output layer, since we have 10 clothing classes and want probabilities, which activation function is a must?"

• Student\_3: "Softmax\!"

• Teacher: "Perfect\! Now for our hidden layers, we'll experiment. First, build a model using relu activation for your hidden layers. Compile it with the 'Adam' optimizer and sparse_categorical_crossentropy loss. What was a key advantage of ReLU that we discussed in the lecture?"

• Student\_4: "It helps with the vanishing gradient problem, so it learns faster."

• Teacher: "Exactly\! After training that, build a second model, identical in structure, but use sigmoid for the hidden layers. What was the main drawback of Sigmoid, especially in deeper networks?"

• Student\_1: "Vanishing gradients, which make learning very slow in early layers."

• Teacher: "You're on fire\! After training both, make sure to plot their training history. What are you looking for when comparing these plots?"

• Student\_2: "How quickly they learn, and what their final accuracy is."

• Teacher: "Excellent\! Compare the convergence speed and final performance. This hands-on comparison will solidify your understanding of why ReLU is often preferred."

• Session 3: Exploring Different Optimizers

• Context: Understanding how different optimizers influence the training process.

• Narrative Content:

• Teacher: "Great work with the activation functions\! Now, let's keep our MLP architecture and ReLU activations consistent, but change the 'engine' that drives learning: the optimizer. We'll try SGD, Adam, and RMSprop. What does an optimizer fundamentally do during training?"

• Student\_3: "It updates the weights and biases to reduce the error."

• Teacher: "Spot on\! It's like guiding a blindfolded person down a mountain. First, implement the model with optimizer='sgd'. What's the main characteristic of SGD's updates?"

• Student\_4: "They're noisy because it updates for each sample, or small batch."

• Teacher: "Yes, that noise can help it escape local minima, but it can also lead to oscillations. Next, implement with optimizer='adam'. Why is Adam often considered a good default choice?"

• Student\_1: "It adapts the learning rate for each weight and combines ideas from other optimizers, so it usually performs well."

• Teacher: "Precisely\! It's generally very robust. Finally, try optimizer='rmsprop'. What's its main contribution?"

• Student\_2: "It also adapts learning rates, but based on squared gradients, helping with vanishing/exploding gradients."

• Teacher: "Fantastic\! After training all three, again, plot their histories and compare. What differences in the training curves might indicate how well each optimizer performs?"

• Student\_3: "How smooth the loss curve is, how fast it goes down, and the final accuracy."

• Teacher: "Absolutely\! Pay attention to oscillations and convergence speed. This will give you a real feel for how these optimizers guide the learning process."

• ---

• Audio Book

• Chunk Title: Lab Introduction: Building Your First Neural Network

• Chunk Text: Welcome to the Deep Learning Lab\! Here, you'll gain hands-on experience by constructing and training Multi-Layer Perceptrons (MLPs) using TensorFlow and Keras. This practical session will deepen your understanding of neural network components.

• Detailed Explanation: This lab is your opportunity to apply the theoretical knowledge of neural networks you've gained. We'll be using Keras, a high-level, user-friendly API built on top of TensorFlow, which simplifies the process of defining and training neural networks. You'll start by preparing a dataset, typically image data like Fashion MNIST, by normalizing pixel values and flattening the images into a format suitable for an MLP. This preparation is crucial because it ensures your data is in the right shape and scale for the network to learn effectively.

• Real-Life Example or Analogy: Imagine you're an aspiring chef. You've studied recipes (theory), and now in this lab, you're stepping into the kitchen. Your first task is to prepare the ingredients (data preprocessing) – washing, chopping, and measuring – before you even start cooking (building the model).

• \--

• Chunk Title: Experimenting with Activation Functions

• Chunk Text: In this section, you will build and train MLPs using different activation functions like ReLU and Sigmoid in the hidden layers, observing their impact on model performance.

• Detailed Explanation: Activation functions are critical non-linear components within a neural network. You will construct at least two separate MLP models with identical structures: one using the Rectified Linear Unit (ReLU) activation in its hidden layers, and another using the Sigmoid activation. Both models will use Softmax in their output layer for multi-class classification, as it provides interpretable probabilities. By training both models and comparing their learning curves (loss and accuracy over epochs), you will empirically observe how ReLU helps mitigate the "vanishing gradient problem" leading to faster convergence, compared to Sigmoid which can suffer from this issue, especially in deeper networks. This direct comparison will highlight the practical implications of choosing the right activation function.

• Real-Life Example or Analogy: Think of activation functions as filters in a water purification system. A ReLU filter (like a simple gate) lets clean water through efficiently, while a Sigmoid filter (like a more complex valve) might slow down the flow (vanishing gradient) if the water pressure is too high or too low. You're testing which filter helps process the "data flow" most effectively.

• \--

• Chunk Title: Understanding Optimizers in Practice

• Chunk Text: This part of the lab focuses on training MLPs using various optimizers such as Stochastic Gradient Descent (SGD), Adam, and RMSprop, to see how each guides the weight updates and affects convergence.

• Detailed Explanation: Optimizers are the algorithms that control how the neural network's weights and biases are adjusted during training, based on the gradients calculated during backpropagation. You will build and train more MLPs, this time keeping the architecture and activation functions (e.g., ReLU for hidden layers) consistent, but varying the optimizer. You'll experiment with SGD, known for its noisy but potentially efficient updates; Adam, a popular adaptive optimizer that often converges quickly; and RMSprop, another adaptive optimizer good for non-stationary objectives. By comparing their training histories and final performance metrics, you'll see how different optimizers influence the learning rate, the smoothness of convergence, and the ultimate accuracy of your model.

• Real-Life Example or Analogy: Imagine you're trying to find the bottom of a dark valley (minimum loss) while blindfolded. SGD is like taking small, random steps based on where your foot just landed. Adam is like having a smart guide who remembers past successful steps (momentum) and adjusts your stride dynamically based on how rocky the terrain has been (adaptive learning rate). RMSprop is similar to Adam but focuses more on the terrain's 'rockiness' for step adjustments. You're seeing which 'guide' helps you reach the valley floor fastest and most smoothly.

• ---

• Glossary

• Multi-Layer Perceptron (MLP): A type of feedforward artificial neural network consisting of multiple layers of nodes in a directed graph.

• TensorFlow: An open-source machine learning framework developed by Google.

• Keras: A high-level neural networks API, typically running on top of TensorFlow, designed for fast experimentation.

• Activation Function: A non-linear function that determines the output of a neuron, such as Sigmoid, ReLU, or Softmax.

• Optimizer: An algorithm (e.g., SGD, Adam, RMSprop) used to adjust the weights and biases of a neural network during training to minimize the loss function.

• Epoch: One complete pass through the entire training dataset during model training.

• Batch Size: The number of training examples utilized in one iteration during training.

• Loss Function: A function that quantifies the error between the predicted output and the actual target value.

• Metrics: Quantities that are monitored during training and evaluation to assess model performance (e.g., accuracy, mean squared error).

• ---

• Estimated Study Time

• 60-90 minutes (including coding and analysis)

• ---

• Reference Links

• TensorFlow Keras Sequential Model Guide

• Keras Documentation: Layers API

• TensorFlow Tutorials: Basic Classification (Fashion MNIST)

• Deep Learning with Python (Chollet) - Chapter on Keras basics (Conceptual reference, not directly linkable content)

• An overview of gradient descent optimization algorithms (More advanced, but good for optimizer details)

• ---

• Key Concepts

• MLP Construction: Using tf.keras.Sequential and Dense layers.

• Activation Function Impact: Comparing relu vs. sigmoid for hidden layers.

• Optimizer Comparison: Evaluating SGD, Adam, RMSprop for convergence and performance.

• Data Preprocessing: Normalization and flattening for image inputs.

• Training Workflow: model.compile() and model.fit().

• ---

• Examples

• Code for a ReLU-activated MLP:

• from tensorflow.keras.models import Sequential

• from tensorflow.keras.layers import Dense

• Assuming input_shape=(784,) and output_units=10

• model_relu = Sequential([

• Dense(units=128, activation='relu', input_shape=(784,)),

• Dense(units=64, activation='relu'),

• Dense(units=10, activation='softmax')

• ])

• model_relu.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

• history_relu = model_relu.fit(X_train, y_train, epochs=10, batch_size=64, validation_split=0.2)

• Code for an MLP with SGD optimizer:

• from tensorflow.keras.optimizers import SGD

• model_sgd = Sequential([

• Dense(units=128, activation='relu', input_shape=(784,)),

• Dense(units=64, activation='relu'),

• Dense(units=10, activation='softmax')

• ])

• model_sgd.compile(optimizer=SGD(learning_rate=0.01), loss='sparse_categorical_crossentropy', metrics=['accuracy'])

• history_sgd = model_sgd.fit(X_train, y_train, epochs=10, batch_size=64, validation_split=0.2)

• Plotting Example (Conceptual):

• import matplotlib.pyplot as plt

• plt.plot(history_relu.history['accuracy'], label='ReLU Training Accuracy')

• plt.plot(history_relu.history['val_accuracy'], label='ReLU Validation Accuracy')

• plt.title('Model Accuracy with ReLU Activation')

• plt.xlabel('Epoch')

• plt.ylabel('Accuracy')

• plt.legend()

• plt.show()

• ---

• Flashcards

• Term: Dense layer

• Definition: A fully connected layer in a neural network where every neuron is connected to every neuron in the previous layer.

• Term: model.compile()

• Definition: Keras method to configure the learning process of a model, specifying the optimizer, loss function, and metrics.

• Term: model.fit()

• Definition: Keras method to train a model for a fixed number of epochs on a given dataset.

• Term: validation_split

• Definition: A parameter in model.fit() that reserves a fraction of the training data to be used as validation data for monitoring training progress.

• ---

• Memory Aids

• Rhyme: Activations ignite, optimizers make weights right, for a network that learns and shines bright.

• Story: Imagine your MLP is a student preparing for an exam. The activation function is like the student's study method – some methods (ReLU) might be direct and efficient, while others (Sigmoid) might involve more processing and lead to slower understanding. The optimizer is like the tutor guiding the student's learning strategy. An SGD tutor might give quick, frequent tips, while an Adam tutor adapts their teaching style based on the student's past performance and learning speed. This lab helps you see which combination (study method + tutor) leads to the best exam results (model performance).

• Mnemonic: Think A.L.O.M.T.E. for the Keras workflow: Architecture, Loss, Optimizer, Metrics, Train, Evaluate.

Examples & Applications

Memory Aids

Interactive tools to help you remember key concepts

Glossary