Context
To reiterate, my research aims specifically to address the responsiveness of Convolutional Neural Network (CNN) algorithms to data shifts. CNN algorithms are a significant component of computer vision and image classification and train on input datasets to classify/identify objects contained in images.
My research focuses on a statistic known as model overfit, in which the CNN algorithm memorizes the dataset it is trained on without underlying pattern recognition. Models with higher levels of recorded overfit are largely less responsive to future shifts/changes in data and exhibit limited elements of pattern recognition.
Definitions
Title | Definitions |
|---|---|
Neural Network | Neural Networks are Machine Learning algorithms that Mimic the brain’s nervous system (see fig 1). CNNs (a subset of Neural Network Algorithms) work to classify objects contained in images (computer vision). |
Hyperparameter | Model Hyperparameters are the actual components/parameters that define how the neural network functions. Hyperparameters are essentially the building blocks of neural network algorithms and come together to create the finalized model. |
Activation Function | Activation functions are a type of hyperparameter that determine how data is transformed throughout the neural network algorithm to spur the classification/object identification process. Similar to how neurons transform electrical signals to pass on in the brain, activation functions transform imaging data before passing it on further in the model. The three most common activation functions used are Relu, sigmoid, and tanh. |
Hidden Layer | Hidden layers represent the intermediary stages of the CNN between the input and output. The maximum number of hidden layers a neural network can have depends on the sizes of the images that are being classified. CNNs contain two different types of layers: convolutional and pooling layers. Convolutional layers filter and traverse through the image while pooling layers divide the input image into smaller and smaller regions. Data is transformed and passed along through each hidden layer.
|
Training Dataset | The dataset imported into the algorithm is divided into the training and validation dataset. The training dataset is essentially what teaches/instructs the model. |
Validation Dataset | The validation dataset is essentially what the neural network algorithm tests on. The algorithm uses what it learned in the training dataset to correctly classify the images that are in the validation dataset. |
Training Accuracy | The training accuracy is a metric that essentially measures how well the model learns the training dataset. |
Validation Accuracy | The validation accuracy is a metric that measures how well the model correctly classifies the images that are in the validation dataset. |
Model Overfit | Model Overfit represents a phenomenon in which a neural network memorizes (learns the training data too well) without an element of actual pattern recognition. This makes the neural network algorithm less susceptible to changes in data that may arise in the validation dataset, or when the algorithm is actually used in the field. Model Overfit is quantified when the training accuracy exceeds (is greater than) the validation accuracy.
|
Epoch | Each epoch represents a cycle in which the model learns the training dataset and tests against the validation dataset. The validation and training accuracy is recorded for each epoch. In essence, the number of epochs represents how many times the model actually runs. In the context of model overfit, models that overfit at a lower epoch (training accuracy > validation accuracy) indicate that that model is more susceptible to overfit. |
Fig 1. Neural Network Structure accessed 15 March, 2024, <https://bhatnagar91.medium.com/building-a-basic-neural-network-from-scratch-a-step-by-step-guide-7a6f97979ddd>
The vast majority of research conducted on the applicability as well as on the construction process of Convolutional Neural Network focus on maximizing the model accuracy. However, minimal research has been done to address model overfit.
My research aims to close this gap by identifying the multivariate relationships between model parameters and overfit to identify the combination of parameters that minimize overfit levels. As a result, my research aims to provide much needed insight on the prevalence of overfit to make CNN algorithms more responsive to data shifts and less malleable to incorrect, corrupted, or potential training datasets.
To do this, I worked to code my own CNN Algorithm from the ground up, and conducted an experiment in which I changed the activation functions and number of hidden layers used in the model. I then recorded levels of model overfit by identifying at which epoch the training accuracy exceeded the validation accuracy. The epoch number was then recorded for each trial of the experiment.
Research Gap
Process
Materials
Fig 2. Python logo 01 accessed 15 March, 2024 <https://commons.wikimedia.org/wiki/File:Python_logo_01.svg>
I utilized Python to create my neural network due to its relative simplicity and due to the fact that all packages required to construct the most were easily accessible in Python.
Fig 3. Google Colaboratory SVG Logo accessed 15 March, 2024 <https://en.m.wikipedia.org/wiki/File:Google_Colaboratory_SVG_Logo.svg>
I used Google Colaboratory as the IDE/platform in which I wrote and compiled my code. Neural Networks take significant processing power to run, however, Google Colab's free Graphical Processing Units allowed me to run otherwise intensive programs easier.
Fig 4. Tensorflow Logo accessed 15 March, 2024 <https://upload.wikimedia.org/wikipedia/commons/a/ab/TensorFlow_logo.svg>
MNIST Database
Fig 5. Visualization of the MNIST Test Dataset accessed 15 March, 2024 <https://datasets.activeloop.ai/docs/ml/datasets/mnist/>
Fig 6. MNIST Data Plotted accessed 15 March, 2024 <https://www.digitalocean.com/community/tutorials/mnist-dataset-in-python>
Tensorflow's MNIST dataset is a standardized image classification experimentation dataset that features several thousand images of digits with different handwriting styles. MNIST has been used extensively for all types of research regarding image classification. As a result, I utilized the MNIST database to perform my experiment on recording overfit. The Nipuna and E-tanh studies both extensively used the MNIST database for experimentation.
Steps/Process
1: Import the MNIST Database and all necessary Python packages into Google Colaboratory to construct the neural network. These packages contain the various activation functions and hyperparameters used throughout the experiment and are also vital for the model's actual creation. (This includes but is not limited to scikeras, numpy, and matplotlib).
Fig 7 and 8. All Import Systems used in the algorithm
2: Code the actual Neural Network algorithm (this includes coding the components of the algorithm as well as writing and completing the model's function definition). In other words, this step actually involves creating the Neural Network Algorithm to read the MNIST dataset.
Fig 9 and 10. Code used to construct the parameters
Changes in Initial Methodology: Throughout the research process, I changed several of the constant parameters from my initial methodology. I changed variables such as the optimizer (I used Binary Crossentropy) rather than the SGD Optimizer I was planning on using from the Nipuna study. I did this because Binary Crossentropy was easier to implement in the model I was creating. Since i kept the optimizer constant through all trials, it did not impact my results.
3: Code the hidden layers as well as the activation functions that will be used in the neural network. These variables will be changed throughout the experiment. I conducted my experiment with three strata: 2, 4, and 6 hidden layers with one convolutional and one pooling layer in each pair. I then inputted Relu, Sigmoid, and TanH respectively to conduct the various trials of my experiment.
Fig 11. Code used to define Hidden Layers
This trial had two hidden layers (one convolutional and one pooling) and used Relu as the activation function
Dense layers are layers that connect the other layers (convolutional and pooling layers together). Due to this, I did not count these layers in the overall count of hidden layers (however these layers remained constant throughout the experiment (except the activation function)
Softmax was used as the activation function for the last dense layer for each trial of the experiment. Softmax had to be used to correctly output the neural network's classification
Fig 12. (Sigmoid: 6 layers)
4: Run/Compile the neural network for each trial (I ran each activation function three times for each group of hidden layers (2, 4, and 6) and record at which epoch the training accuracy exceeds the validation accuracy.
30 epochs were used in the experiment rather than the initial 100 I had planned in the initial methodology due to processing power limitations.
Fig 13 and 14. Code to compile/run model and to graph val/training accuracy
The above two excerpts of code both run/compile the code and record the validation and training accuracy for each epoch. The code will then automatically output a graph that shows the progression of training and validation accuracy throughout the model.
Example of Data
(Training Accuracy)
This is an example of raw data outputted from one trial of the experiment and shows all thirty epochs as well as the training accuracy and validation accuracy for each epoch)
In this trial, overfit occurred at the 26th epoch in which the validation accuracy first exceeded the training accuracy (0.9412 > 0.9410)
Fig 15 and 16. Example Raw Data
An example of the automatically outputted graph that illustrates the progression of validation and training accuracy throughout the algorithm
Fig 17. Example of Outputted Graph
Results
All Results including all trials, raw data, and graphs can be found on this document: https://docs.google.com/document/d/1o8Y2QoG8jBDqESihU8qmU1dv0-2J4Da8HgaKQkFKrJ0/edit
Trial | Average Epoch of Overfit (over 3 trials) |
|---|---|
Tanh - 6 Layers | 5.67/30 |
Sigmoid - 6 Layers | 22/30 |
RELU - 6 Layers | 11/30 |
Tanh - 4 Layers | 6.33/30 |
Sigmoid - 4 Layers | 26.67/30 |
RELU - 4 Layers | 8.33/30 |
TanH - 2 Layers | 5.67/30 |
Sigmoid - 2 Layers | 27.33/30 |
RELU - 2 Layers | 5/30 |
Averaged Results
Using a Chi Squared test for Significance among hidden layer groups (p = 0.05)
Conclusions
-
Overall, models with Sigmoid as the activation function exhibited lower levels of model overfit across all strata (hidden layer sizes)
-
There was a statistically significant correlation between the number of hidden layers and the performance of models that employed Relu - An increased number of hidden layers led to decreased levels of model overfit.
-
There was no statistically significant correlation between the number of hidden layers and the performance of models that employed Tanh
-
There was actually a statistically significant negative correlation between the number of hidden layers and the performance of models that employed Sigmoid - An increased number of hidden layers led to increased levels of model overfit.
Discussion
-
My data disproves my initial hypothesis: I initially expected relu to exhibit lower levels of model overfit, however, sigmoid was shown to provide the best results in mitigating model overfit in all groups of hidden layers (even despite the performance of sigmoid models decreasing as the number of hidden layers increased).
-
My data also shows that Relu has the potential to create models with lower levels of overfit when the amount of hidden layers is increased.
Limitations
-
Processing Power/Time: I was not able to run my models at the initially specified 100 epochs due to the significant amount of power and time needed to run the model. As a result, there were several inconsistencies in my data that could have been prevented if more trials were run.
-
To overcome this, I ran multiple trials for each activation function - hidden layer combination to generate statistically accurate predictions
-
Observed Data Inconsistency - Sigmoid: 4 HL
In this trial, overfit occurred at the 28th epoch (0.9790 > 0.9783). However, the 29th and 30th trials recorded no overfit.
Fig 18 and 19. Example of Inconsistent Data
Conclusion
Model Overfit in neural networks present an extensive societal drawback of the use of AI. To reiterate, in a social context, with computer vision models being increasingly incorporated into fields such as criminal justice, the potential that models overfit on inherently biased data presents numerous implications, such a discriminatory criminal profiling. Due to the tremendous implications attributed to this marginally researched topic, my research presents a broader view on the multivariate relationships between model parameters and model overfit.
My research is largely representative of the next step to make AI Algorithms more susceptible to data shifts, less prone to errors/bias contained in training data, and more equitable as a whole.