Acknowledgements and Final Postings

Neural networks are machine learning algorithms that are inspired by the workings of the nervous system. In the field of neural net experimentation, the vast majority of research focuses specifically on maximizing model accuracies and the speed in which such models train on data and work to classify images. However, very minimal research has focused specifically on model overfit: a phenomenon in which the AI algorithm completely memorizes training data - including any external noise and variance - without recognizing the underlying patterns behind the dataset. Model overfit can have enormous potential implications in that if algorithms overfit on potentially incorrect or corrupt data which contain implicit biases, those biases will essentially be replicated when the algorithm is implemented in the field. In order to address the profound gap in research centered on this specific subject, I developed the research question: How does the use of different activation functions impact the presence of model overfitting in neural networks when the number of hidden layers used is altered? To answer this research question I built off the methodologies of previous research done in the field of image classification to build my own neural network model. Subsequently, I conducted a stratified experiment in which I recorded the influence of changing model hyperparameters (type of activation function used and the # of hidden layers) on model overfit. I quantified model overfit by the training cycle in which the model's training accuracy exceeds the validation, or testing, accuracy. Overall, I found that the Sigmoid activation function had the best performance in terms of mitigating overfit and that changing the number of hidden layers used ultimately had a different impact on the presence of overfit in regards to the type of activation function used.