How can minimalist models achieve even higher accuracy on MNIST?
Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.
Ask questions, find answers and collaborate at work with Stack Overflow for Teams. Explore Teams
I recently came across a fascinating discussion on how a simple logistic regression model can achieve around 92% classification accuracy on the MNIST dataset (reference: How does a simple logistic regression model achieve a 92% classification accuracy on MNIST?).
For a linear model, 784 neurons (one for each pixel) seem to be the minimal configuration. However, I’ve been exploring how far we can push minimalistic models further while still achieving high performance. I’ve developed a neural network with only 702 parameters, and it achieves 98.2% accuracy on MNIST. You can find the implementation of my model here: 702-parameter MNIST model.
I’m curious if there are other minimalist approaches—like neural networks with very few parameters—that can achieve accuracy beyond 92%, possibly approaching 98-99%, without resorting to highly complex architectures.
