ML System Design for Data Scientists (How Senior Data Scientists Actually Think About ML Systems)
This is the real shift in the job market
Something I've noticed over the years working with data scientists at all levels...
A lot of times, seniors and juniors don’t differ too much in raw skill. Both can build models, both can tune them, both can read a paper and implement it. And thanks to AI, that technical gap is becoming even narrower.
But when you ask a senior data scientist to build an ML model, they don’t start with the model itself (like most juniors do); they start sketching out the whole system: where the data comes from, how it flows, where the model sits, how it gets served, how you know it’s working after it ships.
The model becomes just one box in a bigger picture.
The reason senior data scientists can think this way is because they’ve built a foundation of what’s called ML system design. It’s how you learn to see the whole picture instead of just the model. And it’s what a lot of the education out there just doesn’t teach.
That’s what I want to break down for you today.
And I made a video and built an interactive tool to help data scientists get to this foundation earlier in their careers, because it took me way too long to get there on my own.
Let’s get to it!
The ML System Design Framework
Here is the 6-step framework I like to use as a Senior Data Scientist. This same framework has helped me ace multiple technical interviews over the years.
Frame the problem: What is the system actually solving? What decision does it inform? How will you know if it worked? Most ML projects that fail didn’t fail at the modeling; they failed here.
Data and data flow: What data do you have? How does it get from where it lives to where the model runs? Can you get it at the moment you need it, in the shape you need it?
Feature engineering: What features does the model need, and how do you compute them consistently in training and in production? Training-serving skew starts here.
Model design and selection: What kind of model actually fits the constraints (latency, size, cost, interpretability)? How do you train it and evaluate it in a way that reflects how it’ll be used?
Serving and deployment: How does the model get delivered so it stays reliable, scales when demand shifts, and can be maintained by more than one person?
Evaluation and monitoring: Once it’s live, how do you know it’s still working? What signals tell you the world has shifted, or that your model has stopped keeping up?
Now, I know that's a lot to take in at once, and it's even harder to really internalize these concepts without a system you can tie them back to.
So I think this will help…
Learn ML System Design interactively
Every time I’ve tried to teach ML system design, static diagrams don’t really do the job well.
They help, but they don't really show how the pieces connect in an effective way, which is kind of the whole point.
So I built this interactive tool to help data scientists learn ML system design more effectively.
It’s based on a trial-to-paid conversion XGBoost model, with every component clickable.
You get to see: data ingestion, feature engineering, training, experiment tracking, the registry, serving, monitoring, retraining. The whole thing, end-to-end, with the real tools and the real trade-offs.
Watch me walk through it
The video walks through the six steps applied to a real production ML system: the trial-to-paid conversion model from the interactive tool.
A lot of my thinking on this is grounded in Chip Huyen's Designing Machine Learning Systems, which I lean on throughout.
If you want the full framework and the applied walkthrough, watch it here:
One last thing
"Believe it or not, the gap between "I train models" and "I design systems" is smaller than most people think. For many data scientists, it's not a massive technical leap so much as a shift in how you look at the work.
But small doesn’t mean easy.
Most data scientists learn the model-first approach because that’s how school and most bootcamps teach it. Getting to the system-first way of thinking usually requires deploying a few things, watching them break in weird ways, and slowly building the mental model for how everything fits together. That process took me years, and it happened mostly by accident.
I built the tool and made the video because I wanted to give people a shortcut to that mental model. Not to replace the experience of building real systems, because nothing replaces that. But to make the mental model easier to see and start applying earlier.
I hope you can make the most out it!
A couple of other great resources:
📚 Want to learn what it takes to deploy ML models? Check out my free mini-course on MLOps for Data Scientists.
🎥 Want to follow along on YouTube? I just launched a channel for data scientists. Don’t forget to subscribe to not miss any videos.
Thank you for reading! And stay tuned for next week’s article in the Engineering Skills for Data Scientists series.
- Andres Vourakis
Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!