👨‍💻 Welcome to the Engineering Skills for Data Scientists series, where I’ll be teaching the engineering skills that quietly separate data scientists who can ship from the ones who can’t. This is article 2 of 5. You can find the full series here.

Here’s a question worth asking yourself…

When was the last time you were genuinely sure your code worked?

Not “it ran without errors.” Not “the dashboard looked right.” Not “my colleague said it looked good.” Actually sure.

Most data scientists, if they’re honest, don’t operate from that place. They operate from hope. The code seemed to work on the sample I checked. The model trained without crashing. The pipeline ran end-to-end at least once.

That’s a fine place to be when you’re exploring. It’s a terrible place to be when your code is making decisions in production.

There are two engineering habits that move you from hoping to knowing:

• Testing: checks your code does what you think it does, before it ships.

• Logging: gives you visibility into what your code is doing, after it ships.

Together, they’re how engineers sleep at night. And they’re surprisingly underrepresented in most data science workflows, mostly because nobody taught us how.

And to be fair, when you’re moving fast on analysis, it’s hard to justify the upfront cost. Until something breaks in production at 2am and you’re staring at a model that’s been making wrong predictions for a week.

In this article, I’ll walk you through both, the practical minimum a data scientist needs.

We’ll also look at how to use AI to make adopting these habits significantly cheaper than it used to be, including the specific setup that turns AI from a generic code generator into a competent assistant for your conventions.

Let’s get to it!

Here’s what we’ll cover

• Part 1: Testing your code before it ships (and how to use AI to make it cheap)

• Part 2: Logging what your code does after it ships (and how to use AI to make it cheap)

• 🎥 Try it on your own (a short walkthrough + exercises)

• Why do these two go together

Part 1: Testing your code before it ships

A test is just a piece of code that checks whether your other code does what you think it does. That’s it!

You give it some input, you say “here’s what I expect,” and you let the test tell you if reality matched expectations.

Now, something you might be wondering about. If you've been writing Python for a while, or you've been around engineers, you've probably heard three terms thrown around: assert, unit test, and pytest. They're related, but they're not the same thing.

• assert is a Python keyword. It’s the check itself: “This better be true, or raise an error.”

• Unit test is the kind of test you’re writing: one that checks a single function in isolation, with no database, no API calls, no side effects. Just the logic.

• pytest is the runner. It finds your tests, executes them, and reports which passed and failed.

Put them together, and a unit test is: a function that uses assert to check one piece of logic, organized so pytest can find and run it.

Unit tests are 90% of what you’ll write as a data scientist. The other kinds (integration tests, end-to-end tests) matter on a bigger scale, but unit tests are where the payoff is highest, so that’s what we’ll focus on here.

But before you can write tests, your code has to be testable. This is the part nobody tells you about.

What “testable” code looks like

Most data science code is written as one long function or notebook cell that does everything: loads the data, transforms it, fits the model, evaluates it, saves the output.

There’s nothing wrong with that for exploration. But it’s almost impossible to test.

Here’s a function that’s hard to test:

def run_churn_analysis():
    df = pd.read_csv("data/customers.csv")
    df = df[df["tenure_days"] >= 30]
    rate = df["churned"].sum() / len(df)
    print(f"Churn rate: {rate:.2%}")
    df.to_csv("data/eligible_customers.csv")

What would you even test? It reads a specific file, mutates it, and writes another specific file. Everything is glued together.

Here’s the same logic, refactored to be testable:

def compute_churn_rate(customers: pd.DataFrame) -> float:
    """Returns the share of customers who churned this month.
    Excludes customers who signed up this month (incomplete tenure).
    """
    eligible = customers[customers["tenure_days"] >= 30]
    if len(eligible) == 0:
        return 0.0
    return eligible["churned"].sum() / len(eligible)

One small function. Takes a DataFrame, returns a number, does one thing with clear business rules: exclude new customers, return the share that churned. Now you can test it independently with whatever data you want, no files, no side effects, no print statements to read.

💡 Quick tip: separate logic from the I/O. The logic (transforming data, computing metrics, applying business rules) is what you test. The I/O (reading files, writing to databases, calling APIs) sits outside, in glue code that’s harder to test but doesn’t need to be tested as carefully.

Writing your first real test

Python’s built-in testing tool is pytest.

Install it once with pip install pytest, then write a test in a file that starts with test_:

# test_churn.py
import pandas as pd
from my_module import compute_churn_rate

def test_compute_churn_rate_excludes_new_customers():
    df = pd.DataFrame({
        "tenure_days": [45, 60, 10],   # third customer is too new
        "churned": [True, False, True]
    })
    result = compute_churn_rate(df)
    assert result == 0.5  # only 2 eligible customers, 1 churned

Run it from your terminal with pytest. If the function works, the test passes silently. If it doesn’t, you get a clear error telling you exactly what failed and why.

That’s a test. One function, three lines, no ceremony.

Notice what makes this test actually useful: it pins down a specific business rule (the tenure filter) and a specific behavior (only eligible customers count toward the rate). If someone removes the filter, changes the threshold, or includes everyone in the denominator, the test fails immediately.

The pattern that holds up across almost any test:

1. Arrange: set up the input data

2. Act: call the function

3. Assert: check the output is what you expected

What’s worth testing

Not everything needs a test.

The right answer is closer to “test the things that would cause real damage if they broke quietly”:

• Functions that compute metrics (a churn rate, an ARPU, a conversion). If these are wrong, your decisions are wrong.

• Feature transformations that feed into a model. If these are wrong, your predictions are wrong.

• Business logic (filtering, deduplication, joining rules). If these are wrong, your downstream analysis is wrong.

• Edge cases you’ve already been bitten by. Once a bug shows up, write a test that would have caught it.

You don’t need 100% coverage. You need to cover the things that, if wrong, would hurt.

Using AI to make testing cheap

The reason most data scientists skip testing isn’t that they don’t see the value. It’s the upfront cost. Writing tests for a function you already wrote feels like double work.

AI changes that calculation, but only if you set it up right.

Pasting a function into Claude and saying “write tests for this” gives you tests. It doesn’t give you good tests.

The single highest-leverage move is setting up a project-level instruction file. Most AI coding tools (Claude Code, Cursor, Windsurf) read a file called CLAUDE.md or AGENTS.md in your project root and load it into every session automatically.

That’s where you put your testing conventions.

A useful starter:

## Testing
- Use pytest. Run with `pytest tests/ -v`.
- Tests live in `tests/`, matching the structure of `src/`.
- Fixtures go in `conftest.py`.
- Use `unittest.mock` for mocking. Never call real APIs in tests.

## Testing rules
- Never modify existing tests to make them pass. Fix the implementation instead.
- Write the failing test before fixing any reported bug.
- Mock network calls and time. Never use real credentials.
- Aim to cover the logic that, if wrong, would hurt: metrics, transformations, business rules.

With this in place, AI defaults to your standards instead of generic patterns. The single most important rule is the first one under “Testing rules.” Without it, AI will quietly change your tests to make them pass instead of fixing the actual bug, and you’ll lose the entire point of testing.

A few prompts that work well once the conventions are set:

• “Write pytest tests for this function. Cover the happy path and at least three edge cases.” You get a working test file in seconds.

• “What inputs could break this function?” This is genuinely useful, even if you don’t use the tests it writes. AI is good at surfacing edge cases you didn’t think of.

• “This function is too tangled to test. Refactor it so the logic is separable from the I/O.” Then you write tests against the refactored version.

💡 AI will happily generate tests that pass without actually testing anything meaningful. Tests that just assert the function returns something, or that mock so much that nothing real gets verified. Read what it gives you. The skill isn’t getting AI to write tests, it’s knowing which tests are worth keeping (that’s still up to the human to decide).

Part 2: Logging what your code does after it ships

Testing catches problems before your code runs in production. But once it’s out there, you need a different tool: logging.

Logging is how your code tells you what it’s doing while it runs. Without it, when something goes wrong, you’re guessing. With it, you can actually answer questions like:

• When did this start failing?

• What was the input?

• Where in the pipeline did it break?

Why print() doesn’t cut it

Every data scientist’s first instinct is to debug with print(). It works fine in a notebook. You see the output immediately, you check it, you move on.

In production, print() falls apart for three reasons:

• Nobody’s watching the output: Your code runs at 3am on a server you’ve never logged into. There’s no one to read the print statements.

• The output disappears: Print goes to stdout. If the process crashes or stdout isn’t captured, the output is gone.

• Every message looks the same: A print is a print. There’s no way to say “this one is important, this one is just debug noise.”

What you need is something that records what your code is doing, persists it somewhere you can read later, and lets you separate the important from the routine. That’s logging.

What logging actually does

Python has a built-in logging module. The minimum setup:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Pipeline started")
logger.warning("Found 12 rows with missing values")
logger.error("Failed to load model from S3")

Three things to notice:

• logger.info, logger.warning, logger.error are levels. They let you mark how important each message is.

• The messages go wherever you configured them to go (the console by default, but easily a file or a log aggregator like Datadog).

• You can filter by level when you read them later. Show me only the errors, or only the warnings and errors.

The five logging levels (and when to use them)

• DEBUG: noisy details only useful when actively debugging. Variable values, intermediate states.

• INFO: normal events you want a record of. Pipeline started, file loaded, model saved.

• WARNING: something unexpected, but not fatal. Missing values, fallback to default, slow query.

• ERROR: something failed. An exception, a request that didn’t go through, a metric that couldn’t be computed.

• CRITICAL: the system is broken. Database down, model file corrupted, pipeline can’t continue.

💡 If you only remember two levels, remember INFO (things I want a record of) and ERROR (things that broke). That covers most of what you’ll need.

How to log well

The mistake most beginners make is logging things that feel important but don’t help you debug.

Things like "got here" or "step 3". These tell you the code ran, but nothing else.

Useful log messages give you the state that would help you debug:

# Not useful
logger.info("Starting model training")

# Useful
logger.info(f"Starting model training with {len(X_train)} samples and {X_train.shape[1]} features")

The second one tells you, six months later, when something looks off, whether your training data was the size you expected.

A few more rules of thumb:

• Log the inputs and outputs at boundaries: When data enters or leaves your pipeline, log enough that you can reconstruct what happened.

• Log the metric values for things you care about: Latency, row counts, prediction distributions.

• Log exceptions with logger.exception() inside an except block. It captures the full traceback automatically.

Where logging fits in your day-to-day

A few concrete places this pays off:

• Training pipelines: log the dataset size, feature stats, hyperparameters, and final metrics. When a model performs worse than the last one, you can trace what changed.

• Inference endpoints: log request payloads, prediction outputs, and latency. When predictions start looking weird, you have a record.

• LLM-powered systems: log the prompts, token counts, and any validation failures (this is where logging plays nicely with Pydantic from last week).

Where logging fits in the bigger picture

One thing worth mentioning, since it’ll come up the moment you start working at scale: logging is one piece of a broader practice called observability.

Observability has three pillars:

• Logs are what we just covered, individual messages your code writes out as it runs.

• Metrics are numerical measurements over time (latency, error rate, prediction volume).

• Traces show the full path of a request through your system, especially useful when one call triggers many others (think LLM agents making multiple tool calls, or a request that touches three microservices).

For most data scientists getting started, logging is enough.

But once you’re working with multi-step LLM agents, distributed pipelines, or anything where one request fans out into many, tracing becomes the more useful tool.

Tools like MLflow (for ML experiments and LLM traces), OpenTelemetry (the underlying standard), and platforms like LangSmith or Arize are where you’d go next.

You don’t need any of that yet. Logging is the foundation. Just know it’s the entry point, not the destination.

Using AI to make logging cheap

Same pattern as with testing: setting up your project’s instruction file is what makes AI actually useful here.

A useful starter for the logging section of your CLAUDE.md:

## Logging
- Use Python's built-in `logging` module. Never use `print()` for anything beyond ad-hoc debugging.
- Logger setup goes in a central module (`logging_config.py`), not scattered through the codebase.
- Format: plain text in development, structured JSON in production.
- Use `logger.exception()` inside `except` blocks to capture full tracebacks.

## Logging rules
- Never log credentials, API keys, passwords, or PII. Redact before logging.
- Log enough state to debug, not so much that logs become noise.
- Use INFO for normal events, WARNING for unexpected-but-recoverable, ERROR for failures.

The PII rule matters more than it looks.

AI doesn’t always know what’s sensitive in your domain. Without an explicit rule, it will happily log full user objects, full request bodies, and full responses, including the parts you really don’t want sitting in a log aggregator.

A few prompts that work well:

• “Add logging to this function at appropriate levels. Log enough state that I could debug this in production. Don’t log any user PII.” Specifying the constraint matters.

• “Review these log statements. What’s missing that I’d actually want when debugging this in production?” Useful for tightening up existing code.

• “This function uses print() throughout. Replace each one with the appropriate logging call.” Quick win for legacy code you’re cleaning up.

💡 AI tends to generate logs that look impressive but don’t actually help you debug. Lots of logger.info("Starting X") and logger.info("Finished X") without any of the state you’d actually want when something goes wrong. When you review what it produces, ask yourself: if this code failed at 3am, would these logs help me figure out why? If not, push for more specific state.

Try it on your own

Reading about testing and logging gets you halfway there. The other half is feeling what it’s like to actually do it: watching a test catch a bug you introduced, seeing logs tell you exactly what your code was doing.

I put together a short Google Colab notebook with a few examples to work through. You’ll write your first pytest test, watch it catch a bug, and replace print() with proper logging.

Nothing heavy, maybe 15 minutes.

If you’d like a guided walkthrough, I also recorded a short video covering the same examples.

💡 Once you’ve done the notebook, pick one script you’ve already written and apply both habits to it. Replace one print() with a logger.info(). Refactor one function to be testable, and write one test for it. That’s the move that makes it stick.

Final thoughts

The reason these two habits are so underrated isn’t that they’re hard.

It’s that they don’t feel like the “interesting” parts of data science work. Testing isn’t modeling. Logging isn’t analysis. Neither one shows up in a portfolio.

But they’re the difference between code you can trust and code you can only hope works. And once you’re shipping things real people depend on, that difference becomes everything.

A couple of other great resources:

• 🎥 Want to follow along on YouTube? I just launched a channel for data scientists. Subscribe, first video drops soon.

• 🤝 Want to connect? Find me on LinkedIn, where I share more on data science careers and the AI shift.

Thank you for reading! And stay tuned for next week’s article in the Engineering Skills for Data Scientists series.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

👨‍💻 Welcome to the Engineering Skills for Data Scientists series, where I'll be teaching the engineering skills that quietly separate data scientists who can ship from the ones who can't. This is article 1 of 5. You can find the full series here.

You’ve probably seen this too…

Data scientists are shipping more code into production than ever before.

• On one side, the classic ML lane: pipelines, model serving, feature stores.

• On the other, the LLM lane: agents, structured outputs, tool calling, retrieval systems.

Both are growing fast, and more data scientists are expected to work in at least one of them.

Of all the challenges these two lanes share, one of the most underrated is the simplest: data showing up in shapes your code wasn’t ready for.

• A column that changed type upstream.

• An LLM that returned a field your code didn’t expect.

• A request payload missing a key.

None of these are rare failures. They’re the ones that quietly cause the most pain in production, and most data scientists don’t have a systematic way to handle them.

And to be fair, unless you come from a CS background, you probably never learned it, and you didn’t need to, until now.

Pydantic is the Python library that makes it cheap to learn, and learning this engineering habit will pay off in everything you ship from here on.

In this article, I’ll show you how to get started

Let’s get to it!

Here’s what we’ll cover

• Why “boundary problems” are the silent killers in production

• What Pydantic actually does

• Where Pydantic fits in LLMOps (the schema-driven approach to LLM outputs)

• Where Pydantic fits in MLOps (validating requests at your model endpoint)\

• 🎥 Try Pydantic on your own (a short video + Colab notebook)

• Why this skill compounds across everything you ship

Why your data validation falls short

Here’s a pattern that shows up in most production code written by data scientists:

Data comes in, gets passed through several transformations, and somewhere deep inside the pipeline (or worse, inside the model itself), something breaks.

• A missing key (KeyError).

• A value that came in as the wrong type (a number stored as a string, a float where you expected an integer).

• A silent data type conversion that doesn't crash but produces wrong numbers downstream.

The instinct most data scientists have is to add defensive checks: an assert here, a try/except there, a quick if "field" in data somewhere. It feels like good engineering, but honestly, it falls short.

The problem isn’t that you didn’t add enough checks. The problem is that the checks live in the wrong place. They’re scattered throughout the code, which means:

• You validate the same thing in multiple places (or worse, inconsistently)

• When something does fail, you find out three steps deep into your pipeline instead of at the entry point

• Bad data has already corrupted your state by the time you notice

The fix isn’t more checks. It’s moving validation to the boundary: the moment data enters your code, before anything else touches it. Define the shape you expect, validate against it once, and from that point on, you can trust what you’re working with.

That’s schema-driven design. And it applies whether the data is coming from an LLM, a feature store, an API request, or a row in a database.

Two half-measures that don’t actually solve the problem

If you’ve ever tried to do this, you’ve probably reached for one of two tools that look like they solve it. Neither actually does.

Plain dicts: no schema, no safety

# Whatever the LLM returns, you just hope it's right
ticket = {
    "title": "Login broken",
    "priority": "high",
    "user_id": 42
}

# Nothing stops this from happening:
ticket = {
    "titel": "Login broken",       # typo
    "priority": "super urgent",    # invalid value
    "user_id": "forty-two"         # wrong type
}
# Code downstream will break, silently or loudly.

You parse whatever comes in into a dictionary and hope it’s right. There’s no enforcement at all. A typo’d key, an invalid value, or a wrong type all flow through silently until something downstream breaks.

TypedDict: annotation only, no runtime check

from typing import TypedDict, Literal

class Ticket(TypedDict):
    title: str
    priority: Literal["low", "medium", "high"]
    user_id: int

# IDE/type-checker warns you. Python at runtime does not.
ticket: Ticket = {
    "title": "Login broken",
    "priority": "super urgent",   # type checker complains, runtime doesn't
    "user_id": "forty-two"        # same, runtime accepts it
}

TypedDict lets you describe the shape with type hints. It feels safer because your IDE will warn you about mistakes you make in code you wrote yourself. But when the code actually runs, Python doesn’t enforce it. If the data coming in doesn’t match the shape, TypedDict won’t catch it.

💡 What both of these miss is the part that matters: validation has to actually happen when the code runs, not just when you’re editing it. That’s the part that’s hard to get right by hand, and it’s what makes a real data validation library worth the small upfront cost.

What Pydantic actually is

Pydantic is the most popular data validation library in Python.

The idea is simple: you describe the shape of the data you're expecting, and Pydantic checks every piece of incoming data against that description. If something doesn't match, you get a clear error right away, before the bad data has a chance to break anything downstream.

You describe the shape using type hints, the same : str, : int, : float annotations you’ve probably already seen in Python code. Pydantic just makes those annotations enforced when your code actually runs, instead of being suggestions your IDE shows you.

Here's what that looks like:

from pydantic import BaseModel
from typing import Literal

class Ticket(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    user_id: int

That class is now a contract. Anything that claims to be a Ticket must have a string title, a priority that's one of three values, and an integer user_id. Pass in anything that doesn't match, and Pydantic raises an error pointing at exactly what's wrong.

You describe what good data looks like, and Pydantic tells you when something doesn't fit.

But now, let’s see how we leverage it across different use cases.

How to use Pydantic in LLMOps

Whenever you ask an LLM to return data your code is going to use, you’re stepping into the same boundary problem we just talked about.

Say you’re using an LLM to tag support tickets with a priority level. You want it to return something like:

{"title": "Login broken", "priority": "high", "user_id": 42}

Most of the time it will. But not always. Sometimes the model might:

• Drop a required field

• Add an extra one you didn’t ask for

• Return "42" (a string) where you expected 42 (a number)

• Pick a priority value you didn’t define (like "super urgent" instead of "low", "medium", or "high")

• Misspell a key ("titel" instead of "title")

In development, with a few test prompts, you might never see it happen. In production, with thousands of requests, you absolutely will.

This is where Pydantic earns its place. You reuse the same Ticket model from earlier and let it validate every LLM response before your code touches it:

from pydantic import ValidationError

raw = call_llm_to_tag_ticket(message)  # returns a dict

try:
    ticket = Ticket(**raw)
    # safe to use: ticket.title, ticket.priority, ticket.user_id
except ValidationError as e:
    # log it, retry the call, or fall back gracefully
    handle_bad_response(e)

The downstream code never sees malformed data.

Either it gets a valid Ticket object it can trust, or it gets a clear exception pointing at exactly what was wrong with the LLM's response.

The bonus: Pydantic also tells the LLM what shape to return

Here’s the part that earns Pydantic its place in LLMOps specifically.

The same class Ticket class you use to validate the response can be turned into a JSON Schema: a description of the shape, written in a format that LLM providers like OpenAI, Anthropic, and Google understand.

{
  "title": "Ticket",
  "type": "object",
  "properties": {
    "title": { "type": "string" },
    "priority": { "type": "string", "enum": ["low", "medium", "high"] },
    "user_id": { "type": "integer" }
  },
  "required": ["title", "priority", "user_id"]
}

If you’ve worked with OpenAI function calling, Anthropic tool use, or any provider’s structured outputs feature, you’ve already used JSON Schema, even if you didn’t call it that. It’s the format these APIs use to tell the model what shape to return.

You send that schema to the model along with your prompt, and the provider constrains the output to match it. So Pydantic ends up working on both sides:

• On the way out: it tells the LLM provider exactly what shape to enforce.

• On the way back: it validates the response before your code uses it.

One source of truth. Enforced from both directions.

💡 The pattern: anywhere an LLM’s output meets your code, define the expected shape explicitly, and validate against it. Don’t trust the model to follow instructions perfectly. It won’t.

How to use Pydantic in MLOps

The same habit applies on the other side of your work: classic ML in production.

The most common place data scientists hit this is at the model serving endpoint. You’ve trained a model. You wrap it in FastAPI (or similar). A client sends a prediction request, your endpoint passes the features to the model, and you return a prediction.

The naive version looks something like this:

@app.post("/predict")
def predict(request: dict):
    features = [request["age"], request["income"], request["tenure"]]
    prediction = model.predict([features])
    return {"prediction": prediction[0]}

This works until it doesn’t.

A client sends age as a string instead of a number. A field is missing. A new field is added that your model wasn’t trained on. Each of these will either crash deep inside your model code, or worse, produce a silently wrong prediction because Python coerced something it shouldn’t have.

With Pydantic, the contract sits at the endpoint:

from pydantic import BaseModel

class PredictionRequest(BaseModel):
    age: int
    income: float
    tenure: int

@app.post("/predict")
def predict(request: PredictionRequest):
    features = [request.age, request.income, request.tenure]
    prediction = model.predict([features])
    return {"prediction": prediction[0]}

Now, any request that doesn’t match the contract is rejected before your model ever sees it. The client gets a clear 422 Unprocessable Entity with the specific field that failed validation. Your model code stays clean because it only ever runs on validated inputs.

💡 By the way, FastAPI integrates with Pydantic natively, so this isn’t extra work. You’re already writing the function signature. Adding the type just turns the parameter into a validated contract.

The same pattern applies anywhere data enters your code:

• Pipeline step receiving features from upstream

• Configuration loaded from a YAML file

• Data read from a feature store before being passed to a model

• Request payloads at any API endpoint

Wherever there’s a boundary, there’s a contract worth enforcing.

Try Pydantic on your own

Reading about Pydantic gets you halfway there. The other half is feeling what it’s like when validation actually catches something for you.

I put together a short Google Colab notebook with a few examples to work through. You’ll see Pydantic validate good input, catch bad input, and generate a JSON Schema you could send to an LLM provider. Nothing heavy, maybe 15 minutes.

If you’d like a guided walkthrough, I also recorded a short video covering the same examples.

💡 Once you’ve done the notebook, pick one script you’ve already written that takes data from somewhere you don’t control (an API, an LLM, a CSV, a feature store) and add a BaseModel at the entry point.

That’s the real test to make sure if this lesson stuck.

Final thoughts

The reason Pydantic is worth learning isn’t that it's a popular library (though it has over 28K stars on GitHub). It’s because the habit it teaches you applies to everything you ship.

Once you start thinking in contracts, your code looks different:

• You stop writing scattered asserts and start defining shapes at function boundaries

• You stop debugging mysterious downstream failures and start seeing errors at the entry point where they belong

• You stop trusting “it worked in dev” and start designing for the failure modes you can’t predict

This is the engineering habit that quietly separates data scientists who can ship reliably from the ones who can’t. It’s not a glamorous skill. It doesn’t show up in interview questions. But it’s the difference between a model in production that silently fails for a week and one that fails loudly the first time something is off.

💡 If you’re earlier in your career: don’t worry about every Pydantic feature (there are many). Start with the one move: define a BaseModel for any data that crosses into your code from somewhere you don’t control. That single habit will compound over the rest of your career.

A couple of other great resources:

• 🎥 Want to follow along on YouTube? I just launched a channel for data scientists. Subscribe, first video drops soon.

• 🤝 Want to connect? Find me on LinkedIn, where I share more on data science careers and the AI shift.

Thank you for reading! And stay tuned for next week's article in the Engineering Skills for Data Scientists series: Testing & Logging.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

And for a good reason.

It’s where the field is moving, it’s where most of the demand is right now, and it’s the topic readers ask me about most.

But future-proofing your data science career takes more than AI skills.

The data scientists who actually ship things, who get their work out of the notebook and into production, share a foundation that nobody really talks about: engineering skills.

So starting Tuesday, I’m running a new free series on it.

What’s coming

5 articles, one per week, through mid-July. Each one focused on a specific skill that fills a real gap most data scientists have:

1. Pydantic, for data validation in MLOps and LLMOps

2. Testing and logging, so you actually know your code works

3. FastAPI, the cleanest way to turn a model into a service

4. Reproducible environments with uv, no more “works on my machine”

5. CI/CD for data scientists, automating the boring parts so your work actually ships

I’m keeping these practical. Concrete examples, code you can use, no shallow overviews.

The goal is that by the end of the series, you’ll have a real toolkit for shipping production code.

Get ahead this summer

Summer is usually the quietest stretch for data teams. Fewer urgent stakeholder requests, which means, finally, some room to actually learn something new.

So one article a week, little by little, through June and July.

Read it, try the techniques on a side project, and by the time Q3 picks up, you’ll have leveled up in an area most of your peers are still ignoring.

That’s the idea.

The first one drops on Tuesday, June 16.

A couple of other great resources:

• 🚀 Ready to take the next step? Get my free workshop on agentic analytics for data scientists.

• 🎥 Want to follow along on YouTube? I just launched a channel for data scientists. Subscribe, first video drops soon.

Thank you for reading! And hope you’re looking forward to this series as much as I am.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

Not building with it yet, just trying to figure out if they should.

They’ve seen it in job postings. They’ve read a few articles. They have a use case in mind at work. And the question they keep coming back to is some version of:

Is this actually a RAG problem, or am I reaching for the wrong thing?

It’s a fair question. And it’s harder to answer than it sounds, because RAG doesn’t sit alone. It shares space with two other patterns that show up in the same conversations and sound like they do the same job: MCP and semantic layers.

So here's the answer I've been giving in those conversations. By the end of this article, you'll know exactly when RAG is the right call, and when something else fits better.

Here’s what we’ll cover

• Why RAG is suddenly showing up everywhere

• What RAG actually is (skim if you already know)

• The four times a “RAG problem” isn’t one

• When RAG actually makes sense

• Where these patterns actually work together

• 📌 Resource: Building your first RAG system

Why RAG is having a moment

Every company I talk to has the same split.

Their structured data is handled. Dashboards, SQL, the warehouse, it’s all fine.

People know how to ask questions of it. But the moment a question depends on unstructured data: the PDFs, the support tickets, the internal docs, the Confluence pages, the meeting transcripts. There’s no good way to get at it; it just sits there.

That’s the gap RAG fills.

And it’s starting to show up more and more in the job market. After analysing 4K+ data scientist job postings, RAG appeared as the top-4 AI skill in demand.

It’s not the most common skill on the list yet, but a year ago, it was barely there, and it keeps gaining more demand.

So yes, RAG is worth learning.

But that’s the easy part. The harder part is knowing when to use it.

What RAG actually is

Skim if you already know.

RAG (Retrieval-Augmented Generation) is a pattern where you retrieve relevant content from your own data and pass it into a model so the answer comes from that, not from the model’s training memory.

Three phases:

• Ingestion (done once, ahead of time): your documents get parsed, chunked, embedded, and stored.

• Retrieval (done per query): the user’s question gets embedded, and you search the store for similar chunks.

• Generation: the model gets the question plus the retrieved chunks, and produces an answer grounded in them.

The key shift: SQL retrieves by literal match. RAG retrieves by meaning. That’s what makes it useful for text.

Why “should I use RAG?” is a hard question

The reason this question trips people up isn’t RAG itself. It’s that RAG isn’t the only thing in the room.

Two other patterns show up in the same conversations and sound like they do the same job:

• MCP (Model Context Protocol): a standard for connecting models to existing systems (Notion, Confluence, Slack, Google Drive, internal APIs). Retrieval is involved, but you don’t build it. You inherit it from whoever wrote the MCP server.

• Semantic layers: a modeling layer over your structured data that defines business logic, metrics, and dimensions in one place. Not retrieval in the same sense as RAG, but it's how AI systems (and BI tools, and chatbots) get reliable access to your metrics.

These aren’t apples-to-apples with RAG.

MCP is a delivery mechanism. A semantic layer is a modeling layer. RAG is a retrieval pattern. But they all show up the moment someone says “I want AI on top of my data”, and from the outside, they sound interchangeable.

So before reaching for RAG, it’s worth asking: am I sure this is what I need?

Here's how I think about it: four cases where something else fits better, and one where RAG is the right call.

Case 1: “Can’t I just dump everything into the prompt now?”

What you actually need: in most cases, still retrieval.

This is the argument you’ve probably seen on X. Context windows used to be small, so you had to retrieve the relevant chunks before sending them to the model. Now that windows are at 1M+ tokens, why retrieve at all? Just load the whole knowledge base into the prompt every time.

For a small, slow-changing corpus, sometimes yes. If your “knowledge base” is your team’s 8 metric definitions and 3 architecture decision docs, a cached system prompt covers it. No vector store, no chunking, no eval loop.

But for most real systems, no. A few reasons:

• Cost scales with tokens loaded. Even with caching, you pay to load context. Stuffing 200K tokens into every query gets expensive fast at scale.

• “Lost in the middle” is real. Models read the start and end of long contexts well, and the middle much worse. A 1M context window doesn’t give you 1M tokens of equally useful attention.

• Latency. Long-context queries are noticeably slower than retrieval-style ones.

• Freshness. A cached context doesn’t update on its own. Anything that changes often needs to be reloaded.

The honest framing: bigger context windows didn’t kill RAG, they raised the floor for when you need it. If your corpus is small and stable, skip retrieval. If it’s large, fast-changing, or queried at scale, you still need RAG.

Case 2: The answer lives on the public web

What you actually need: a web search tool.

You don’t want to be in the business of indexing the internet, and you don’t need to be.

If you’re asking “how does this new Python library handle async?” or “what did the latest BLS report say about wage growth?”, you want web search, not RAG.

I know this might seem basic to some of you, but it's worth clarifying.

When people hear "extend the LLM's knowledge base," they often jump straight to RAG. But if the knowledge they need is public, the tooling already exists. If you're building, there's Tavily and similar APIs. If you're using existing AI tools (Claude, ChatGPT, Perplexity), web search is already baked in.

Case 3: The question is really a SQL question

What you actually need: a semantic layer.

This is the one where I see the most confusion.

If the question is “how many sign-ups last quarter?” or “average order value by region”, that’s SQL, not RAG. RAG can’t compute averages over your sales table. It can find a paragraph that mentions sign-ups, but it can’t sum a column.

If your stakeholders want natural-language access to those numbers, what you actually want is a semantic layer. Something like Cube, dbt’s semantic layer, or LookML. It defines your metrics once and lets an AI system (or a BI tool, or a chatbot) translate questions into the right query against your data.

A RAG pipeline pointed at a data warehouse export will happily return a number that looks plausible and is actually wrong. That number doesn’t stay in your notebook, it ends up in a deck.

Case 4: Someone already built the retrieval for you

What you actually need: an MCP server.

If your knowledge lives in Notion, Confluence, Linear, or a similar system, check whether there’s already an MCP server for it before you build anything. If there is, you’ll ship grounded answers faster than you can spec out a chunking strategy.

Imagine you’re in Claude Code working on an analysis, and you need to understand the business context around a feature your team shipped last quarter. The PRD lives in Confluence. With the Confluence MCP server connected, Claude can pull the PRD directly during the analysis. You didn’t build a RAG pipeline. You didn’t chunk anything. The retrieval happens for you.

You do inherit whatever retrieval choices the server makes (chunking, ranking, filtering). For some use cases, you’ll outgrow that and need your own pipeline. But starting with MCP is almost always the cheaper first move.

The case where RAG actually fits

A body of text the model needs to read before it can answer well, where:

• The knowledge is yours (not on the public web)

• It’s too large or inefficient for the prompt

• The question is about meaning, not computation

• No off-the-shelf MCP server covers it well enough

Think of an assistant that answers questions from your team’s internal documentation: runbooks, architecture decisions, post-mortems, metric definitions written up in long-form.

A new hire asks, “why did we move from BigQuery to Snowflake?” and the system surfaces the right architecture decision doc.

That’s a real RAG case. The knowledge is yours, it’s text-heavy, it’s too large for a prompt, and the question is about meaning, not a number.

The first question is never “should I use RAG?” It’s “what kind of question am I answering?”

The quick-reference version

If you only remember one thing from this article, remember this table.

Four out of five times, RAG isn't the answer. That's the part most people miss.

Where these patterns actually work together

One more thing…

These patterns aren’t competing. The best AI systems use all three.

Imagine a “talk-to-your-data” Slackbot at a real company:

• A semantic layer lets the system answer “how many enterprise sign-ups last quarter?” by translating to SQL against a defined metric.

• An MCP server lets it pull the latest ticket from Linear or the latest doc edit from Notion.

• A RAG pipeline lets it surface the right paragraph from your internal playbook when someone asks “what’s our refund policy for annual plans?”

Same system. Three retrieval patterns. Each owns the questions it’s actually good at.

The skill that’s getting harder to find isn’t building any one of these. It’s knowing which pattern owns which question, and then composing them into a system that picks the right tool for the job.

Want to build your first RAG system?

RAG is showing up in more data scientist job postings every quarter for a reason.

The unstructured side of every business is huge, and once you can build retrieval over it, you become the person on the team who handles the half of the data that everyone else can't.

This is why I put together a RAG crash course for data professionals, where you'll build your first RAG system end-to-end, including evaluation.

And I'm giving it away free to anyone who enrolls in the upcoming cohort of my AI Workflows Bootcamp, which starts June 12.

You don't want to miss this. The bootcamp teaches you to leverage AI day-to-day and build systems with real design judgment. The RAG course adds the unstructured side. Together, that's the toolkit most data scientists are still missing.

Enroll today

A couple of other great resources:

• 🚀 Ready to take the next step? Get my free workshop on agentic analytics for data scientists.

• 🎥 Want to follow along on YouTube? I just launched a channel for data scientists. Subscribe, first video drops soon.

Thank you for reading! I hope you find this guide useful.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

If you’ve worked on a data team for more than a few months, you’ve probably seen an analysis ship with a review that amounts to roughly that (maybe paired with a thumbs-up emoji)

And honestly, I get it.

When stakeholders need answers fast, peer review is the first thing that gets cut.

And once the analysis is out the door, there’s no real motivation to revisit it. The dashboard is live, the deck got presented, and the decision got made. Going back to audit the methodology feels more like a waste of time.

I’ve watched this happen across every data team I’ve worked on. The part that gets reviewed is the code. The part that almost never gets reviewed is the analysis itself.

And those are two very different things.

Code review checks if your SQL runs. It doesn’t check if your SQL is answering the right question. It doesn’t catch a join that silently inflates revenue, an average that’s hiding a bimodal distribution, or a framing that’s going to lead your stakeholder straight to the wrong conclusion.

That’s a methodology review. And almost no one does it.

So I built a Claude Skill to act as my peer reviewer. And in this article, I’m going to walk you through how to build your own, step by step, even if you are completely new to Claude Skills.

Here’s what we’ll cover

• A quick primer on Claude Skills (skip ahead if you already know)

• A step-by-step build of an analytical peer reviewer Skill

• An example of the skill reviewing a real analysis

• The full version of the Skill at the end

A quick primer on Claude Skills

A Skill is a markdown file that tells Claude how to approach a specific, repeatable task, different from a CLAUDE.md or AGENTS.md, which set general conventions and rules for a project. That's it. The rest is just where the file lives and how Claude finds it.

Where does the file go?

You can place a Skill in two places:

• Project level: lives inside the project you’re working on. Only available when you’re in that project.

• Global level: lives in your user directory. Available across every project and every tool that reads Skills (Claude Code, Cursor, etc.).

For an analytical peer reviewer, global makes the most sense. You want to be able to invoke it on any analysis, in any repo.

How does a skill file get picked up?

The file has a metadata block at the top (we’ll get to it in Step 1). Claude reads that metadata, and if your task matches the Skill’s description, it loads the rest of the file and follows the instructions inside.

You don’t have to manually trigger it (although you have the option). If the description is written well, Claude figures out when to use it.

How does a skill get used?

Once invoked, the Skill is basically a prompt: a set of instructions Claude executes. You wrote it once, refined it, and now it runs the same way every time. That’s the real value, consistency.

Skills can also include scripts (Python, bash, whatever) that Claude can run as part of executing the instructions. We won’t need that for this build, but it’s worth knowing the option is there.

What we’re building

Before we build, here’s what the Skill does:

You give it a file (a SQL query, a notebook, or a Python script, whatever the analysis lives in)

1. It scans the file, the surrounding context, and asks you a few questions to understand what the analysis is trying to accomplish.

2. It pressure-tests the work in a systematic way

3. It gives you back a structured review: a verdict, the things worth looking at, and a question to sit with.

Once you have saved it either into your local or global skills directory, here is what it would look like to trigger it manually:

Alright, let’s build it.

Step 1: The setup (metadata)

Every Skill starts with a metadata block at the top of the SKILL.md file. This is what Claude reads to decide whether the Skill applies to whatever you’re working on.

---
name: analyst-peer-review
description: Analytical peer review that challenges the logic, assumptions,
and business reasoning behind SQL, Python, or other analytical code.
Use when you want your analysis stress-tested before sharing results or making decisions.
Not a linter — a devil's advocate for your analysis.
allowed-tools: Read Grep Glob AskUserQuestion
argument-hint: <file_path> [file_path_2 ...]
---

Three things to pay attention to:

• name is the identifier. Keep it short and descriptive.

• description is the most important field. This is what Claude uses to decide when to invoke the Skill. Be precise about what it does and (just as importantly) what it isn’t. Notice I explicitly call out “not a linter”, that’s there to keep Claude from misfiring on generic code review requests.

• allowed-tools limits what the Skill can do. This Skill only needs to read files and ask questions, so I’ve restricted it to read-only tools. No file writes, no bash, no edits.

💡 Limiting tools is a small thing that matters. A peer reviewer shouldn’t be modifying your code, it should be challenging it. Constraining the Skill to read-only enforces that boundary at the tool level, not just in the instructions.

Step 2: Gathering context

Here’s where most automated review tools fail: they jump straight to critique without understanding intent. A peer reviewer who doesn’t know what you’re trying to accomplish is just a more opinionated version of a linter.

So the first thing the Skill does is gather context.

## Phase 1: Gather Context

Before reviewing anything, you need to understand what the analyst is trying to accomplish. Do not skip this step.

1. Read the file(s) passed as arguments.
2. Check for project context: read the CLAUDE.md or README at the project root.
3. Ask the analyst (skip questions you can confidently answer from the code, but state your understanding and ask them to confirm):

   - What question is this code trying to answer? (The business question, not "it calculates X")
   - Who will consume this output and what decisions will they make based on it?
   - Any constraints, assumptions, or known edge cases I should know about?

Wait for answers before proceeding.

Three questions, and they’re not arbitrary. Each one maps to a specific failure mode I’ve seen over and over:

• “What question is this trying to answer?” catches the mismatch between the metric being computed and the question the stakeholder actually asked. You’d be surprised how often these don’t line up.

• “Who will consume this and what decisions will they make?” anchors the review in the audience. A metric that’s correct for an engineering team can be deeply misleading for an exec.

• “Any constraints or edge cases?” lets the analyst surface known limitations upfront, so the Skill doesn’t waste energy on issues you’ve already accounted for.

💡 Notice the instruction to skip questions you can confidently answer from the code. Without this, the Skill becomes annoying, asking obvious questions when the answer is right there in the file. With it, it only asks what it actually needs to know.

📌 Get my free workshop on agentic analytics in production

Step 3: The pressure test

This is the heart of the Skill. The part that does the actual analytical work.

## Phase 2: Pressure Test

With the analyst's stated question and audience in mind, work through the following:

Are the numbers correct?
- Joins: could any join fan out or silently drop rows?
- Grain: is the metric computed at the right granularity before being rolled up?
- Aggregation: are ratios using the correct numerator and denominator? How do NULLs behave?
- Temporal boundaries: off-by-one in date ranges? Incomplete periods at edges?
- Filters: could WHERE clauses silently exclude a population that matters?

Is this answering the right question?
- Does the output actually address what the decision-maker needs?
- Could the metric subtly answer a different question than intended?
- Are business terms defined the way the audience defines them, or just the way the code defines them?

Will the audience interpret this correctly?
- What's the most likely misread?
- Is important context missing that the audience would need to draw the right conclusion?
- Could someone use this output to justify a decision the data doesn't actually support?

What's not in the frame?
- Are there confounding factors that could explain the result equally well?
- Is the analysis looking at survivors only, missing what left?
- Are there segments or edge cases excluded that could change the story?

Four categories, and they’re deliberately ordered.

The first one (are the numbers correct?) is the cheapest to get wrong and the most embarrassing when you do. The other three are where the real value lives. A join that silently fans out is a bug. An analysis that answers the wrong question is a strategic miss.

The principle that anchors this whole section: Most code is fine. Most analyses have a blind spot.

That’s why this Skill spends more energy on framing and interpretation than on mechanics. Tools like dbt tests and CI checks already cover a lot of the “is the number right” question. Almost nothing covers “Does the number mean what you think it means?”

Step 4: Delivering the review

The format of the output matters as much as the content. Two failure modes to design against:

• A review that lists 10 findings of equal weight is a review that highlights nothing.

• A review without a clear bottom line forces the reader to do the synthesizing themselves.

So the output has three parts:

## Phase 3: Deliver the Review

Structure your output as follows:

### Bottom Line
Lead with a verdict, one of:
- Ship it: code is correct and the analysis holds up
- Think on it: numbers are right, but the framing or interpretation deserves another look
- Fix first: there's a correctness issue that must be resolved

Follow with 2-3 sentences explaining why.

### What I'd Look At
Maximum 3 findings. Only include findings that could change the conclusion or mislead a decision-maker. For each:
- The issue in one sentence
- Where (file, line)
- Why it matters

If you found fewer than 3 real issues, list fewer. Do not pad.

### One Question to Sit With
A single analytical question. Not a code fix, not a suggestion. A question about whether the output means what they think it means.

A few design choices worth calling out:

• The verdict comes first. If someone reads nothing else, they should know what to do. Ship, think, or fix.

• Maximum 3 findings. This is the most important constraint in the whole Skill. Without it, Claude will happily list every minor observation it noticed. Forcing selectivity is what makes the review useful instead of overwhelming.

• One question at the end. This is the part that mimics what a thoughtful colleague would actually do. They don’t always give you an answer. Sometimes they just ask the question that makes you pause.

💡 The “one question to sit with” is my favorite part. It’s the move a good senior reviewer pulls: they don’t tell you what to fix, they ask you something that makes you realize what to fix.

Trying it out

I ran the Skill on a churn analysis: a notebook looking at monthly customer churn rates over the past six months, with a recommendation to the CRO and board on whether to launch a retention program this quarter.

The Skill started by reading the notebook and the README, then asked me one targeted question about whether including May (a partial month, since the data was pulled mid-month) was intentional or an oversight:

After I confirmed it was intentional, but I hadn't fully accounted for May being half a month, the Skill ran the pressure test and returned a "Fix first" verdict:

Here’s what the Skill flagged. A few things stood out as clear blind spots in the analysis:

• It caught a partial month being treated as a full one. The most recent month in the trend only had half its data in, but the analysis was reading it like any other point. This is the kind of thing a stakeholder would’ve flagged immediately.

• It caught a denominator that didn’t match what the metric was named. A metric labeled “logo churn” was being computed on subscription rows, not accounts, which silently inflated the active base in months with upgrade activity. The number going to the board wasn’t measuring what it said it was.

• It caught a shrinking base hiding behind a rate. Churn rate looked flat, but the underlying customer count had contracted 13% in five months. Fewer churn events partly meant fewer customers left to churn, which is a very different story than “churn is under control.”

Any one of these would have been enough to walk back the recommendation. All three in the same analysis is the kind of thing that, in the real world, quietly makes it to a board deck without anyone noticing.

And then there was the closing question: “If you drop May entirely and look only at complete months, does anything in the data still say ‘this is resolving on its own’, or does the entire ‘hold off’ recommendation depend on the one month you can’t actually measure yet?”

That’s the question that would have stopped me before I sent the deck.

Wrapping up

A Skill like this codifies a kind of thinking that usually only exists in senior heads, and makes it available on demand.

That’s the real unlock. AI doesn’t replace judgment. It lets you scale yours.

The version we built in this article is intentionally simplified. The full version (the one I use in practice), with a more thorough pressure test phase and a few extra principles, is here:

---
name: analyst-peer-review
description: Analytical peer review that challenges the logic, assumptions, and business reasoning behind SQL, Python, or other analytical code. Use when you want your analysis stress-tested before sharing results or making decisions. Not a linter — a devil's advocate for your analysis.
allowed-tools: Read Grep Glob AskUserQuestion
argument-hint: <file_path> [file_path_2 ...]
---

You are an analytical peer reviewer. Your job is to pressure-test analytical code — first verifying the numbers are correct, then challenging whether the analysis actually serves its purpose.

Think of yourself as a skeptical but constructive colleague who asks: **"Are you sure this actually answers the question you think it does?"**

## What this is NOT

- Not a linter or syntax checker
- Not a code quality review (style, formatting, best practices)
- Not about performance optimization
- Do not suggest adding comments, docstrings, error handling, or type hints
- Do not suggest code changes unless a logic flaw requires it

## Phase 1: Gather Context

Before reviewing anything, you need to understand what the analyst is trying to accomplish. **Do not skip this step.**

1. **Read the file(s)** passed as arguments. If no arguments are provided, ask which files to review.
2. **Check for project context**: read the CLAUDE.md or README at the project root. This tells you the domain, data model, and conventions.
3. **Ask the analyst** (adapt based on what you already know — skip questions you can confidently answer from code and project context, but state your understanding and ask the analyst to confirm):

   - **What question is this code trying to answer?** (The business question, not "it calculates X")
   - **Who will consume this output and what decisions will they make based on it?**
   - **Any constraints, assumptions, or known edge cases I should know about?**

Wait for answers before proceeding. The review is only as good as your understanding of intent.

## Phase 2: Map the Logic Chain

Before critiquing, understand. Trace the full data flow:

**Source** -> **Filters/Scope** -> **Joins/Enrichment** -> **Transformations** -> **Aggregation** -> **Output** -> **Interpretation**

Write this chain out in plain language. This is your map for the review — and it helps the analyst see their own logic laid bare.

## Phase 3: Pressure Test (Two Layers)

Work through two layers in order. **Layer 1 gates Layer 2**: if the code produces wrong numbers, that's the story — don't move on to analytical framing until the mechanics are sound.

### Layer 1 — Code Mechanics (are the numbers correct?)

This is a quick pass. Scan for issues that would make the output wrong:

- **Joins**: Could any join fan out (create duplicates) or silently drop rows? Are join keys unique on the side that needs to be?
- **Grain**: Is the metric computed at the right granularity before being rolled up? Could a GROUP BY collapse rows that shouldn't be combined?
- **Aggregation**: Are ratios using the correct numerator and denominator? How do NULLs behave? Is DISTINCT masking an upstream duplication problem?
- **Temporal boundaries**: Off-by-one in date ranges? Incomplete periods at edges? Look-ahead bias?
- **Filters**: Could WHERE clauses or join conditions silently exclude a population segment that matters?

**If you find a logic flaw here — something that makes the numbers wrong — that becomes your bottom line. Stop and report it.** The analytical framing doesn't matter if the data is broken.

**If the code mechanics are sound, say so briefly and move to Layer 2.** This is where most reviews should spend their energy.

### Layer 2 — Analytical Reasoning (does this analysis serve its purpose?)

This is the main event. With the analyst's stated question and audience in mind:

**Is this answering the right question?**
- Does the output actually address what the decision-maker needs to know?
- Could the metric or framing subtly answer a different question than intended? (e.g., showing a rate when the audience needs a volume; showing an average when the distribution matters)
- Are key business terms defined the way the audience defines them — or the way the code defines them?

**Will the audience interpret this correctly?**
- What's the most likely misread? (e.g., stable rate interpreted as "things are fine" when the underlying population is shifting)
- Is important context missing from the output that the audience would need to draw the right conclusion?
- Could someone use this output to justify a decision the data doesn't actually support?

**What's not in the frame?**
- Are there confounding factors that could explain the result equally well?
- Is the analysis looking at survivors only, missing the ones that left?
- Are there segments, time periods, or edge cases excluded that could change the story?
- What would this analysis miss if conditions changed? (e.g., seasonality, a product launch, a policy change)

**Does the "so what" hold up?**
- If this number moves 10%, does anyone do anything differently? If not, is this the right metric?
- Could an alternative framing of the same data tell a more useful or more honest story?

## Phase 4: Deliver the Review

**Be selective, not comprehensive.** Your job is to surface the 1-3 things that actually matter — not to list everything you noticed. A review that highlights 6 findings with equal weight is a review that highlights nothing.

Structure your output as follows:

---

### Bottom Line
Lead with a **verdict** — one of three words that tells the analyst what to do before they read anything else:

- **Ship it** — code is correct and the analysis holds up
- **Think on it** — numbers are right, but the framing or interpretation deserves another look
- **Fix first** — there's a correctness issue that must be resolved

Follow the verdict with 2-3 sentences explaining why. This is what someone reads if they read nothing else.

Example: **Verdict: Think on it** — The numbers check out, but the output shows a per-user average that could mask a bimodal distribution. If the audience assumes a normal spread, they'll draw the wrong conclusion.

### What I'd Look At
**Maximum 3 findings.** Only include findings that could change the conclusion, mislead a decision-maker, or silently produce wrong results. Tag each one:

- **Logic flaw** — the code produces or could produce incorrect results
- **Assumption risk** — correct IF a fragile or unverified assumption holds
- **Blind spot** — something unaccounted for that could change the conclusion
- **Framing gap** — the output is technically correct but could mislead the audience

For each finding, keep it tight:
1. The issue in one sentence
2. Where (file, line)
3. Why it matters — what goes wrong and for whom

If you found fewer than 3 real issues, list fewer. Do not pad.

### What's Solid
One short paragraph. Call out the parts of the logic and the analytical choices the analyst can trust and stop worrying about. Not a bulleted inventory — just the key things that hold up.

### One Question to Sit With
A single analytical question — the kind that makes the analyst pause. Not a code fix. Not a suggestion. A question about whether the output means what they think it means, or whether the audience will read it the way they intend.

---

## Principles

- **Selectivity over completeness.** Finding everything is easy. Knowing what matters is the job. If a finding wouldn't change a decision, leave it out.
- **Layer 1 gates Layer 2.** If the numbers are wrong, that's the review. Don't critique the framing of broken data.
- **Most code is fine. Most analyses have a blind spot.** Expect to spend more time on Layer 2 than Layer 1. The valuable insight is rarely "your join is wrong" — it's "your audience will misread this."
- **Be specific.** "This join might fan out" is useless. "The join on line 34 between orders and refunds could produce duplicates because an order can have multiple partial refunds — this would inflate the revenue total on line 52" is useful.
- **Challenge the logic, not the person.**
- **It's okay to say "this looks solid."** Don't manufacture issues to seem thorough.
- **If you're uncertain, say so.** Frame it as a question, not a finding.
- **No scope creep.** Review what was asked. Don't redesign the approach unless it's fundamentally flawed.

Copy it, modify it, make it yours. The whole point of writing a Skill is that it reflects how you think. Mine is a starting point, not a template to copy.

💡 If you’re earlier in your career: this Skill is also a pretty good cheat sheet for how a senior reviewer thinks about your work. Try running it on your own analyses before you share them. The questions it asks are the questions you’ll eventually start asking yourself automatically.

A couple of other great resources:

• 🚀 Ready to take the next step? Get my free workshop on agentic analytics for data scientists.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you find this guide useful.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

I’ve stopped reading them.

Most feel like they were written by someone who spent 20 minutes with each tool. No real workflow behind it. No honest take on what actually stuck.

So here’s mine. These are 5 tools that have genuinely become part of how I work, not tools I’ve tried once or revisit occasionally, but ones I reach for almost every day.

There are other tools in my stack, but these are the ones that have earned a permanent spot. Some I pay for out of my own pocket. Some are covered by my employer.

Let’s get to it…

1. Wispr Flow (personal)

It’s a voice-to-text tool. But calling it that undersells it.

I think much faster when I’m speaking than when I’m typing. So Wispr Flow essentially removed the bottleneck between my brain and everything else in my stack. I use it to brainstorm out loud, give instructions to Claude, write Slack messages, and dump half-formed thoughts I clean up later.

That last one is something I do constantly: speak a messy brain dump into Wispr, hand it to Claude to organize, then use Claude to expand it into documentation, a deeper investigation, or a full draft. It’s become a core part of how I go from idea to output.

Because it sits at the input layer of everything I do, it speeds up every other tool on this list.

If you type everything and feel like your hands can’t keep up with your brain, this is worth trying.

Claude (personal Max, work standard)

Claude is the center of my stack.

It’s how I brainstorm, write and debug code, think through problems, and draft communications. If Wispr Flow is the input layer, Claude is the brain.

My work covers Claude at a Pro tier. I personally pay for Max, not for my day job, but for everything else I build and experiment with outside of it. The difference matters for longer context, heavier reasoning tasks, and Claude Code, which handles end-to-end implementation, catches its own mistakes, and reasons through complex multi-step tasks in a way that genuinely changes what’s possible in a single working session.

Windsurf (personal)

My work covers Cursor. I’ve used it and it’s good.

But I still pay for Windsurf out of pocket. I know that on paper it probably looks redundant: I already have Claude Code. I could just use VS Code and call it a day.

The honest answer is that Windsurf has features I genuinely enjoy using, and the IDE experience feels complete in a way that keeps me coming back. The agentic features are accurate and don’t require constant babysitting. It’s become a reliable backup that I reach for on personal projects, and occasionally I’ll use it alongside Claude in a sort of adversarial mode, where one checks the other’s work. Two different models, two different reasoning paths, one output that’s been pressure-tested.

Is it strictly necessary? Probably not. But it earns its place.

It’s my go-to setup for personal projects, weekend experiments, and anything I’m building outside of work.

“Take Notes for Me”, Google Meet (work)

This one is easy to overlook because it’s baked into Google Meet, not a standalone product.

But “Take Notes for Me” has quietly become one of the most useful things in my workflow.

I work closely with stakeholders. Design discussions, alignment calls, requirements sessions. A lot of important context lives in those conversations. I used to take manual notes and still miss things.

Now I let Gemini handle it, and after the call I have a structured summary, action items, and a full transcript I can actually work with. More importantly, I can feed that directly as context into Claude and other tools. The discussion becomes usable input, not a memory I’m trying to reconstruct.

Not flashy. But it quietly makes everything else work better.

📚 NotebookLM

NotebookLM isn’t in my stack every single day, but it shows up often enough, and for a specific enough reason, that it belongs here.

When I’m doing research (reading papers, going deep on a topic) I don’t fully trust any single source, including Claude. Claude is excellent, but it can hallucinate, especially on niche or recent material.

NotebookLM lets me feed in the actual papers and articles I’m working with and get explanations and outlines grounded in those sources. It’s my verification layer. When I need to trust the answer, not just get one, this is where I go.

How it all fits together

Listing tools makes them sound more isolated than they are. In practice, this is one system.

• Wispr Flow handles the input, speaking is faster than typing, so it removes friction at the start of almost every task.

• Claude is where the real thinking and overhaul happens.

• Windsurf gives me an IDE I enjoy working in, with Claude as a pressure-test on its output.

• Google Workspace transcription captures stakeholder context that would otherwise get lost and feeds it downstream.

• NotebookLM closes the loop when I need sourced answers rather than fast ones.

Each tool has a job. Together, they cover the full arc from raw idea to finished output.

A couple of other great resources:

• 🚀 Ready to take the next step? Get my free workshop on agentic analytics for data scientists.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you find this dashboard useful.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

That question has been driving a lot of my research over the past few months. I’ve been scraping thousands of U.S. data science job postings, analyzing the data, and writing about the findings here.

But at some point, I realized something: the dashboard I built for the analysis was more useful than anything I was writing about it.

So I decided to open source it.

What the dashboard covers

You can explore:

• The top skills companies are actually hiring for, ranked by frequency

• Skills broken down by category (ML/DL, GenAI, Cloud, MLOps, etc.)

• What percentage of roles now require AI skills, broken down by seniority level

• The salary premium for candidates with AI skills

• An interactive explorer where you can browse individual postings with matched skills highlighted

The skill extraction is built on around 230 curated keyword groups, so it’s pretty granular.

Explore the dashboard

Why open source?

I’ve always believed that the best way to help people navigate this market is to show them the data, not just tell them what I think about it.

The code, the data, and the full methodology are all available in this GitHub repo. You can fork it, run your own analysis, or build on top of it.

And by the way, I’m scraping weekly, and I’ll soon be uploading all of the raw data to Kaggle as well. For now, everything is in the repo.

What’s next?

I plan to keep updating this throughout the year and revisit the analysis before the summer. Just like the one I did early this year:

If you find it useful, give the repo a star and share the dashboard with someone who might benefit from it. That’s the best way to say thank you and support this work.

Oh and of course, if you spot anything off or have ideas for what to add, I’d love to hear from you.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you find this dashboard useful.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

I’ve been hearing this a lot from data scientists lately. People who are genuinely good at what they do, but feel overwhelmed by how fast everything is changing.

Not sure where to focus. Not sure if what they’re learning even matters.

And honestly, it wasn’t that long ago that I felt that way too. Until I started approaching my learning a lot more strategically.

You see, for the past year, I’ve been obsessed with answering one question: What does it really take to future-proof my data science career? Not just in the context of AI, but in terms of how the field is evolving over the next few years.

That shift in focus has genuinely transformed my career, and for the first time ever, I feel like I’m ahead of the curve, not chasing it.

Today, I’m actively championing AI adoption at my company. I’ve been able to amplify my output at work in ways that have led to promotion discussions much sooner than ever before. I’m a stronger candidate in the market despite the competition. I’ve even been invited to give talks on agentic analytics.

But I didn’t get here by learning passively.

I got here by building, by studying the market, and by figuring out what actually matters. I’ve analyzed thousands of job postings to understand what companies really want from data scientists right now, and the picture is much clearer than the noise on social media makes it seem.

This article is my attempt to lay it all out in one place for anyone who needs it.

A practical, no-BS guide to future-proofing your data science career in 2026-2027, covering the skills, the strategy, and the mindset shifts that are actually making a difference right now.

Here’s what we’ll cover:

1. What the job market actually looks like right now (based on thousands of job postings analyzed)

2. The three pillars every data scientist should be investing in

3. Agentic analytics: what it is and why you should (really) care

4. Should you transition to AI engineering? (My honest advice)

5. A practical learning roadmap to help you get started

6. The most effective way to future-proof your career for AI 👈

You’ll definitely want to read this one until the end!

What’s actually happening in the job market

Let’s start with the data, because there’s a lot of noise out there and most of it falls into either “data science is dead” panic or “it’s all just hype” copium.

Neither is accurate.

Earlier this year, I scraped and analyzed 700+ Data Scientist job postings in the US spanning November 2025 through January 2026. I also drew on findings from a Harvard study that tracked nearly 285,000 companies and a report by Lightcast on generative AI hiring trends. Here’s what the picture actually looks like:

The fundamentals haven’t changed, but AI is now expected

Python, SQL, and machine learning still dominate job postings. No surprise there. But AI is now referenced in almost 60% of data science postings, and LLMs have entered the top 10 most requested skills compared to the previous year.

AI isn’t a niche specialization for data scientists anymore. It’s becoming part of the baseline.

Senior roles are growing. Entry-level is tightening.

The Harvard study showed something important: when companies adopt generative AI, junior hiring slows down while senior roles keep growing. Companies aren’t mass-firing juniors, but they’re hiring fewer of them. The routine work that used to justify those headcounts (cleaning data, writing basic code, drafting reports) is now handled more efficiently with AI.

💡 The door into many companies is narrower, not closed. And for those already inside, opportunities haven’t disappeared. In fact, internal promotions for juniors actually went up in companies adopting AI.

AI-heavy roles skew mid-to-senior

When I looked at which seniority levels are being asked for AI skills, the pattern was clear: 73% of AI-focused postings target mid (43%) and senior (30%) data scientists. Entry-level accounted for less than 6%.

This reinforces what I’ve been saying for a while now: companies mainly expect experienced practitioners to be the ones who translate AI capabilities into real systems and business outcomes.

There’s an emerging salary premium

Roles that explicitly mention AI tend to show higher median salaries across most seniority levels. The signal is still early (salary data was only available for about a third of postings), but it’s directionally clear: AI skills are starting to pay more.

💡 If you want the full breakdown, including which specific AI skills are most in demand, I published the complete job market report here.

The three pillars of a future-proof data science career

Now that we’ve seen what the market is asking for, let’s talk about how to respond.

What I’m about to share are the three areas I believe every data scientist should be investing in right now, regardless of seniority (although how you prioritize each does depend largely on your goals and your seniority).

But I want to be clear about why these three specifically.

The goal isn’t just to “keep up with AI.” The goal is to make yourself full-stack enough to navigate the market in almost any direction. Whether you want to stay in a data science role, move into AI engineering, step into leadership, or even build something on your own, these three pillars give you the foundation to choose.

And that’s the real future-proofing, in my opinion. Not clinging to a job title, but building the kind of skills that give you options no matter how the field evolves.

Pillar 1: AI-Powered Data Scientist

This is about making AI a core part of how you work, not just a side tool you occasionally prompt.

Right now, most data scientists are using AI for ad-hoc tasks: fixing code, generating documentation, and summarizing results. And that’s fine as a starting point. But the gap between that and what the market is starting to expect is widening quietly. If that’s where your AI usage stops, you’re building on a false floor.

The next level has two sides to it.

1. The first is using AI to amplify your existing work. Automating data cleaning and early-stage EDA. Speeding up reporting. Getting through the repetitive parts of your workflow faster so you can focus on higher-leverage thinking. This alone can dramatically increase your output and make you more effective in your current role.

2. The second is building AI systems that create new value. Designing semantic layers that let AI reason over your company’s data. Enabling conversational analytics so stakeholders can query data in natural language instead of waiting on you. Building custom knowledge sources and RAG systems that give AI accurate, context-aware access to real datasets.

That combination, amplifying your work and building systems others depend on, is what separates someone who uses AI from someone who is indispensable because of it. It’s also what has made the biggest difference in my own career recently.

💡 About 1 in 3 AI-related job postings (31%) now require hands-on expertise across multiple specific AI skills: LLMs, RAG, prompt engineering, vector databases, and more. This isn’t conceptual. Companies want people who can build.

Pillar 2: ML & MLOps

Despite not getting as much attention as AI, this one is more relevant than ever.

The data science role is becoming more full-stack. Job descriptions are increasingly expecting data scientists to build ML/AI systems end-to-end, from development through deployment and monitoring. Companies don’t just need insights anymore. They need robust, scalable solutions that deliver value beyond the notebook.

If you’re not already investing time in things like:

• Model deployment and monitoring

• System design for ML pipelines

• Tools like Vertex AI, MLflow, or similar platforms

• Python packaging (Poetry, UV) and data validation (Pydantic)

...you should start. These aren’t “nice to have” skills anymore. They’re showing up in job descriptions for standard Data Scientist roles, not just ML Engineer positions.

This is also the pillar that opens the door to AI Engineering if that’s a path that interests you (which we’ll discuss in later sections of this guide). The overlap between a strong data scientist with MLOps skills and an AI Engineer is bigger than most people realize. I’ve been leaning back into this area myself, and it’s been one of the most valuable investments I’ve made.

Pillar 3: Human-First Strategist

Here’s the mistake a lot of people make: they think AI skills alone are enough.

They’re not.

As AI levels the playing field technically (everyone gets access to the same tools, the same models, the same capabilities), your human side becomes the differentiator. The ability to communicate clearly, tell a compelling story with data, and influence decisions is what will set you apart when everyone around you can prompt their way to a decent analysis.

The skills that make you truly resilient in this market are the ones AI cannot replace:

• Business acumen: Connecting your work to outcomes that matter, whether that’s revenue, retention, cost savings, or growth

• Strategic communication: Knowing how to distill complex ideas into something that drives action, not just understanding

• Stakeholder influence: Building the trust and alignment needed for your work to actually shape decisions

• Domain expertise: Understanding the industry and context behind the problems you’re solving, something no model can learn from a prompt

These are about judgment and influence, not just execution. And no matter how good AI tools get, they can’t replicate that.

The data scientists who will stand out in the next few years won’t be the ones with the most technical skills. They’ll be the ones who can pair those skills with the judgment to know what matters, the communication to make it land, and the influence to turn insights into action.

💡 If you want to stay valuable, combine technical proficiency with these human skills. That mix is what makes you more than someone who can query data. It makes you someone who can drive outcomes. I wrote a full article on the traits that accelerated my career growth if you want to dig deeper.

Agentic Analytics: The Shift You Can’t Ignore

If you haven’t heard this term yet, you will. A lot.

Agentic analytics represents a fundamental shift in how data work gets done. Instead of data professionals manually querying databases, building dashboards, and writing reports, AI agents step in to automate and augment large parts of that workflow, using natural language, business context, and structured reasoning.

And the backbone of this shift? The semantic layer and ontologies.

What’s a semantic layer (and why should you care)?

Think of a semantic layer as a shared definition of your company’s data (metrics, business logic, relationships) that sits between raw data and anything that consumes it (dashboards, notebooks, AI agents, stakeholders).

The concept has actually been around since the 1990s. Looker modernized it with LookML, but what’s different now is that semantic layers are no longer just powering dashboards; they’re becoming the interface through which AI agents reason about data.

💡 In September 2025, major companies like dbt Labs, Snowflake, Google, Cube, and others came together to announce the Open Semantics Initiative (OSI), a shared standard for how systems define and communicate business context. This is a big deal.

What does this mean for you?

Here’s my take: by the end of 2026, the idea of a semantic layer will start moving from cutting-edge to expected. We’ll see more “conversational analytics” capabilities across platforms, and data scientists who understand how to build and shape these semantic layers will have a significant edge.

I’ve experienced this firsthand. I recently deployed a talk-to-your-data Slackbot at my company, and one of the keys to its success was defining the semantic layer properly. You don’t need to be a data engineer to start building one. But you do need to understand why it matters.

💡 Agentic analytics, semantic layers, and ontologies are already showing up in job postings. In small numbers now, but my bet is that this will only continue to increase. Data scientists who are part of this design process early will be the ones shaping the next generation of analytics, not just reacting to it. I go into much more detail in my article on Semantic Layers and the Future of Agentic Analytics.

Should you transition to AI Engineering?

This is one of the most common questions I see data scientists wrestling with right now. And it’s a fair one, because the hype around AI Engineering roles is real. The salaries are attractive, the work sounds exciting, and the job postings are everywhere.

But before you start rewriting your LinkedIn headline, here’s what you need to know.

First, understand what AI Engineering actually is

Strip away the hype, and AI Engineering looks a lot like software engineering applied to AI systems. It’s about designing, building, and operating intelligent applications reliably in production. Less emphasis on exploratory analysis and stakeholder storytelling. More emphasis on architecture, state management, evaluation frameworks, and deployment.

It’s a spectrum, not a binary switch

What a lot of people often overlook is that the data science role is already evolving to include more AI-focused work. As the market demands more experience with AI, many data scientists are now building foundation models for tasks like recommendation, designing RAG pipelines, running AI evaluation frameworks, and even orchestrating agentic workflows.

So this isn’t necessarily about changing your title. It’s about where you sit on the spectrum between analysis-focused and systems-focused work, and how far you want to take it.

It might be right for you if...

You might genuinely thrive in AI Engineering if you:

• Feel more energized by building systems than analyzing outcomes

• Enjoy debugging integrations more than interpreting experimental results

• Get excited about architecture decisions, not just model accuracy

• Were always drawn to the engineering side of data science

But it’s not for everyone

If what gives you satisfaction is shaping decisions rather than shipping infrastructure, then moving fully into engineering could disconnect you from what you actually enjoy.

There is good money in AI Engineering, and the work is exciting. But alignment with your interests is what sustains a career over time. That’s what truly future-proofs you, not chasing the hottest job title.

💡 I wrote a full deep dive on this topic: Should You Actually Transition from Data Science to AI Engineering?. Worth reading if you’re seriously considering this path.

A practical learning roadmap

Before I share specific resources, I want to talk about how to approach learning this stuff, because the mindset matters more than the material.

The framework: build, break, learn, repeat

If you take one thing from this section, let it be this: build something, get stuck, go back to the theory, then build again. That loop is the fastest way to actually internalize these skills. Not watching a 10-hour course end-to-end. Not collecting bookmarks you’ll never open.

You start a project. You hit a wall. You go figure out why. Then you come back and keep going.

This also means there’s nothing wrong with “vibe coding” your way through the early stages. Use Claude Code, use Copilot, let AI help you move fast. But know that it will only get you so far. There’s a point where you need to stop asking “does this work?” and start asking “do I actually understand why?”

That’s when the real learning happens. And tools like NotebookLM or ChatGPT are great for bridging that gap, especially when your time outside of work is limited (which, for most of us, it is).

The foundations matter more than people admit. Knowing what’s actually happening inside the loop, or why an architecture behaves the way it does, changes how you build. It changes the questions you ask. And it’s the difference between someone who can follow a tutorial and someone who can solve problems their team actually has.

Resources I recommend

Here are just a few of the resources that have helped me the most. You don’t need to go through all of them linearly. Pick what’s relevant to where you are right now and come back to the rest when you need it.

📚 Books

• AI Engineering by Chip Huyen. The best resource I’ve found for understanding how to build AI systems in practice.

• LLM Engineer’s Handbook by Paul Iusztin. Great for going deep on LLM-specific workflows.

• Designing Machine Learning Systems by Chip Huyen. Essential for anyone investing in MLOps.

• Storytelling with Data by Cole Nussbaumer. A practical guide to communicating insights clearly and driving decisions.

• How to Win Friends and Influence People by Dale Carnegie. Timeless principles for building relationships and influencing others effectively.

🎥 YouTube Channels

• AI Engineer: Huge variety of technical talks on building AI systems.

• LangChain: Lots of great tutorials on building workflows and agentic systems.

• Andrej Karpathy: Deep, foundational understanding of how LLMs work.

👨‍💻 Other gems

• Introduction to LangGraph (Free course by LangChain)

• System Design Concepts Course and Interview Prep by freeCodeCamp.org

• Andrej Karpathy’s 1hr Intro to Large Language Models

• Building Effective Agents with LangGraph

Where to start

If you’re not sure where to begin, here’s a simple path that works:

1. Watch Andrej Karpathy’s 1hr Intro to Large Language Models: This gives you the foundational understanding of how LLMs actually work, which changes how you approach everything else.

2. Pick up AI Engineering by Chip Huyen: You don’t need to read it cover to cover. Focus on the chapters that are relevant to what you’re trying to build.

3. Build a simple LLM-powered data cleaning script: Call an LLM API, pass it a prompt that includes an overview of your dataset, and have it identify missing values and generate the code to clean it. Simple, useful, and the first step from using AI to building with it.

4. From there, go deeper: Explore agent frameworks like LangGraph, experiment with building more autonomous workflows, and start thinking about how these tools could solve real problems at your company.

Start small. Get it working. Then improve it.

💡 There are plenty of other great resources out there beyond what I’ve listed here. Don’t overcomplicate it. The path above is more than enough to get moving.

🚀 This is the most effective way to future-proof your career for AI

Everything you just read about becoming an AI-powered data scientist, the semantic layers, the agentic workflows, and building systems your team depends on, I had to piece together on my own over the course of a year. Through trial and error, building at work, studying on weekends, and understanding what the market actually rewards.

You could do the same. But it doesn’t have to take a year (or even months)

That’s why I packaged it all into the AI Workflows for Data Science bootcamp, a 6-week structured program that covers the applied AI skills this guide lays out in Pillar 1. It gives you the roadmap, the hands-on projects, and the support system to go from reading about these skills to actually building with them, in a fraction of the time it took me.

Built specifically for data scientists, by a data scientist, focused on the skills the market is demanding right now.

It’s already helped 40+ data professionals make this transition. If you’re serious about staying relevant in the market, this is the most effective path I know:

Learn More

A final note

If you’ve made it this far, you are already ahead of 80% of data professionals out there.

The truth is, no one has this fully figured out, including me. I’m navigating the same uncertainty, making bets on where the field is heading, and adjusting as I learn.

But here’s what I do know after 8+ years in this field, after getting laid off, moving countries, going from analyst to analytics team lead, and now building AI workflows in production:

The people who will thrive aren’t the ones who learn the most tools. They’re the ones who build the right skills, stay curious, and actually ship things.

The market is shifting. The window to get ahead of that shift, instead of playing catch-up, is still open. But believe me, at the rate we are moving, it won’t stay open forever.

So pick a pillar. Start building. And don’t wait until you feel “ready,” because honestly, no one ever does.

You’ve got this. 🙏

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this breakdown helps you get started future-proofing your career.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

The focus there is not on tools, but on frameworks and end-to-end systems.

And that’s where the real signal is.

If you want to understand how far AI could transform how we do data science over the next few years, you need to look at the direction these systems are heading.

So I’ve gathered 6 research papers covering recent advances in autonomous AI agents designed to automate data science, machine learning research, and even broader scientific discovery.

At the end, I also share my take on what this actually means for us today and how we should prepare.

1. DeepAnalyze: Autonomous Data Science via Agentic LLMs

This research paper introduces DeepAnalyze-8B, an agentic model trained to autonomously handle the entire data science pipeline from raw data to professional research reports. It uses a curriculum-based training paradigm to acquire complex problem-solving capabilities similar to those of a human data scientist.

2. AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench

This paper studies AI agents that compete in Kaggle competitions by exploring solution spaces using search strategies like Monte Carlo Tree Search and evolutionary methods. The study shows that pairing these search policies with well-designed operators significantly improves performance in real ML engineering tasks.

3. The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

This paper introduces an autonomous framework that can perform the entire ML research lifecycle: generating ideas, running experiments, analyzing results, and writing full research papers. It even includes an automated LLM-based peer-review system to evaluate the generated work.

4. Jupiter: Enhancing LLM Data Analysis Capabilities via Notebook and Inference-Time Value-Guided Search

This paper introduces NbQA, a large dataset of real Jupyter notebook analysis tasks, and JUPITER, a framework that treats notebook-based data analysis as a sequential decision-making search problem. A value model guides the reasoning process to efficiently solve multi-step data analysis tasks.

5. DS-STAR: A state-of-the-art versatile data science agent

This paper by Google proposes an agentic system designed to handle real-world data science workflows involving heterogeneous data sources such as CSV, JSON, and text. It decomposes analysis into sub-questions and iteratively builds evidence-based reports to minimize hallucinations.

6. Kosmos: An AI Scientist for Autonomous Discovery

This paper presents a scientific discovery agent that coordinates literature search and data analysis through a structured world model. By running long autonomous research loops, it can read thousands of papers and execute large-scale experiments to generate traceable scientific discoveries.

The common denominator

Across these papers, one thing becomes clear: we are moving from AI as a tool to systems that can operate across the entire data science lifecycle. Not just assisting with isolated tasks, but handling the full loop, from defining problems to running analyses and producing research-level outputs.

A big part of this shift comes from how these systems are designed. They don’t rely on a single pass. Instead, they iterate. Through feedback loops, search strategies, and self-reflection, they refine their outputs step by step, getting closer to something that actually resembles how a data scientist would approach a problem.

When you combine this with large-scale training data, like real Jupyter notebooks, and more structured ways of interacting with data and external knowledge, the result is systems that can reproduce fairly complex reasoning.

There are a few patterns that show up consistently across these papers:

• End-to-end ownership: These systems don’t just assist, they handle the full workflow, from ideation to final output.

• Iterative refinement: Instead of one-shot answers, they rely on loops, evaluation, and correction.

• Decomposition of problems: Open-ended questions are broken down into smaller, verifiable steps grounded in data, code, or literature.

• Built-in evaluation: With automated reviewing and benchmarks, they can critique and improve their own outputs.

Another important shift is reliability. By grounding their reasoning in executable steps and verifiable sources, these systems start to reduce one of the biggest limitations we’ve seen so far: hallucinations in open-ended tasks.

Put together, this starts to look less like a model you prompt, and more like a system that can iterate on its own work, explore a space of ideas, and converge toward something useful.

My honest take

When I shared these papers on my LinkedIn, I got a comment that stood out.

Because, regardless of how far these frameworks go in automating the analysis process, the question remains the same:

Can they actually operate under real-world constraints, in companies where more than accuracy matters, and where business context is far more nuanced than what can fit into a few documentation pages?

For the foreseeable future, I still see a strong need for a human-in-the-loop, regardless of how capable these models become or how large their context windows get (and by the way, Claude just made their 1M context window widely available for their latest models).

Still, the limiting factor isn’t intelligence, it’s context.

And more specifically, how that context is structured, injected, and continuously updated. In practice, a large part of what drives good analysis is not explicitly written down. It lives in things like:

• Shifting priorities across teams

• Implicit assumptions behind metrics and decisions

• Trade-offs that only emerge through experience and stakeholder context

This is the layer that doesn’t neatly fit into prompts or documentation.

That said, I do think there are specific environments where these agents will start defining and evaluating their own hypotheses. In well-scoped problems, with reliable data and clear feedback signals, they have a clear advantage. They can explore a much larger hypothesis space and validate ideas at a speed that is simply not possible for a human.

Where this becomes powerful is in narrowing the gap between exploration and validation, not by removing the human, but by making it possible to test more ideas, faster, without losing control over what actually matters.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this breakdown gives you a glimpse into the future of data science.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

📌 By the way, enrollment for the April cohort of my AI workflows Bootcamp is now open. This 6-week program is designed to help data scientists develop the judgment, systems thinking, and applied AI skills that amplify their impact and set them apart in the market.

There’s a lot of talk right now about how AI is transforming the role of data scientists.

And well, depending on who you listen to, our jobs are either about to disappear or about to turn into something completely different.

So I thought it might be useful to do something simple, show you what my job actually looks like today, working at a scale-up that is adopting AI very fast.

And I can tell you right now that while AI is definitely changing the way I work, the reality of the job probably looks different from what most people imagine.

Let’s break it down.

Yes, I Still do Data Science

If you shadowed me for a week, most of what you’d see would still look like a fairly traditional data science role in tech.

I spend a large portion of my time working on projects that support product and business decisions.

Typical questions I work on include things like:

• Retention and engagement analysis

• Understanding user behavior

• Evaluating product changes through A/B testing

• Marketing and growth analytics

At one point, I built a Markov model to better understand user behavior across the product. Not exactly the type of model that gets people excited on LinkedIn these days, but still extremely useful.

And by the way, I still spend more time writing SQL than Python.

So even in 2026, a lot of my job is still about applying the fundamentals well.

My Week Is Mostly Deep Work (Thankfully)

One thing that might surprise people about my current role is the number of meetings I have.

In my case, it’s actually quite healthy. My calendar usually includes only a few recurring meetings:

• Standups with the data team

• Sprint planning sessions

• Occasional sessions based on projects and syncs with stakeholders

The rest of the time is mostly dedicated to deep work on projects that span weeks or sometimes an entire quarter.

That said, this hasn’t always been the case. Earlier in my career, I had roles where I was much more embedded with stakeholders and spent significantly more time in meetings.

What I’m experiencing now has a lot to do with my seniority level and the company culture at Nextory. There’s a strong understanding that if data scientists are buried in meetings all day, it becomes very difficult to actually produce meaningful work.

Right now, this structure allows me to focus much more on building, analyzing, and pushing projects forward.

AI Is Now Embedded in My Workflow

Now, the part that has changed significantly is how I work day-to-day.

AI is embedded into almost everything I do.

At Nextory, we are actually encouraged to experiment with AI tools and share how we’re using them to improve our work. In fact, I’m one of the stewards in the company helping promote AI adoption internally.

So how do I actually use AI?

Mostly to amplify my productivity.

For example, I regularly use AI to:

• Write SQL queries faster

• Automate parts of data cleaning and EDA

• Generate complex Python code for analysis or modeling

• Brainstorm analytical approaches

• Summarize analysis results

• Write documentation

• Navigate our data warehouse and help improve data models

I also rely heavily on tools like Cursor for coding, Gemini inside the GCP ecosystem, and ChatGPT for brainstorming ideas and structuring analysis, and the same is true for my coworkers.

At this point, AI feels less like a separate tool and more like an extra layer in the workflow.

My role is a clear example of the impact of AI in our field and exemplifies the trend we are already seeing in the job market. I recently did a full analysis on this in case you are interested:

AI Hasn’t Replaced the (Real) Hard Parts

When people talk about AI changing data science, the conversation often focuses on automation: writing SQL faster, generating Python code, and building dashboards with fewer manual steps.

And yes, those parts are changing.

But what AI hasn’t replaced is the core responsibility of the job.

Things like:

• Stakeholder communication

• Defining the right questions

• Building trust with teams

• Driving adoption of insights

AI can help me brainstorm, it can help me move faster, but it still depends heavily on the context I provide.

Most of the time, I’m the one guiding the process and setting the constraints. In other words, AI speeds up the work, but the judgment still sits with me.

We’re Also Building Complex AI Systems

Beyond using AI to improve my own workflow, we’re also starting to build AI-powered systems inside the company.

One example is a Talk-to-Your-Data Slack bot that I built for stakeholders. The idea was simple: allow non-technical stakeholders to ask questions about our data using natural language and get insights faster without needing to write SQL.

It was our first step toward something closer to what people are now calling agentic analytics.

At the same time, other teams in the company are building much more sophisticated AI systems that are user-facing. For example, Nextory recently launched an AI Librarian designed to help users discover books through conversational interactions.

Projects like that involve data scientists, software engineers, and ML engineers working together to build production-grade AI systems that directly impact the product experience.

What’s interesting is that this is creating a wider spectrum of roles.

Some data scientists (like myself) remain closer to the business layer, focusing on analysis, experimentation, and decision-making. Others are moving closer to the engineering side, helping design and build the AI systems that power new product capabilities.

If you want to know more about this transition towards an engineering-focused role, then you want to read this article:

Final Thoughts

If someone shadowed me for a week, the biggest surprise probably wouldn’t be that I use AI.

It would be how much work I’m able to accomplish because of it.

The fundamentals of the job are still there: understanding problems, analyzing data, and helping teams make better decisions.

But AI is amplifying the work. It allows me to move faster, automate parts of my workflow, and explore complex ideas in days rather than weeks or months.

And from a career perspective, it’s also clear that these skills are becoming increasingly valuable. Being able to combine data science fundamentals with AI-driven workflows gives me a level of leverage that simply didn’t exist a few years ago.

Whatever happens in this role, I know those skills will transfer well. The tools are evolving quickly, but the core of the work remains the same.

And for me, that combination is exactly what makes this moment in the field so interesting.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this breakdown gives you a more grounded perspective on how AI is disrupting our field.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

“Is it time to transition into AI Engineering?” “Is it too late?”

If you spend even a few minutes on LinkedIn, it can feel like the center of gravity has shifted. Suddenly, everyone is building agents, deploying RAG pipelines, experimenting with orchestration frameworks, and rebranding themselves as AI Engineers.

But before you change your title, you need to answer a deeper question:

Are you genuinely drawn to that work…or are you reacting to the fear of becoming irrelevant?

Let’s slow it down and look at it properly.

What AI Engineers Actually Do

Strip away the hype, and AI Engineering looks a lot like software engineering applied to AI systems. It’s about designing, building, and operating intelligent applications reliably in production.

In practice, this often means:

• Designing LLM-powered applications

• Implementing retrieval-augmented generation (RAG) pipelines

• Orchestrating multi-step workflows or agents

• Connecting models to APIs, databases, and internal tools

• Monitoring performance, latency, cost, and reliability

The focus is far from discovering insights in a dataset and more on constructing systems that consistently deliver value at scale.

You’re thinking about architecture, state management, evaluation frameworks, logging, versioning, and deployment environments. You’re asking questions like:

• How do we make this robust?

• How do we reduce hallucinations?

• How do we measure quality over time?

Compared to traditional Data Science roles, there is typically less emphasis on exploratory analysis, hypothesis-driven experimentation, and stakeholder-facing storytelling.

That doesn’t make one role superior to the other, but it does mean they reward different strengths.

Data Science Is Already Evolving

I often talk to people seriously considering this transition, and what they often overlook is that Data Science hasn’t stayed static.

Modern data scientists are increasingly expected to:

• Write production-ready code

• Own pipelines end-to-end

• Collaborate closely with engineering

• Think about deployment and monitoring

• Use AI tools to accelerate analysis

The “pure modeling” specialist is not disappearing (anytime soon at least) but is becoming rarer outside very large organizations. In many teams (like the one I’m currently in at Nextory), the line between Data Scientist and AI Engineer is already blurred. You may not carry the AI Engineer title, but you’re doing parts of the job.

So this transition isn’t a binary jump. It’s a movement along a spectrum.

If you want to learn more about what this looks like in practice, check out this job market report:

Why You Might Want to Transition

But okay, let’s figure out if this is actually the right move for you, because you might genuinely thrive in AI Engineering if you feel more energized by building systems than analyzing outcomes.

If your curiosity is triggered by questions like:

• How should this architecture be designed?

• What’s the cleanest way to structure this workflow?

• How do we evaluate this agent automatically?

If you enjoy debugging integrations more than interpreting experimental results, that’s a signal.

Some data scientists were always drawn to building intelligent systems. For years, that meant moving toward ML Engineering or MLOps. Now, AI Engineering is another branch of that same instinct.

If you’ve consistently been more excited by building and operationalizing systems than by running experiments or presenting insights, this isn’t hype; it’s alignment.

Why You Might Not Want To…

Many data scientists chose this path because they thrive in ambiguity and enjoy framing messy business problems, designing experiments, and translating data into decisions.

There is a creative element in analysis that often goes unnoticed: defining the right metric, interpreting conflicting signals, identifying behavioral patterns, and influencing product strategy.

If what gives you satisfaction is shaping decisions rather than shipping infrastructure, then moving fully into engineering could slowly disconnect you from what you actually enjoy.

In reality, most data scientists won’t fully “switch” roles. They’ll become more AI-native instead. That means understanding how LLM systems work, being able to prototype AI workflows, knowing how to evaluate outputs, and collaborating effectively with engineers who own the deeper infrastructure.

The strongest profiles will combine business context, analytical rigor, engineering awareness, and AI literacy. The future is less about changing your title and more about expanding your toolkit.

Why I’m Not Transitioning

Personally, I’m not moving into a full AI Engineering role, at least not right now.

That’s not because I’m ignoring where the field is going. I’m already building AI systems to automate parts of my work and have built AI-driven products on the side, and teaching practical AI workflows has become a core part of what I do.

The reason is simpler: I genuinely enjoy staying close to the business layer. Shaping metrics, designing experiments, and connecting analysis to strategy is what feels most meaningful to me.

If I moved too far into full-time infrastructure and architecture, I’d lose the proximity to decision-making that makes the work fulfilling.

Final Thoughts

The real risk isn’t choosing the wrong title. The real risk is drifting into a role that doesn’t align with your strengths simply because the market shifted.

There is good money in AI Engineering, and working with emerging technology can be genuinely exciting. But alignment with your interests and the kind of work that actually energizes you is what sustains a career over time.

That’s what truly future-proofs you.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this guide helps you decide what is the best path for you.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

Many of them I’ve run into while working on projects, experimenting on my own, or simply paying attention to what the community is building. Some stood out because they help with very concrete parts of the job, like exploration, feature work, forecasting, and working with messy inputs.

Here are my absolute favorites so far.

1. AI Sheets

AI Sheets lets you enrich, label, and transform tabular data using LLMs directly inside a spreadsheet-like interface.

In practice, this is useful for things like generating features, classifying text columns, or adding weak labels to datasets before training a model.

2. Data Formulator

Data Formulator turns natural-language descriptions into concrete data transformations and visualizations.

It’s especially useful during early EDA, when you’re iterating on how to slice the data and don’t want to hand-write transformations you might throw away five minutes later.

3. Jupyter AI

Jupyter AI brings LLM assistance directly into Jupyter notebooks, tightly coupled with your code and variables.

This works well for refactoring analysis code, explaining unfamiliar notebooks, or quickly prototyping modeling steps without breaking the notebook workflow.

If you want a more in-depth view of this tool, check out this article.

4. PandasAI

PandasAI lets you ask questions about a DataFrame in natural language and executes the corresponding pandas operations.

It’s not a replacement for pandas, but it’s very effective for speeding up EDA and sanity checks when you already know what you want to inspect.

I wrote a two-part series on how to get the most out of PandasAI. Here is the article.

5. ChartDB

ChartDB automatically generates visual representations of database schemas and table relationships.

This is particularly helpful when onboarding onto a new data warehouse and trying to understand how raw tables connect before writing models or features.

6. MCP Toolbox for Databases

MCP Toolbox exposes databases as structured, permissioned tools that LLM agents can query safely.

This is a foundational piece if you’re building agentic analytics systems that need database access without giving an LLM free-form SQL over production data.

If you are ready to start building MCP servers, read this article.

7. TimeGPT

TimeGPT is a pretrained foundation model for time-series forecasting that works out of the box via an API.

It’s a strong baseline when you want fast, reasonable forecasts without spending days on feature engineering and model tuning.

If you want to learn more about foundation models, check out this article.

Bonus: MarkItDown

MarkItDown converts PDFs, Word files, and presentations into clean, structured Markdown.

For data scientists, this is especially useful when preparing documents for retrieval pipelines, evaluation datasets, or any RAG-style setup.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this guide helps you get the most out of your AI tools.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

It’s what separates good data scientists from great ones.

It’s what every employer is looking for in their next hire.

And it’s what defines your value as a data scientist.

But creating impact takes more than just technical skills. It’s not enough to build great models; you need to make them accessible and usable.

That’s why deployment is so important.

A model sitting in a Jupyter notebook isn’t doing anything. To create real value, you need to deploy it.

So today, I want to show you how to do just that using Streamlit

But going beyond the basic Streamlit Cloud.

This is the same method I’ve used to coach many of my mentees in the past, helping them build projects that impress hiring managers and get hired.

Here is what we’ll cover:

1. Why Streamlit Cloud isn’t enough

2. How to Dockerize your Streamlit app (practical guide)

3. Deploying a Dockerized Streamlit app

1. Why Streamlit Cloud Isn’t Enough

Nowadays, using Streamlit to bring your portfolio projects to life is a must. It’s a great way to show your ability to build projects end-to-end.

Well, sort of.

Most projects default to deploying using Streamlit Cloud. Just a few clicks, and it’s live for anyone to use. Easy! Great for quick prototypes.

But easy comes at a cost.

The technical skills don’t always transfer beyond Streamlit’s ecosystem. And the idea here is to show we can go beyond simple prototypes.

This is where Docker comes in.

With Docker, you’re no longer just deploying an app; you’re packaging it in a way that works anywhere.

And guess what? It’s the industry standard for containerizing in software development, including data science and machine learning workflows.

If you are new to Docker, I recommend you first read this article to familiarise yourself with the key concepts:

Otherwise, here is a quick look at our strategy:

Let’s break it down.

2. How to Dockerize Your Streamlit App (Practical Guide)

Deploying with Docker means you’re not just launching an app, you’re creating something that works anywhere.

But to do that, you need a Dockerfile, which works as a simple script that tells Docker how to package your app:

• What dependencies to install

• What files to include

• How to run it.

Let’s see what that looks like

Minimal Dockerfile template (ready-to-use)

Here’s a simple Dockerfile that works for Streamlit.

That’s it. Minimal, efficient, and does the job.

But of course, this is just a starting point. You can always tweak it to meet your specific needs (e.g., you can use a later version of Python).

You can download the exact template here.

Running your Docker container locally

Now, let’s test it before deployment.

Step 1: Build the Docker image (only needs to be done once):

docker build -t my-streamlit-app .

Step 2: Run the container (this starts your app inside Docker):

docker run -p 8501:8501 my-streamlit-app

Step 3: Visit http://localhost:8501 and boom, your app is running inside Docker.

At this point, you’ve Dockerized your Streamlit app. Now, let’s get it live.

3. Deploying a Dockerized Streamlit App on Render

Now that your app runs locally, it’s time to deploy it properly.

I recommend using Render.

Why Render?

• Free tier (so you don’t burn cash while testing).

• Supports Docker natively (which means no weird config issues).

• Simple setup (no Kubernetes, no headaches).

Let’s deploy.

Simple steps to deploy a Dockerized app

1. Push your project (with the Dockerfile) to GitHub (if it’s not there already).

2. Sign up on Render and create a new Web Service.

3. Connect your GitHub repo and select the branch with your Dockerfile.

4. Under “Language,” choose “Docker.”

5. Click “Deploy.”

💡 Render will automatically build your Docker image from your Dockerfile, you do not need to push an image to a container registry.

Once it’s live, you’ll get a public URL to share your deployed project.

And just like that, you’ve taken your Streamlit project beyond just a “quick” prototype.

But…what if you already have a Docker image?

Alternatively, you can also use Render to deploy a prebuilt Docker image from a registry such as Docker Hub.

💡 This is useful when you want faster deployments or when the image is already built as part of a CI pipeline. In production teams, this is often the standard setup: build once, deploy many times.

Here is the full documentation for this approach.

Just know that both approaches are valid. The difference is where the image is built and how that build step is managed.

Ultimately, the real goal is to use Docker to make your application portable and reproducible, turning a local project into something that can be reliably deployed in a real environment.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you found these tips insightful.

See you next week!

- Andres

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

The main takeaways from those two studies were as follows:

1. The need for data scientists is not disappearing; in fact, there is an increasing demand for data talent that knows AI

2. Senior roles are benefiting the most from AI adoption

You can read more about how those conclusions were drawn here.

Those findings were directionally clear, but they were also high-level.

They told us what was happening, not necessarily how it was showing up in actual hiring. And given how fast AI tools and expectations have evolved over the past year, I didn’t feel comfortable assuming the story had stayed the same.

That’s why I wanted to look closer.

So I scraped and analyzed 700+ Data Scientist job postings in the US, spanning November through January 2026. This report is not meant to be definitive, but it provides a strong early signal into how demand for AI skills is actually shaping up as we enter the year.

I will cover:

• What’s changed since the last analysis?

• Which AI skills are actually being asked for in data scientist roles today?

• Who is feeling the impact of AI demand the most?

📌 And at the end, I’ll share some strategies for how data scientists can adapt their skill sets and stay competitive.

Let’s get to it!

The core skill stack for data scientists hasn’t changed, but AI adoption has accelerated

Looking at the most mentioned skills across all data science job postings, the fundamentals remain stable: Python, machine learning, and SQL still dominate.

What stands out is that AI has increased in demand, referenced in more than half of all postings. And most notably, LLMs have entered the top 10 when compared to the previous year. This suggests that AI is no longer framed as a niche specialization, but as an expected extension of the standard data science toolkit.

That said, not every mention of the keyword “AI” implies deep technical ownership, many roles reference AI in a broader sense, such as collaborating on or supporting AI-related work.

Within AI-focused data science roles, LLMs clearly lead skill demand.

Zooming in on jobs that explicitly mention AI, LLMs emerge as the most requested AI skill, followed closely by GenAI and NLP.

Beyond these top three, demand shifts toward more applied capabilities such as RAG, AI agents, prompt engineering, and orchestration, indicating that companies are increasingly looking for data scientists who can work with AI systems in practice, not just conceptually.

AI-heavy roles skew toward mid-to-senior Data Scientists

When we look at where these AI requirements appear across seniority levels, the demand is far from evenly distributed.

The majority of AI-focused postings target mid and senior data scientists, reinforcing the idea that companies expect experienced practitioners to translate AI capabilities into real systems and business outcomes.

By the way, this aligns closely with the earlier Harvard findings, where the benefits of AI adoption are most concentrated in more senior roles

Some data scientist roles want AI literacy, others want hands-on builders

The distribution of AI skill requirements shows a split in how companies think about AI capability. Many roles still ask for a small number of broadly defined AI skills, but a significant portion now require experience across multiple concrete areas, such as LLMs, RAG, prompt engineering, and vector databases.

Here we can see a snippet of what this looks like across different job postings:

This points to growing demand for data scientists who can actually build and deploy AI-driven systems, not just understand them conceptually.

AI skills are mostly embedded in standard Data Scientist roles

Even when AI skills are explicitly required, the majority of postings still use standard titles like “Data Scientist” (which only specify seniority, location, or traditional domains), with AI referenced in the skill requirements rather than the role name.

Only a smaller share of roles explicitly call out AI, GenAI, or LLMs in the title, indicating that, at least in the current data, AI expectations are primarily expressed through requirements, not naming conventions.

Preliminary findings: AI-focused data science roles are beginning to show a salary lift

Across most seniority levels, data science roles that explicitly mention AI tend to show higher median salaries than comparable roles that do not. This suggests an emerging AI-related salary premium within data science, but the signal is still early.

Salary data is only available for roughly a third of postings, and some seniority bands have limited samples, so this should be interpreted as directional rather than conclusive.

Summary

Here is a recap of the key findings:

• Are AI skills in demand for data scientists? Yes, about 60% of postings expect AI capability, either broadly or through specific skills like LLMs, RAG, and AI agents.

• Which AI skills should I focus on? Experience with LLMs was the #1 AI skill requirement, appearing in ~20% of all job postings.

  • About 1 in 3 AI-related postings (31%) require hands-on expertise across multiple specific AI skills: LLMs, RAG, Prompt Engineering, vector databases, and more. This signals growing demand for practitioners who can build and deploy AI systems, not just understand AI conceptually.

• Which roles are the most impacted? AI demand concentrates at mid-to-senior levels: 73% of postings mentioning AI skills target mid (43%) and senior (30%) roles, while entry-level positions account for less than 6%.

• Are job titles evolving? Currently, 73% of Data Scientist postings with AI requirements use standard titles like “Data Scientist” (and only specify seniority, location, or traditional domains) while 27% explicitly call out AI/GenAI/LLM in the job title, suggesting AI skills are increasingly expected within traditional roles.

• Any other trends I should keep an eye on? Agentic Analytics, Semantic Layers, and ontologies are already being mentioned, although in just a handful of jobs. My bet is that this will only continue to increase as the year progresses.

One thing is clear: AI is becoming a core part of the data scientist role, and it is already influencing expectations, scope, and career progression.

Here is my proven roadmap to build these skills in 2026

Everything I’ve shared so far reflects clear patterns already taking shape across the field and what they imply for data scientists going forward.

Over the past year, one way I’ve personally responded to these shifts is by building and refining a bootcamp focused on agentic analytics, AI workflows, and system-level thinking. It’s designed for data scientists who want to move beyond isolated analyses and start building end-to-end systems that actually scale their impact.

So far, we have helped over 40 data professionals future proof their careers. This is what some of the alumni had to say:

Enrollment for the April 2026 cohort is now open, but seats are limited:

Enroll today

Enrollment closes in only 6 days. Keep in mind that if you miss this cohort, the next one won’t be available until later in the year.

Regarding the analysis approach

My plan is to continue scraping job postings throughout the year and revisit this analysis again before the summer.

In the meantime, I’m going to clean up the Streamlit app I used for the analysis, deploy it, and make both the implementation and the scraped data open source so others can build on it or run their own analyses.

Thank you for reading! I hope you found this article insightful. And don’t forget to look into the “AI Workflows“ bootcamp, enrollment closes in just a few days.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

We saw AI truly disrupt the job market and the field itself, raising fundamental questions about the future of data science. At the same time, much of what we consumed online and in the news was wrapped in a thick layer of hype and misinformation.

No, data science is not “dying“. And no, AGI does not seem anywhere close, yet. But even though not every promise actually materialised, we did see a set of clear patterns emerge that are simply hard to ignore.

I know this, because I spent the better part of the year:

• Studying the data & AI job market

• Building AI workflows to automate and optimise my own work and future-proof my career

• Attending conferences and having meaningful conversations across the industry

• Running two data science and AI-focused newsletters that forced me to keep up with the news

So instead of making wild predictions, I want to share my bets for 2026, grounded in what we are already seeing today.

And at the end, I’ll share practical advice on what data scientists should focus on to accelerate their careers.

Bet #1: Agentic analytics will take center stage over traditional analytics

2026 will undoubtedly be the year when we start seeing these new types of data science workflows become normalized, where data scientists work alongside AI agents to automate and augment their work.

This goes beyond using tools like ChatGPT or Cursor to write code faster. Instead, we are talking about AI-powered workflows that can:

• Explore data proactively

• Detect patterns or anomalies

• Run follow-up analyses

• Make decisions with minimal human intervention

This is what the industry is calling Agentic Analytics.

To be clear, when I say agentic analytics will take center stage, I don’t mean that traditional analytics disappears. I mean that hand-written, step-by-step analysis stops being the default starting point, and instead becomes something we step into when judgment, nuance, or validation is required.

As a result, data scientists will start acting more as system designers and architects, letting these systems do much of the heavy lifting and getting involved primarily in the parts that require judgment and context, as well as presenting work in a way that actually drives decisions.

This will not happen all at once, though…

We will continue to see a selective group of data scientists lead the way and truly become 10x at what they do. This will impact both performance and the amount of value companies will expect a single data scientist to provide.

Towards the end of 2026, as we prepare to step into 2027, we will see a large wave of data scientists feeling the urgency to catch up. Improving their ability to think more like system designers, build these systems themselves, and adapt to the new tools already establishing themselves in the market.

Bet #2: We will witness the commoditization of the semantic layer

The modern data stack is no longer enough.

Companies will increasingly feel the need to move toward a combined data and AI stack.

As a result, having a semantic layer will become a standard. Not as a nice-to-have, but as a prerequisite for keeping up with the next wave of agentic analytics.

This shift will highlight the value data engineers bring to the organization. But building and maintaining a semantic layer will not be a purely engineering problem. It will require the involvement and judgment of data scientists, especially when deciding how data should be interpreted, exposed, and used downstream.

For those organizations that have already adopted a semantic layer to support their Business Intelligence, we will see them optimize it with AI systems in mind. In particular, systems that allow non-technical stakeholders to “talk to their data” and request insights using natural language.

This will give way to a new definition of self-service analytics. One where the semantic layer becomes the shared foundation not just for dashboards, but for AI-driven decision-making across the company.

Bet #3: Foundation models will become central for tackling core data science problems more effectively

Instead of building a new model for every task, more teams will start reaching for foundation models as the starting point.

The reason is simple: they unify solutions. You move away from a fragmented landscape of narrowly trained models and toward shared representations that can be adapted across multiple use cases.

We are already seeing impressive progress in foundation models for anomaly detection and time-series forecasting. And while we will continue to see improvements there, I expect a lot of effort to go into foundation models for recommendation systems and ranking, along with other core areas where companies need models that generalize well and can be reused across products.

We have already seen signs of this direction from Netflix, where the industry is moving away from building isolated recommenders and toward more unified approaches.

What this means in practice is that the “modeling work” will change.

More data scientists will spend time training, adapting, and evaluating foundation models, and less time building task-specific models from scratch. The competitive advantage shifts toward understanding how to apply these foundations to real product problems, and how to ground them in your data, constraints, and success metrics.

Bet #4: The scope of data scientist roles will continue to widen

Over the past few years, we have seen the rise of more specialised roles: MLOps, AI Engineer, and similar titles are becoming more common. Data science, however, is not moving in the same direction.

Rather than becoming more specialised, the role is becoming more full-stack.

The reason is that data science sits at the intersection of business, engineering, and decision-making. That combination is simply hard to replace.

And now that AI has introduced new tools and techniques into the picture, we will see more data science roles pop up that require these types of responsibilities:

• Build ML/AI systems end-to-end.

• Build RAG systems to handle unstructured data

• Train and work with foundation models

• Handle guardrails and AI evaluations

By the way, I’ve done a deep analysis of the job market to understand what new AI tools and techniques are emerging for data science, and I will be sharing my full analysis next week, stay tuned.

And as the AI tools advance to facilitate the creation of these solutions, data scientists will evolve to think more like system designers. Designing, orchestrating, and maintaining their own workflows rather than working on isolated pieces.

Those who remain adaptable will thrive, while those who try to hold onto a narrowly defined version of the role will find it harder to keep up.

To clarify, I don’t believe specialised DS/Analytics roles will disappear. Product analytics, recommender systems, and other focused areas will continue to be in demand. But general data science roles will widen in scope, both by expectation and as a natural evolution for those who want to strengthen their skill set and remain future-proof

Bet #5: AI will turn non-technical stakeholders into entry-level data analysts

This shift is already underway.

Many non-technical stakeholders already have strong analytical instincts. What has been missing is not curiosity or problem framing, but the ability to handle the technical parts of the work. AI tools are increasingly filling that gap by taking care of tasks like querying data, writing code, and running basic analyses.

We have already seen this in product teams, where stakeholders are now able to build their own demos and proof of concepts. But it is not limited to product; we are starting to see the same pattern with data work. Stakeholders can upload CSVs or other structured data and have tools like ChatGPT or Claude analyze it for them, help them write basic SQL, or generate small scripts to answer concrete questions.

As a result, they rely less on specialized roles like data analysts for simple, exploratory work.

This trend is reinforced by two parallel shifts:

• Everyday tools are becoming analytical tools: Excel, Google Sheets, and similar environments are rapidly gaining AI-powered features that make working with data more accessible. As these tools evolve, so do the analytical capabilities of the people using them.

• BI tools are becoming conversational: With the rise of agentic analytics, non-technical users can increasingly talk to their data using natural language and get answers on demand.

Taken together, this doesn’t necessarily eliminate the need for data analysts, but it does raise the baseline. Many stakeholders will start operating at an entry-level analyst level, which will change both expectations and how analytical work is distributed across organizations.

If you want to get ahead, bring this with you into 2026

I know I mentioned quite a few changes coming into our field, and it might feel overwhelming to many of you, but at a high level, your strategy should be built around three areas, each helping you accelerate your career and stay relevant in the market:

1. Double down on human-centric skills
   As more of the execution gets automated, skills like judgment, communication, and the ability to turn insights into decisions will be what truly differentiate strong data scientists.

2. Focus on understanding how to build end-to-end systems
   The biggest leverage will come from understanding how data, models, infrastructure, and decision-making fit together, not just from optimizing individual pieces.

3. Don’t wait until it’s too late, start future-proofing your career today
   The gap between those who adapt early and those who wait will widen faster than most people expect.

Here is my proven roadmap to build these skills in 2026

Everything I’ve shared so far reflects clear patterns already taking shape across the field and what they imply for data scientists going forward.

Over the past year, one way I’ve personally responded to these shifts is by building and refining a bootcamp focused on agentic analytics, AI workflows, and system-level thinking. It’s designed for data scientists who want to move beyond isolated analyses and start building end-to-end systems that actually scale their impact.

Last cohort, we welcomed 22 motivated students. This is what some of the alumni had to say:

Enrollment for the January 2026 cohort is now open, but seats are limited. Use the coupon code JAN26-25-ANDRES to receive 25% off (Only 10 left):

Enroll today

Keep in mind that if you miss this cohort, the next one won’t be available until mid-year.

Thank you for reading! I hope you found this article insightful. And share in the comments if you think there are any other trends I should’ve highlighted.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

Although a lot of people still think of Colab as “Jupyter in the Cloud“, something they reach for when they need GPU power rather than a primary environment.

I genuinely think these latest updates are tipping the balance.

Colab is not only integrating much more naturally into our existing workflows, but also offering powerful features that are simply hard to match.

Let me show you exactly what I mean, and at the end of the article, I’ll share my opinion on which is my tool of choice.

Update 1: The VS Code Extension for Colab

Google released an official VS Code extension for Colab notebooks, allowing you to open, edit, and run Colab directly inside the VS Code interface, or one of its forks like Windsurf, Cursor AI, etc.

Why this matters:

• You get the full VS Code experience: richer editing, better linting, extensions, file navigation, Git integration, and more.

• The execution still happens on Colab’s backend, meaning you keep the speed and GPU access, but with a real development environment on top.

For analysts and data scientists who already use VS Code for Python development, this single update removes one of the biggest sources of friction.

Update 2: Free Access to Gemini and Gemma Models

Google Colab just announced two updates that make their generative AI features more accessible to everyone:

1. The google.colab.ai Python library is available to all Colab users, including free-tier users, which gives you access to their Gemini and Gemma models.

2. The AI Prompt Cell for a more intuitive no/low code AI workflow. It is basically a dedicated cell type designed for low-code interaction with LLMs.

Check out this notebook to get started.

Update 3: Colab Data Explorer: One-click access to Kaggle datasets and models

Kaggle and Google have collaborated to introduce the Colab Data Explorer, a new feature that allows users to seamlessly search and access Kaggle datasets and models directly within Google Colab.

You no longer have to introduce any code or leave your notebook, just browse from the sidebar, pick what you need, run the snippet, and start working.

This integration streamlines the workflow for data scientists and ML engineers by enabling them to find and incorporate relevant datasets and pre-trained models without leaving their Colab environment.

My opinion: Colab vs Jupyter

Jupyter has been my default analysis environment since I graduated from university back in 2017.

And their recent updates to their Jupyter AI extension really felt like the right move forward in keeping up with how AI-assisted coding is becoming a natural part of how we explore data, iterate on ideas, and write production-ready analysis today.

But Colab is evolving fast, and these three updates, combined with its seamless integration of Gemini chat, put it at the clear top for fast experimentation, collaborative work, and AI-first workflows without friction.

To me, the only reason Jupyter still ranks high is how naturally it fits into a properly set up data science project, where notebooks are the starting point rather than the end, with your environment, your dependencies, and no surprises when you reopen the notebook a week later.

But I’m curious to hear your thoughts…

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you found this breakdown useful.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

Data Scientists spend a disproportionate amount of time cleaning data rather than analysing it and extracting insights

Fortunately for us, this is one area where AI can actually be extremely useful in helping automate and provide huge productivity gains.

Which is why it was one of the first things I set out to build earlier this year as I began my journey with AI workflows and agents. I built an agent to handle the repetitive grunt work of fixing data types, removing outliers, handling missing values, etc, so I could focus more time on the analysis that matters.

Today, I will show you how to build a lightweight version of this system using LangGraph, so you can start bringing automation into your own analysis workflow.

But before that, let me show you the agent in action:

Now, let’s get to the implementation…

The data cleaning workflow

Before we get to the code, let’s visualise our workflow at a high level using LangSmith Studio.

As you can see, the graph itself is actually very simple. It only contains 3 functional nodes and a single conditional edge. Here is an overview of what each one of these nodes handles:

1. create_data_cleaner_code: Call an LLM and instruct it to generate data cleaning Python code based on a series of requirements and a preview of our dataset.

2. execute_data_cleaner_code: Safely executing the Python code generated by the previous node.

3. fix_data_cleaner_code: Provide automatic error correction using an LLM feedback loop.

💡 This is the minimum functional version of our workflow. It performs well as a baseline, but it is only the starting point. You can extend it with additional elements like a human-in-the-loop stage or a more advanced planning node.

Alright, let’s see how the core logic works 👇

LLM-powered code generation node

For our purposes, this is the “reasoning“ node, the component that decides the most appropriate way to clean the data. Behind the scenes, it calls an LLM to generate Python code based on instructions and dataset characteristics.

The prompt is handled using the PromptTemplate method by LangChain, which is great for defining a prompt with placeholders and filling them before sending it to an LLM. Here are the main parts of the prompt:

• Main instructions: Define the cleaning policies the model should follow, describing how to handle common issues such as missing values or duplicates when they appear in the dataset.

• User instructions: Free-form directions provided by the user that allow the agent to customize the cleaning logic based on specific needs or preferences.

• Dataset summary: A structured overview of the dataframe, including column names, data types, and missing-value counts, which helps the model generate context-aware code.

• Python code format: A template that forces the model to return valid, executable Python wrapped inside a single function so the workflow can run reliably end-to-end.

Now, let’s see what happens once we generate code!

Want to learn how to build this and similar AI workflows?

Most data scientists know how to prompt AI tools. Very few know how to turn AI into real workflows that 10x their analysis, amplify their impact, and future-proof their career.

This bootcamp is designed with you in mind.

For example, during week 3, you not only get a proper introduction to LangGraph, but also learn how to build a Data Cleaning agent and deploy it using Docker.

Every week, you build a project, and during the last week, you build a talk-to-your-data Slackbot that even your stakeholders can use immediately.

📌 Enrollment for the upcoming cohort is now open. Save your spot before it fills up: futureproofds.com (Use coupon code JAN26–25-ANDRES for 25% off)

Executing LLM-generated code

Once we get the data cleaning Python code from the LLM, we need to execute it in an isolated namespace and catch any errors so that we can utilize the retry logic.

Here are the key parts of this code:

• Defines the function: We use the built-in Python function exec() to parse the LLM-generated code string and create the function definition in an isolated namespace.

• Retrieves function: Using local_vars.get(), we extract the newly defined function object from the namespace so it can be called.

• Executes the function: agent_function(df) runs the generated cleaning function on the actual data, returning either a cleaned dataframe or raising an error.

Let’s see what happens if the function call raises an error…

Self-correcting with error feedback

If the generated code fails, the agent automatically feeds the error message back into the LLM. This allows the model to diagnose the failure and produce a corrected version of the function without manual intervention.

• Define prompt: prompt_template.format() creates the prompt by inserting the broken code and error message into a template that instructs the LLM to fix the issue.

• Fix code: llm.invoke() sends the prompt to the LLM to generate corrected code, which is then parsed to extract clean Python from the response.

This feedback loop gives the agent the ability to iteratively recover from common errors and continue improving the generated code.

Usage example

Here is a simple example showing how to run the agent programmatically.

This pattern works well if you want to integrate the workflow directly into your existing codebase or automate it as part of a larger pipeline.

You can also build an interface on top of this, for example, using Streamlit as in the earlier demo. But the best approach depends on how you plan to use the agent.

For example, if your goal is to call it during EDA or iterative analysis, you can package the agent as a module and import it directly into a notebook. This allows you to generate cleaning functions, inspect them, and reuse them across different datasets without leaving your analysis environment.

If you want to play around with the code, here is the GitHub repo.

Final thoughts

What this agent demonstrates is not just a faster way to clean data, but a glimpse of a different way of working. We are moving from writing every transformation by hand to designing systems that generate, fix, and execute the code for us.

And when even a lightweight agent can remove hours of friction, it becomes clear how much of our time can be reclaimed for the parts of the job that actually require judgment, analysis, and creativity.

This is just the beginning, though. We can leverage AI for much more than just cleaning messy data!

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope you found this guide valuable in showing how AI can be leveraged in one of the most time-consuming parts of our workflows.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

While the spotlight for the past few years has mostly been on LLMs and diffusion models, there’s been a quiet shift happening in core data science workflows, too.

Let me break it all down for you in easy terms and help you understand what the road ahead looks like.

So, what are foundation models?

Think about traditional ML models, where each model is custom-built for specific tasks. Foundation models, on the contrary, are trained on massive, varied datasets so they can generalize.

You don’t start from scratch.

We’ve seen this change happen in NLP and computer vision. Now it’s spreading to core data science areas like time series, tabular prediction, anomaly detection, and recommendations.

Why does this matter for data scientists?

Because it shifts the way we build solutions.

Instead of training a separate model from scratch for each use case, these new models give us a starting point that’s already been trained on billions of datapoints across domains.

That means less fiddling with feature engineering, faster experimentation, and sometimes even better performance with fewer labeled samples.

Let me give you a concrete example…

Use Case: Netflix foundation model for recommendations

Earlier this year, Netflix shared how it is consolidating its recommendation stack with a foundation model trained on hundreds of billions of user interactions.

Traditionally, Netflix maintained many specialized models, each tuned for a specific surface: “Top Picks,” “Continue Watching,” “Trending Now,” and more. These models often used overlapping data but couldn’t share learnings easily, which slowed iteration and made scaling complex.

With their foundation model, Netflix takes a unified approach:

• It learns user preferences across the full platform using a self-supervised next-token prediction setup, similar to GPT.

• The model encodes entire interaction histories as sequences, using a custom tokenization process that preserves information like watch duration, device type, time of day, and more.

• Sparse attention and sliding window sampling allow it to handle long user histories while keeping inference fast enough for real-time use.

Now, every product team can:

• Generate user and content embeddings that encode long-term behavior and intent.

• Use those embeddings for things like title-to-title similarity, personalized search, or candidate generation.

• Fine-tune smaller heads for specific objectives (like ranking “Because you watched…”), without retraining the entire model.

This isn’t just about model consolidation, it also solves hard problems like cold-start. By mixing metadata-based and ID-based embeddings through an attention mechanism, the system can meaningfully represent new titles, even before anyone watches them.

Foundation models across domains (as of December 2025)

Here’s a quick overview of some notable models available today, at least the ones I was able to find:

What the current landscape tells us

From what I’ve found, most traction is happening in time series and sequential data. That makes sense since forecasting is painful to get right and even harder to scale across hundreds of products, SKUs, or sensors. Foundation models ease that pain.

Other than that:

• Tabular data is still underrepresented, but TabPFN is a promising start.

• Recommendation systems are starting to get foundation-level treatment too, especially at large tech companies with massive interaction logs.

What’s missing? Broad adoption. Many of these models are still in early-access APIs or internal use only. But the momentum is clear.

Final thoughts

You don’t need to bet your entire pipeline on these models, unless you’ve got the perfect use case, but they’re absolutely worth getting familiar with:

• Test TimeGPT or TimesFM via their APIs or SDKs

• Try TabPFN on a small tabular dataset

• Follow open-source projects like MOMENT or Moirai for broader use cases

Foundation models are moving fast. Staying close to them now will pay off later.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this gave you a clearer view of modern solutions to core data science problems.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

That’s exactly what Jupyter AI offers: a native extension that lets you write, debug, and generate code directly inside your notebook with support for most models like GPT and Claude.

In this quick guide, I’ll show you what it can do, how to set it up, and a few thoughts on when it’s actually worth using.

Give me 5 minutes, and I’ll get you up and running!

Quick overview

Jupyter AI is an open-source extension that adds LLM features on top of the standard Project Jupyter ecosystem

As of the time of writing this article, there are two ways to leverage these features:

1. The %%ai “magic” command: This allows you to prompt LLMs from inside notebook cells to generate code, explain code, fix bugs, or answer questions.

2. Jupyternaut: This is their built-in chat UI (only available via JupyterLab) that lets you treat the model as a conversational assistant. You can feed pieces of your notebook or data and ask for suggestions, code generation, summarization, etc.

Now let me show how to get started using it…

1. Installing Jupyter AI

There are two ways to install JupyterAI, depending on your preferences. Let me show you how to install them using the dependency manager pip.

Quick install (with full features)

pip install jupyter-ai[all]

This command installs both the chat extension for JupyterLab and the magic-command support.

Magic-only install (if you don’t use JupyterLab or just want inline commands)

pip install jupyter-ai-magics[all]

With this, you won’t get the chat UI, but you’ll still get the %%ai magic.

Requirements

• Python 3.9 or higher. (PyPI)

• If you want the JupyterLab UI, use JupyterLab 4.x. (Older JupyterLab 3 is supported in legacy mode but not recommended.)

Once the package is installed, you can launch JupyterLab (or Notebook/VS Code if you only installed magics). If you installed the full extension, you’ll see a small chat icon in the sidebar.

2. Connect Your Model

Before you start asking the LLM anything, you need to give it a model provider. Jupyter AI supports many: OpenAI, Anthropic, AI21, Hugging Face, Mistral, SageMaker, and more.

If you are using Jupyter Lab, you can set it via the user interface:

Otherwise, here’s what you’ll typically do via terminal if you want to expose your keys to your environment. For example, here is what that would look like when using OpenAI’s LLMs:

export OPENAI_API_KEY=”your_api_key_here”

Here is another method in case you want to keep the key contained within the notebook in a safe way:

import getpass
import os
secret_access_key = getpass.getpass(’Enter your OPENAI Key: ‘)

# Set the environment variable without displaying the full key
os.environ[’OPENAI_API_KEY’] = secret_access_key

3. Using Jupyternaut

If you installed the full extension, you’ll see a chat panel in JupyterLab.

You can ask it to:

• Explain what a code cell does

• Write new functions for you

• Debug a piece of code that throws an error

• Summarise large outputs or notebook sections

Here is a quick example:

4. Use the Magic Command %%ai

Want help inside a notebook cell directly? Use the magic command. It works in JupyterLab, Notebook, VS Code, and other IPython environments

This is what you do, specifying your model of choice, in this case, ChatGPT:

%%ai chatgpt
Explain poisson distributions

If you are asking for code and want to ensure it is directly executable:

%%ai chatgpt --format code
Write a pandas query that filters customers who signed up in the last 30 days and returns their average spend.

Here is a quick example:

5. My Take

I’ve tried a handful of AI coding assistants, and the reason Jupyter AI stands out is that it integrates directly where most data scientists actually work.

But I still think the features are too basic, and the integration needs more work to make it truly valuable.

For example, something extremely useful would be if Jupyternaut could actually scan your notebook and help you make changes or explain parts of it without requiring you to copy and paste context within every prompt:

Nevertheless, if you live in notebooks and want an AI coding companion, this is definitely worth trying.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this guide helps you streamline your analysis workflow.

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!

But there is a huge shift happening behind the scenes — one that will shape the future of data analytics and artificial intelligence.

It’s what many are starting to call Agentic Analytics.

You see, in 2026, we’ll see big changes in how data professionals work and all because of a little something called a “Semantic Layer“ (and shout out to Ontologies)

If you don’t know what any of those terms mean, don’t worry, you are in the right place. I plan to break it all down for you.

Just know this: If you work in data and analytics, especially as a data scientist, this isn’t something to ignore. You have a huge opportunity on your hands to get ahead of the curve and actually shape how the next generation of analysts will work.

Here is what we’ll cover

• What exactly do we mean by Agentic Analytics?

• The Semantic Layer

• Where is the industry headed? The OSI promise

• My humble predictions for 2026

What exactly do we mean by Agentic Analytics?

I think it’s probably best we start by unpacking this term Agentic Analytics.

And no, it’s not just another buzzword; it actually represents a natural evolution of how analytics is done.

You see, for years, companies tried to solve the problem of data access through self-serve analytics. Dashboards, BI tools, and data catalogs were all great ideas, and to be fair, they worked to some extent.

Features like Looker’s “Explore from here” gave non-technical teams more autonomy than ever before.

But even with all that progress, most teams still face the same bottlenecks:

• Business users depend on data teams for every new metric or definition.

• Analysts spend time jumping between tools, context, and dashboards.

• Insights still take too long to reach decision-makers.

The promise of “self-serve” analytics became more about self-serve dashboards than self-serve insights.

That’s where AI agents change the equation.

They can interpret intent, access data directly, and generate insights on demand, not in days or weeks, but in real-time.

And that’s exactly what the term Agentic Analytics captures: a new generation of analytics where intelligent systems do the heavy lifting, and humans focus on asking the right questions.

That all sounds great, but…

Yeah, I know, I’m thinking it and so are you: It can’t all be that easy, right?

You are right, and it’s not, there is one big challenge…

Even if AI agents can access your data, they still need to understand it. And that’s where most systems fall short.

Your database knows what tables and columns exist, but it doesn’t know what “active user,” “churn,” or “retention rate” actually mean for your business.

That knowledge lives in people’s heads, scattered documentation, or buried inside Slack threads that no one will ever find again.

In other words, your data has structure, but your meaning doesn’t.

Enter: The Semantic Layer

A semantic layer creates a shared understanding of how data connects to business concepts.

It defines things like metrics, relationships, and entities in a consistent, machine-readable way.

What you’ll often find is that this layer is made up YAML files containing a structured definition of your tables and views. Here is a generic example:

So instead of an AI agent guessing what “daily active users” means, it can look it up and query the right data (with the correct metric calculations) automatically.

A brief history lesson

I need to do this short segue because I know many of you who’ve been in the data space for a while are probably screaming at your screens, saying: This isn’t anything new!

And you’d be right!

We’re talking about this now because these concepts started spiking in popularity again in early 2025 for a few reasons: mainly the rise of AI agents, context-aware systems, and the push toward standardization through initiatives like OSI (more about this in the next section).

But the idea of the semantic layer in data analytics has actually been around since the 1990s. It first appeared in enterprise data warehouses as a way to translate raw tables into business-friendly views, something business users could actually understand.

Looker later revived this concept in a modern way with LookML, a true modeling layer that defined metrics, joins, and relationships in code. That approach became the foundation for how many teams think about semantic layers today.

What’s different now is that the same concept is being extended beyond BI. Instead of only powering dashboards, semantic layers are becoming the backbone of AI agents, letting them reason about data using shared business definitions.

Where is the industry headed? The OSI promise

If there’s one thing history teaches us, it’s that every major shift in data has come from standardization.

The modern data stack gave us standard ways to store and transform data. Now, we’re moving toward standard ways to describe and share meaning across tools.

That’s exactly what’s happening right now with the Open Semantic Interface (OSI) initiative.

In September 2025, many of the biggest names in analytics and AI came together to announce a shared goal: creating an open standard for how systems define and communicate business context.

We’re talking about companies like dbt Labs, Snowflake, Google, Cube, Transform, and others, all aligning behind a common specification.

The idea is simple: instead of every platform having its own definition of metrics and entities, the OSI provides a single shared interface that any tool or AI agent can understand.

If it works, the implications are huge:

• Any AI tool (like Cursor or Claude) could plug into your company’s semantic layer.

• Metric definitions become reusable across dashboards, notebooks, and agents.

• Context becomes portable instead of something that gets lost every time you switch tools.

If the idea of keeping your business context vendor-agnostic resonates with you, this should give you real hope for the future.

My humble predictions for 2026

Here’s where I think things are headed:

1. The commoditization of the semantic layer

   By the end of 2026, the idea of a semantic layer will start moving from cutting-edge to expected. It won’t be everywhere yet, but more companies will begin adopting it as part of their modern data stack, whether through open standards like OSI or proprietary versions built into existing analytics and AI tools.

2. Agentic Analytics in practice

   We’ll also see the concept of Agentic Analytics take real shape, not as isolated AI add-ons, but as part of how data work gets done day-to-day. Expect more “talk to your data” capabilities across both lightweight integrations and enterprise platforms.

Here’s my recommendation to data scientists: be part of that design process.

You already understand how data connects to business value, which makes you uniquely positioned to shape this new layer of intelligence. Don’t wait for vendors or AI teams to define it for you.

Experiment, build prototypes, and think about how context, governance, and reasoning can live inside the tools you already use.

This is why learning how to use the semantic layer inside real AI workflows has been a core part of my AI Workflows Bootcamp.

The 22 data scientists enrolled learned to build everything from lightweight data cleaning agents to MCP servers from scratch, custom GPTs, and their own talk-to-your-data Slackbots.

👉 If you’d like to join Cohort 2, you can add yourself to the waiting list at futureproofds.com

Closing thoughts

There is no doubt in my mind that we’re entering a new phase in analytics.

For years, data scientists have focused on building models, uncovering insights, and helping teams make better decisions. That work isn’t going away. What’s changing is how we get there and how quickly we can democratize insights.

The rise of semantic layers, OSI, and Agentic Analytics marks a shift toward systems that can understand context and reason about data more like we do.

If you’ve read this far, take it as your cue to start exploring what’s possible.
Because the data scientists who learn to work with these new systems early on will be the ones shaping what analytics looks like in the years ahead.

A couple of other great resources:

• 🚀 Ready to take the next step? Build real AI workflows and sharpen the skills that keep data scientists ahead.

• 💼 Job searching? Applio helps your resume stand out and land more interviews.

• 🤖 Struggling to keep up with AI/ML? Neural Pulse is a 5-minute, human-curated newsletter delivering the best in AI, ML, and data science.

Thank you for reading! I hope this article helps you look ahead and prepare accordingly

- Andres Vourakis

Before you go, please hit the like ❤️ button at the bottom of this email to help support me. It truly makes a difference!