ML on Varchaleswari Ganugapati

Can Triton Fit It All?

Sun, 05 Jul 2026 09:00:00 -0600

The serving flow I was working on had gotten tangled in a fairly ordinary way. A single request needed features pulled from a feature store, some features computed from the user context, and a few more built directly from the incoming request. All of that fed into a model, and the model produced the prediction that actually mattered. The problem was that all of it, the feature assembly and the model inference, lived inside the same service.

The Handshake: Validating Score Parity Across Training and Serving

Sat, 16 May 2026 09:00:00 -0600

Part 1 ended with a retailer moving a purchase-prediction model from Python training into a Java serving service. The bundle was clean. The service loaded it. Scores came back.

But here is the question that decides whether the project ships or stalls: are those scores actually right?

The Python notebook says feature vector v scores 0.847. The Java service, given the same v, returns 0.846989. Close, but not identical. Should the team ship it? What if it returned 0.83? Or 0.5?

Part 1: Cross-Platform Model Bundling: Formats That Travel

Fri, 15 May 2026 09:00:00 -0600

Imagine a retailer. Their data science team trains a purchase-prediction model in Python every Sunday night on a week’s worth of clickstream data. By Monday morning, every visitor to the homepage gets a personalized product ranking. The scoring has to happen in under 50 milliseconds, from a Java microservice that handles millions of requests a day, with no Python anywhere in its stack.

The model works. The question is: how do you move it from a Python training environment into a Java production service without rewriting it, without breaking the preprocessing, and without the serving team needing to understand the training code?

Powering Models: Feature Engineering, Part 2

Wed, 13 May 2026 10:00:00 -0600

Part 1 covered the offline side: getting features built, validated, and automated. This post picks up at the point where a model is deployed and a service needs to call it.

The question is deceptively simple: how does the model get the features it needs when a request comes in?

There are two answers, and most real systems use both depending on the feature type.

The two patterns

 flowchart LR
 classDef caller fill:#eff6ff,stroke:#3b82f6,stroke-width:1.5px,color:#1e3a5f
 classDef svc fill:#f5f3ff,stroke:#8b5cf6,stroke-width:1.5px,color:#3b0764
 classDef model fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#14532d
 classDef store fill:#fefce8,stroke:#eab308,stroke-width:2px,color:#713f12

 subgraph patternA[" Pattern A · Features in the request "]
 C1[Client / Caller]:::caller -->|"payload includes features"| SVC1[Prediction Service]:::svc
 SVC1 --> M1[Model]:::model
 M1 -->|prediction| C1
 end

 subgraph patternB[" Pattern B · Features from online store "]
 C2[Client / Caller]:::caller -->|"entity ID only"| SVC2[Prediction Service]:::svc
 SVC2 -->|"fetch features"| FS[("Online Feature Store")]:::store
 FS -->|"feature vector"| SVC2
 SVC2 --> M2[Model]:::model
 M2 -->|prediction| C2
 end

Pattern A puts the burden on the caller. The upstream service computes or collects the features and sends them as part of the request payload.

Powering Models: Feature Engineering, Part 1

Wed, 13 May 2026 09:00:00 -0600

The moment a model ships, you start paying for every shortcut taken before it.

Most ML failures in production aren’t model failures. They’re pipeline failures. The wrong features. Features computed differently at training time versus serving time. No one owning the handoff between the data science team and the engineering team. Pipelines that worked once but weren’t designed to run again.

This post is about the offline half of the ML lifecycle: how features get discovered, engineered, and automated into repeatable pipelines, and why the decisions made here directly determine what’s possible in production.