Model-Serving on Varchaleswari Ganugapati

Can Triton Fit It All?

Sun, 05 Jul 2026 09:00:00 -0600

The serving flow I was working on had gotten tangled in a fairly ordinary way. A single request needed features pulled from a feature store, some features computed from the user context, and a few more built directly from the incoming request. All of that fed into a model, and the model produced the prediction that actually mattered. The problem was that all of it, the feature assembly and the model inference, lived inside the same service.

The Handshake: Validating Score Parity Across Training and Serving

Sat, 16 May 2026 09:00:00 -0600

Part 1 ended with a retailer moving a purchase-prediction model from Python training into a Java serving service. The bundle was clean. The service loaded it. Scores came back.

But here is the question that decides whether the project ships or stalls: are those scores actually right?

The Python notebook says feature vector v scores 0.847. The Java service, given the same v, returns 0.846989. Close, but not identical. Should the team ship it? What if it returned 0.83? Or 0.5?

Part 1: Cross-Platform Model Bundling: Formats That Travel

Fri, 15 May 2026 09:00:00 -0600

Imagine a retailer. Their data science team trains a purchase-prediction model in Python every Sunday night on a week’s worth of clickstream data. By Monday morning, every visitor to the homepage gets a personalized product ranking. The scoring has to happen in under 50 milliseconds, from a Java microservice that handles millions of requests a day, with no Python anywhere in its stack.

The model works. The question is: how do you move it from a Python training environment into a Java production service without rewriting it, without breaking the preprocessing, and without the serving team needing to understand the training code?