Model serving connects trained models with production applications. When an application sends input data to a serving system, the system validates the request, applies required preprocessing, executes the model, and returns the prediction. This prediction process is called inference. Serving can be implemented through REST APIs, gRPC services, batch processing systems, message queues, or embedded runtimes. Online serving handles individual or small groups of requests and often prioritizes low latency. Batch inference processes larger collections of data together and is useful for scheduled predictions. Streaming inference processes continuously arriving events. Important serving considerations include throughput, latency, concurrency, memory usage, accelerator utilization, request validation, and failure handling. A serving system may use model replicas, load balancing, caching, and autoscaling to handle changing demand. Model optimization techniques such as quantization, pruning, and compilation can reduce inference cost, although they may affect accuracy. Production systems should also record appropriate operational metrics while protecting sensitive input data. Model serving is therefore an infrastructure concern, whereas inference is the computation performed using the model.
Model Serving and Inference
Model serving is the process of making a trained machine learning model accessible to applications, while inference is the process of using that model to generate predictions from new input data.
Throughput = Number of completed requests / Time interval
Explanation
Variables
| Symbol | Meaning |
|---|---|
Throughput | Number of completed prediction requests per unit of time |
Number of completed requests | Total successful requests processed during the measurement interval |
Time interval | Duration over which throughput is measured |
Example or application
An image classification API receives an image from a mobile application. The serving system validates and preprocesses the image, runs the trained model, and returns the predicted class and confidence scores.
Text
Inference is the model's prediction operation, and serving provides the infrastructure through which applications request and receive those predictions. Serving systems manage requests, model execution, and responses.
Key Points
Inference uses a trained model to produce predictions.
Model serving provides the infrastructure for accessing predictions.
Online, batch, and streaming inference serve different workloads.
Latency and throughput are important serving metrics.
Load balancing and autoscaling help manage demand.
Model optimization can reduce inference cost but may affect accuracy.
Applications
Image classification APIs
Recommendation engines
Real-time fraud detection
Chatbot inference services
Batch forecasting systems
Industrial monitoring
Learning Outcomes
Distinguish model serving from inference.
Describe a typical prediction request lifecycle.
Compare online, batch, and streaming inference.
Define latency and throughput.
Identify common model-serving optimization techniques.
Practice Questions
What is model serving?
What is inference?
How does online inference differ from batch inference?
What does throughput measure?
How can model replicas improve serving capacity?
Related Concepts
Model Deployment
REST APIs
Inference Optimization
Load Balancing
Autoscaling
Quantization
Common Mistakes
Treating serving and inference as identical concepts.
Assuming high throughput always means low latency.
Ignoring preprocessing during production inference.
Measuring performance only with model accuracy.
Optimizing model size without checking prediction quality.
Sources and verification
Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.
- Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly Media.
- Crankshaw, D. et al. (2017). Clipper: A Low-Latency Online Prediction Serving System. USENIX NSDI.
- BentoML Documentation. Model Serving and Deployment.
Suggest a correction or source · More in Artificial Intelligence & Machine Learning