✦ScienexySCIENCE KNOWLEDGE LIBRARY
Science reference · SCI-CS-003-163

Model Serving and Inference

Model serving is the process of making a trained machine learning model accessible to applications, while inference is the process of using that model to generate predictions from new input data.

Throughput = Number of completed requests / Time interval

Explanation

Model serving connects trained models with production applications. When an application sends input data to a serving system, the system validates the request, applies required preprocessing, executes the model, and returns the prediction. This prediction process is called inference. Serving can be implemented through REST APIs, gRPC services, batch processing systems, message queues, or embedded runtimes. Online serving handles individual or small groups of requests and often prioritizes low latency. Batch inference processes larger collections of data together and is useful for scheduled predictions. Streaming inference processes continuously arriving events. Important serving considerations include throughput, latency, concurrency, memory usage, accelerator utilization, request validation, and failure handling. A serving system may use model replicas, load balancing, caching, and autoscaling to handle changing demand. Model optimization techniques such as quantization, pruning, and compilation can reduce inference cost, although they may affect accuracy. Production systems should also record appropriate operational metrics while protecting sensitive input data. Model serving is therefore an infrastructure concern, whereas inference is the computation performed using the model.

Variables

SymbolMeaning

Throughput

Number of completed prediction requests per unit of time

Number of completed requests

Total successful requests processed during the measurement interval

Time interval

Duration over which throughput is measured

Example or application

An image classification API receives an image from a mobile application. The serving system validates and preprocesses the image, runs the trained model, and returns the predicted class and confidence scores.

Text

Inference is the model's prediction operation, and serving provides the infrastructure through which applications request and receive those predictions. Serving systems manage requests, model execution, and responses.

Key Points

  • Inference uses a trained model to produce predictions.

  • Model serving provides the infrastructure for accessing predictions.

  • Online, batch, and streaming inference serve different workloads.

  • Latency and throughput are important serving metrics.

  • Load balancing and autoscaling help manage demand.

  • Model optimization can reduce inference cost but may affect accuracy.

Applications

  • Image classification APIs

  • Recommendation engines

  • Real-time fraud detection

  • Chatbot inference services

  • Batch forecasting systems

  • Industrial monitoring

Learning Outcomes

  • Distinguish model serving from inference.

  • Describe a typical prediction request lifecycle.

  • Compare online, batch, and streaming inference.

  • Define latency and throughput.

  • Identify common model-serving optimization techniques.

Practice Questions

  • What is model serving?

  • What is inference?

  • How does online inference differ from batch inference?

  • What does throughput measure?

  • How can model replicas improve serving capacity?

Related Concepts

  • Model Deployment

  • REST APIs

  • Inference Optimization

  • Load Balancing

  • Autoscaling

  • Quantization

Common Mistakes

  • Treating serving and inference as identical concepts.

  • Assuming high throughput always means low latency.

  • Ignoring preprocessing during production inference.

  • Measuring performance only with model accuracy.

  • Optimizing model size without checking prediction quality.

Sources and verification

Sources are shown as provided in the project data. Not every source has yet been independently verified. Please report errors or missing citations.

  • Huyen, C. (2022). Designing Machine Learning Systems. O'Reilly Media.
  • Crankshaw, D. et al. (2017). Clipper: A Low-Latency Online Prediction Serving System. USENIX NSDI.
  • BentoML Documentation. Model Serving and Deployment.

Suggest a correction or source · More in Artificial Intelligence & Machine Learning