部署

Inference Cost

定义

The compute and financial cost of running a model to produce a single prediction or generated response. Inference cost is often the dominant AI operational expenditure at scale and is managed through model compression, caching, quantization, and batching strategies.

相关术语

Token Budget

The total number of tokens allocated for a model request, encompassing both input (prompt + context) and output. Managing token budgets is central to controlling inference cost in production LLM applications, especially when processing long documents or maintaining conversational history.

Model Compression

A set of techniques—including quantization, distillation, pruning, and low-rank factorisation—that reduce model size and computational requirements while preserving performance. Model compression is essential for deploying powerful models on edge hardware or within cost budgets.

Quantization

A model compression technique that reduces the numerical precision of model weights—for example, from 32-bit floats to 8-bit integers—shrinking memory requirements and accelerating inference with minimal accuracy loss. Quantization is essential for deploying LLMs on-premise or at the edge.

了解术语只是第一步，将其落地应用才是第二步。

预约一次 Physical AI 适配性沟通，探讨这些 AI 概念如何转化到您所在的具体行业与业务挑战中。