Small models on device: where they already pay off

Field technicians worked in remote locations with extremely unreliable cellular coverage that would drop for hours at a time unpredictably. The application needed classification, entity extraction, and data structuring functionality without relying on constant connectivity to central servers during their work.

Running the latest large language models on remote servers meant the app feature became completely useless the instant the technician lost cellular coverage in the field. We began testing smaller quantised language models that could run directly on mobile devices without any network dependency.

Classification accuracy remained acceptable for the narrow use cases in question. Short entity extraction worked adequately. The model quality proved sufficient because each task was narrow in scope and the output format was rigidly defined by the underlying systems and business requirements.

For this client facing serious connectivity constraints and operational challenges, moving inference from cloud servers to the local device transformed a broken feature into genuinely usable functionality that worked reliably. Latency dropped to negligible levels and the app functioned perfectly during coverage outages.

Operational cost shifted away from cloud API billing models to local device memory consumption and the support burden for deploying model updates to field devices. The cost became predictable because it no longer scales with query volume or varies with usage intensity patterns.

Long-form reasoning still requires server connectivity for now, but that remains acceptable since narrow classification and extraction tasks account for the vast majority of high-volume queries in the system.