BERT in production: notes from the first attempt

The quality jump was real and measurable. BERT fine-tuned on ticket classifications improved accuracy by nine points over our previous classifier. That was enough to make the test set look very promising and exciting. The problem appeared when we started costing out how to actually serve it in production.

A BERT model running inference requires a GPU to work reliably. GPUs are not cheap to keep idle or running, and they are definitely not cheap to keep always-on for production systems. The inference cost multiplied the baseline infrastructure bill by six. Nobody had included that line item in the budget.

We looked at distillation because we had to make it work cost-effectively. The idea is to train a smaller model to mimic the behavior of the larger one. We built a much smaller model and fine-tuned it against the BERT outputs rather than the original labels.

The smaller model lost two percentage points of accuracy in testing. In absolute terms, that mattered less than you would think initially. It was good enough for the business threshold. More importantly, it ran on CPU infrastructure. The infrastructure cost dropped back to something reasonable.

The client approved the trade immediately and without hesitation, even though it meant signing up for lower accuracy numbers than the larger model. They were being realistic about what they could afford to run, day after day, year after year, without blowing through their infrastructure budget.

The permanent lesson we still apply: the model that actually runs in production is almost never the one that wins the research evaluation. Find the boundary where good enough meets affordable, and build your production system there with pragmatism.