Language models in production
What people were actually complaining about
Free-form feedback from customers and staff, classified by language models, so the bank saw real complaints instead of a hand-read sample.
- Employer
- Tinkoff Bank
- Period
- 2020-2023
- Stack
- Python, Language models, Text classification, Annotation pipeline, Model evaluation
The problem
Feedback reached the bank as free text: customers writing in, employees reporting what was broken from the inside. What got read was a sample, and a sample tells you what you happened to open, not what people are complaining about. Reports were built on categories whoever wrote the report had invented that week, so one report's top problem was not the next report's.
What I built
I was the product lead. I owned the problem definition, the label set the business actually cared about, and what shipped. The hard part was never the model, it was the taxonomy: a label earns its place only when independent annotators pick it the same way and somebody downstream can act on it, so we measured annotator disagreement and cut or split every label they kept fighting over. Classes drifted as products and wording changed, which meant relabeling against the drift instead of pretending the first taxonomy was permanent. The last call was the one that mattered most: which cases the model closed on its own, and which it had to hand to a person.
Result
Complaint volume by category became a number a manager could look at and argue with, instead of a quote lifted from a sample, and only the labels people acted on survived the argument. This was 2020, so these were the language models of that generation, the same transformer lineage the market now calls an LLM. The line runs straight into the current work: I quantize my own models, write my own evaluation harnesses, and run them locally instead of renting somebody's API.