When classical ML beats an LLM

Two different tools

Classical machine learning means regression, decision trees, gradient boosting (XGBoost, LightGBM, CatBoost) and small neural networks. Such a model learns one task from your historical data, in which you already know the outcome. It gets, for example, 50 thousand orders, some of which ended in a return, and learns to predict the return risk of a new order. Beyond that data it knows nothing, but at its one task it is often very good.

A language model arrives with general knowledge of language and the world. We describe the task to it in words, with no training. That is a huge advantage when there is no data, when the task keeps changing, or when the input is text that cannot be squeezed into columns.

If you have historical data with the answer and the task is stable, classical ML should be at least the baseline.

Seven situations where we bet on classical ML

1. The data is in a table

Demand forecast, churn risk, lead scoring, pricing, fraud detection: all of these are rows and columns. In large comparisons across dozens of tabular datasets, tree-based models beat deep learning on mid-sized datasets of around 10 thousand rows.

Language models cope with tables mainly when there are very few examples, on the order of a handful. As the data grows, gradient boosting catches up with them, and it computes in a fraction of the time. On small datasets, specialized tabular foundation models can beat trees, but those are models built for numbers, not chatbots, and on larger datasets trees are still the safer choice.

2. You are forecasting numbers over time

Sales next week, occupancy in December, energy consumption. It is tempting to feed the history to a language model. Studies of popular LLM-based forecasting methods showed, however, that removing the language model from them did not worsen the results, and often improved them. Time series have their own models, trained on numbers rather than text.

3. There are many decisions and little time

A model from the gradient boosting family computes one prediction in a fraction of a millisecond. Calling a language model through an API takes hundreds of milliseconds or seconds, plus the network and dependence on a provider.

Cost itself is rarely the main argument today. Small language models from the big providers cost about 0.10 dollars per million input tokens, so a million decisions with a thousand-token prompt come to about 100 dollars plus the outputs. Latency, repeatability and accuracy usually matter more.

4. You need a probability that means something

Many product decisions rest on a threshold: a payment with a risk above 0.8 is held, a lead scored above 0.6 goes straight to a salesperson. For such a threshold to be set sensibly, the number from the model must be stable and verifiable on your data, and ideally calibrated: 0.8 should mean roughly eight out of ten.

A language model asked about its confidence usually just declares it, and studies show it tends to be overconfident. A classical classifier returns a number that can be measured and calibrated on your own data with simple, well-known methods.

5. The same case must give the same result

A trained classical model is an ordinary function: the same input, the same output, today and a year from now. With language models this is not a given, even at temperature 0. In experiments, the same prompt sent a thousand times at temperature 0 produced dozens of different answers. The cause lies in how the server computes: the result depends on how many other requests are being processed at the same moment.

You can work around it, but you have to know about it. When a decision goes to an audit, to regression tests or into a dispute with a client, “it came out differently the second time” is a very bad answer.

6. Someone will ask “why?”

Credit scoring of individuals is a high-risk system under the EU AI Act, and the requirements for such systems apply from 2 December 2027. A person affected by a decision has the right to a clear explanation of the role of the AI system and the main elements of the decision. GDPR additionally protects against decisions based solely on automated processing when they have legal effects or similarly significantly affect the person. In such projects, explainability is a requirement, not an add-on.

A tree model with the SHAP method shows how much each feature contributed to a specific decision. A language model will also write a justification, but it is text written on the side, not a record of how the decision was made. Studies have shown that such explanations can systematically omit the real reason for the model's answer.

7. You have a few hundred labeled examples and a fixed task, even a text one

This is the least obvious point, because it concerns text, the domain of language models. In a 2024 comparison, small fine-tuned models beat large models prompted without examples on all the text classification tasks tested, very clearly on some of them. Results stopped improving at just 200-500 labeled examples, so you do not need thousands of data points.

Today's large models would probably do better. The direction is clear, though: if the task has your own narrow definition (for example “is this a complaint under our terms”), a few hundred labeled examples and a small model can beat the general knowledge of the largest model.

When an LLM is the better choice

A language model wins when:

• the input is unstructured and varied: emails, contracts, scans, flyers, conversations;

• there is no labeled data or collecting it would cost more than the whole project;

• the task changes often, and retraining a model with every change makes no sense;

• the output is text: a reply to a customer, a summary, a conversation.

Best together

• An LLM labels the data, a small model learns from it. A language model can label texts more cheaply than crowdsourced workers, and in some tasks more accurately too. A small model trained on such data is then often better than a large one prompted with a few examples.

• An LLM as a source of features. A language model reads the content of a complaint and returns its category and tone. These values go as ordinary columns into a tabular model that knows the rest of the customer's history.

• A cascade. A cheap, fast model settles the certain cases, and the uncertain ones go to a language model or to a human. When most cases are simple, most of the traffic is handled by the cheap model, and the expensive one works only where it is needed.

Four questions before you choose

• Do you have historical data with the answer, even a few hundred cases? If so, start with a simple classical model and treat it as the bar for the LLM.

• Is the data in a table, or in text, an image or sound? A table is the natural territory of classical ML.

• How many decisions, and in what time? Millions of decisions in milliseconds is a job for a small model close to your data, not for an external API.

• Will someone have to explain the decision to a client, an auditor or a court?

A language model is a great tool, but not the only one. The most expensive mistake in AI projects is often not choosing the wrong model, but failing to compare it with a simpler one.