Tell it what to do.
Instructions and examples improve behaviour without changing model weights. Try this first; it may be enough.
READSIGIL / SMALL LANGUAGE MODELS
Small models.
Big claims.
Let's test them.
Do you need a 70-billion-parameter model to classify a support request? Probably not. Does a smaller model automatically become cheaper, private or trustworthy? Also no. Here's how to decide, build, and measure.
MYTHS, TESTEDREAL TRAINING CODEHELD-OUT RESULTS, NOT HYPE
01 / THE BASICS
What is an SLM, exactly?
A small language model is a language model with comparatively few parameters. There is no universal parameter cut-off. It still predicts tokens; being small doesn't make it rule-based, deterministic or immune to hallucination.
Give it what it needs.
RAG supplies relevant information at request time. Better for current documents and changing policies than memorising facts in weights.
Teach a repeatable behaviour.
Adapt weights using labelled examples. Most practical 'build an SLM' exercises start from an existing model, not random initialisation.
02 / MYTHBUSTERS
Ten claims you will hear in the next AI meeting.
Make your call on each one before you see the evidence. Every claim has a grain of truth. The catch is where budgets, timelines and audits go wrong.
Cost and capacity
01
“A smaller model is always cheaper and faster.”
Depends
- Why people believe it
- Fewer parameters need less memory and cost less per token.
- The catch
- At low volume, idle GPUs, serving engineers and monitoring can cost more than a hosted API. Long prompts and poor batching wipe out the speed gain.
- Test it
- Work out cost per correct output at your real volume: infrastructure, operations and the cost of errors, divided by correct outputs. Start with step 03.
02
“Open weights mean free, with no lock‑in.”
Partly true
- Why people believe it
- You can download the weights and host them yourself.
- The catch
- Many “open” models ship under custom licences with usage conditions, and few meet the Open Source Initiative’s definition. You also inherit the base model’s roadmap: a new version means retraining and revalidating.
- Test it
- Read the licence file before the benchmark. Price one full revalidation cycle.
- Source
- OSI Open Source AI Definition 1.0 ↗
03
“You need a GPU cluster and a research team to build one.”
Busted
- Why people believe it
- Training a model from scratch does need that.
- The catch
- Almost nobody trains from scratch. LoRA freezes the base model and trains a small adapter: well under 1% of the weights. This lab runs on a single GPU or a free notebook, and the script prints the exact figure.
- Test it
- Run step 04 in Colab.
- Source
- Hu et al., LoRA (2021) ↗ and Dettmers et al., QLoRA (2023) ↗
Risk and governance
04
“A small model running in-house is automatically private.”
Busted
- Why people believe it
- Local inference keeps prompts inside your boundary.
- The catch
- Logging, access and retention still decide privacy. And a fine-tuned model can memorise training examples and repeat them when prompted the right way.
- Test it
- Prompt your adapted model with the first half of a training record. See what it completes.
- Source
- Carlini et al., Extracting Training Data from Large Language Models (2021) ↗
05
“On a narrow task, it won’t hallucinate.”
Busted
- Why people believe it
- A narrow task leaves less room to go wrong.
- The catch
- It can still invent a label, force a confident answer onto a vague message, or break on phrasing it has not seen.
- Test it
- Track invalid labels and how often the model says
unknown, not accuracy alone. That is why this lab has anunknownlabel. See step 01.
06
“If we host it ourselves, where the model came from doesn’t matter.”
Busted
- Why people believe it
- The data never leaves your infrastructure.
- The catch
- Auditors and regulators look at the supply chain: who built the model, its licence, what it was trained on, the file format and the jurisdiction of origin.
- A note on this lab
- We teach with Qwen2.5 because it is small and permissively licensed. That is a choice for learning, not a production endorsement. The script is model-agnostic: pass
--modelto swap the base. - Source
- NIST AI Risk Management Framework ↗ and RBI FREE-AI Committee report (2025) ↗
07
“Smaller means easier to explain and harder to attack.”
Busted
- Why people believe it
- Fewer parameters sounds like more transparency.
- The catch
- Half a billion parameters is still opaque, and prompt injection works at any size. What an auditor can check is the system around the model: logged inputs and outputs, versioned adapters and stored test results.
- Test it
- Hide an instruction inside a test message and see whether the label changes.
- Source
- OWASP Top 10 for LLM Applications ↗
Engineering choices
08
“Fine-tune it on our policies and it will know them.”
Wrong tool
- Why people believe it
- Training changes behaviour, so it should teach facts too.
- The catch
- Fine-tuning teaches format and narrow behaviour. Facts belong in retrieval or a database, where they can be updated. Research finds that fine-tuning on facts a model does not already know makes it more likely to hallucinate.
- Source
- Lewis et al., RAG (2020) ↗ and Gekhman et al. (2024) ↗
09
“A higher benchmark score means a better model for our task.”
Busted
- Why people believe it
- On broad, open-ended work, stronger general models usually do better.
- The catch
- Public benchmarks rarely look like your task. On this page’s own 25 test messages, a classical text classifier with no language model at all scored 25 out of 25.
- Test it
- We ran it. See the result, then run
baseline_classic.pyyourself.
10
“Quantization is free compression.”
Depends
- Why people believe it
- At 8 bits, models often lose very little.
- The catch
- Losses at 4 bits depend on the task, and rare categories tend to suffer first.
- Test it
- Re-run the same 25 test messages after quantizing. Never assume the scores carry over.
- Source
- Frantar et al., GPTQ (2022) ↗
Make your call on each claim. Your tally appears here.
None of the ten holds as stated. Three hold only under conditions you would have to measure.
baseline_classic.py on this workshop’s data. No sound.We tested claim 09 on a laptop CPU.
25 of 25 held-out messages sorted correctly by a classical text classifier, trained in 0.19 seconds on the same 75 messages the workshop uses.
No GPU and no model download. The settings were fixed before the test set was scored.
A perfect score is also a warning. Twenty-five clean, single-language messages are too easy to tell models apart. Real customer messages mix languages, misspell words and ask two things at once. That is where the comparison starts to mean something.
Download the baselineTry to beat it
Read the video transcript
Claim 09: a higher benchmark score means a better model for our task. We tested it on this page’s own data: 75 training messages, 25 held-out messages, five labels. The method is a TF-IDF character n-gram classifier with logistic regression, and no language model. Command: python baseline_classic.py. Output: trained in 0.19 seconds on CPU; 25 of 25 correct; every label 5 of 5. A perfect score is a warning too: this test is too clean to separate models. Beat the baseline before you fine-tune.
03 / WHERE IT FITS
Give it work it can do. Keep the boundaries explicit.
✓ Reasonable starting points
- Route incoming tickets into known categories
- Extract fields from standardised, low-risk text
- Tag documents for human review
- Summarise short, well-defined content with verification
- Private/on-device inference when the whole deployment is governed
× Not a shortcut for
- Guaranteeing factual or legal correctness
- Keeping rapidly changing knowledge current
- Making credit or fraud decisions without controls
- Handling every language or edge case out of the box
- Eliminating the need for human review, audits or monitoring
The real decision is rules vs traditional ML vs prompted model vs retrieval vs fine-tuned model. Choose using task-level accuracy, latency, operating cost, privacy and risk—not model size alone.
04 / HANDS-ON WORKSHOP
Build a specialised model. Then try to beat the baseline.
We'll adapt an existing 0.5B-parameter model to sort fictional banking support messages into five categories. You'll have genuine training code, a separate test set and a report of what changed. Browser steps guide the build; actual fine-tuning runs in Python.
01 / DEFINE
Choose one bounded job.
Input: one fictional support message. Output: a single lowercase label. No account action, advice, or financial decision.
cardspaymentsaccountotherunknownWhy unknown? Some messages contain too little information. Forcing a confident answer hides a failure.
02 / DATA
See what the model learns from.
Training: 75 invented and labelled messages. Held-out check: 25 separately authored messages, five per category. This tiny dataset teaches the process, not production performance.
MESSAGE
My debit card was declined at the supermarket.
EXPECTED OUTPUTcards03 / BASELINE
Find out if you need to train at all.
The supplied Python script first runs the unmodified model against the 25 held-out messages. You will later run the adapted model on the exact same set. For a credible comparison, also run a classical classifier that uses no language model. On this test it scored 25 of 25, so a fine-tuned model has to justify its cost against that.
Base model→25 untouched tests→Record results
The cheapest baseline, in seconds on any laptop:
python -m pip install scikit-learn python baseline_classic.py
We don't display invented accuracy or simulated model responses here. The scores appear only when you actually run the script.
04 / TRAIN
Fine-tune a model, not a slide.
LoRA freezes the foundation weights and trains a small adapter. You can adjust rank and training epochs; the script will report trainable parameters and save the adapter.
In a terminal, from the slm directory:
python -m pip install -r requirements.txt python train.py --train data/train.jsonl --eval data/eval.jsonl --epochs 3 --rank 8
To run it, you must obtain the model weights and install the dependencies. No model training happens in this web page.
05 / EVALUATE
Did the specialised model actually help?
Check out/metrics.json and out/individual_predictions.json after the run. Enter the number of exact matches from your own results to inspect the improvement or regression.
Before considering deployment, test:
- Invalid labels and unclear messages, not accuracy alone
- Actual customer phrasing, spelling, languages and new scenarios
- Data leakage, privacy, fairness, drift and escalation controls
- Latency, throughput, hardware cost and the cost of mistakes
TAKE IT FURTHER
Reproduce it. Break it. Improve it.
The files are readable Python and JSONL, not a pretend model running inside your browser.
05 / THE DECISION
When should you build one?
Start only when the workflow is narrow, frequent, measurable and stable. If an SLM can't beat a cheaper baseline on the metrics that matter, don't deploy it just because the model is fashionable.
Clear input, clear output.
Can reviewers agree on the correct answer consistently?
Known failure costs.
Do you have held-out examples and an explicit threshold for intervention?
Governed end to end.
Can you monitor quality, protect data and roll back safely?
Go to the primary sources.
Qwen2.5 0.5B model card ↗ · PEFT / LoRA documentation ↗ · Hugging Face training guide ↗
The research behind the claims: LoRA ↗ · QLoRA ↗ · Training-data extraction ↗ · Retrieval-augmented generation ↗ · Fine-tuning and hallucination ↗ · GPTQ quantization ↗ · OWASP LLM Top 10 ↗ · NIST AI RMF ↗ · RBI FREE-AI ↗ · OSI Open Source AI Definition ↗
Claims and sources last reviewed 18 October 2026.