Comparative SMS Spam Detection
A heavier neural classifier improves spam F1 only modestly over TF-IDF logistic regression — raising the more useful question of when that cost is justified.
A coursework investigation, written up as a technical report.
- Dataset
- Approximately 5,572 SMS messages, roughly 13% spam, with a shared stratified 20% test split.
- Evaluation
- Model selection performed on training data only; both pipelines evaluated on the identical held-out stratified test split.
Research questions
- How much does a neural sequence model actually improve over a linear bag-of-words baseline?
- Is that improvement worth its compute and interpretability cost?
- What does accuracy conceal on a 13%-positive dataset?
Spam-class F1 on the shared stratified 20% test split of roughly 5,572 messages, from a linear model over TF-IDF features.
Same test split. Spam precision 0.9793, recall 0.9530, overall accuracy 0.9910, macro F1 0.9804, ROC AUC 0.9950, PR AUC 0.9858.
The dataset and why accuracy is the wrong headline
Approximately 5,572 SMS messages, of which roughly 13% are spam.
On a dataset at that ratio, a model that predicts "ham" for everything scores 87% accuracy while catching nothing. Accuracy is therefore not the metric; spam-class F1 is, and every number below is reported on the spam class specifically.
Protocol
A single stratified 20% test split, shared identically by both pipelines. Model selection — hyperparameters, thresholds, vectoriser settings — used training data only.
Both models seeing exactly the same test set is what makes the comparison a comparison rather than two anecdotes.
The two pipelines
TF-IDF + logistic regression. Bag-of-words with inverse document frequency weighting into a linear classifier. Trains in seconds, runs anywhere, and its coefficients can be read directly — you can see which tokens push a message toward spam.
Fine-tuned GloVe BiLSTM. Pretrained word embeddings, fine-tuned, into a bidirectional LSTM. Word order is available to it, embeddings carry semantic similarity, and it costs substantially more to train and to serve.
Results
| Metric | TF-IDF + LR | GloVe BiLSTM |
|---|---|---|
| Spam F1 | 0.9488 | 0.9660 |
| Spam precision | — | 0.9793 |
| Spam recall | — | 0.9530 |
| Accuracy | — | 0.9910 |
| Macro F1 | — | 0.9804 |
| ROC AUC | — | 0.9950 |
| PR AUC | — | 0.9858 |
Which figure this dossier quotes
The report's results table gives the neural spam F1 as 0.9660, and its later prose rounds differently at approximately 0.956. The results table is the primary record of the experiment, so 0.9660 is the figure used throughout.
Worth noting because the distance between those two candidate values is comparable to the distance between the neural model and the baseline — which is exactly why the comparison below is framed around cost rather than around a decimal place.
Interpretation
The BiLSTM wins by roughly 1.7 points of spam F1.
That is a real improvement, and it is smaller than the framing "neural model versus bag-of-words" would lead anyone to expect. So the interesting question is not which model is better; it is what the 1.7 points cost:
- Training time and hardware, versus seconds on a laptop
- Inference cost per message at SMS volume
- An embedding matrix to ship and version
- Loss of direct interpretability — you cannot read a coefficient off an LSTM and explain to a user why their message was flagged
For a spam filter, being able to explain a false positive to an annoyed user has genuine operational value. Whether 1.7 points is worth giving that up is a product decision, and it depends on volume and on the cost asymmetry between a missed spam and a blocked legitimate message.
The honest conclusion is that the baseline is closer to the frontier than it is usually given credit for.
Scope
- One well-studied dataset, chosen so the comparison is reproducible and comparable.
- SMS spam has its own vocabulary and length distribution, distinct from email or social.
- A single shared stratified split, applied identically to both pipelines.
- The compute-cost argument above is reasoned from the architectures rather than measured.
Where I would take this next
- Repeated stratified splits with confidence intervals, so a 1.7-point gap can be judged against its own variability.
- A cost-sensitive evaluation reflecting that a blocked legitimate message costs more than a delivered spam.
- Measured inference cost, putting numbers behind the trade-off argument above.
Original artifacts
- REPORTOriginal report (PDF)
Notes on evidence
- Figures are taken from the report's results table, which is the primary record of the experiment.
- A coursework investigation, written up as a technical report.