Topic Classification of Reuters Newswires

The problem

46 topic classes, single-label, with pronounced imbalance. Each newswire is represented as a 10,000-dimensional multi-hot vector — which discards word order and term frequency, and is the assignment's constraint rather than a design choice.

Why macro F1 leads

The metric hierarchy is led by macro F1, not accuracy, and this decision determines everything that follows.

Macro F1 weights every class equally. Accuracy weights every sample equally, which on an imbalanced corpus means it is dominated by a handful of common topics. A model can look strong on accuracy while being nearly useless on most of the classes it is nominally predicting.

The floor makes this concrete: uniform chance is 0.0217 accuracy, while a majority-class baseline reaches 0.3517 validation accuracy at 0.0113 macro F1. Thirty-five percent accuracy from predicting one class forever.

Protocol

The Keras test split is untouched during development. All selection uses one reproducible 80:20 stratified development-and-validation split.

Experiments vary one principal factor per phase, holding everything else fixed: capacity, then dropout, then vocabulary size. 53 model fits in total.

This is slower than a random or grid search and it answers a different question. A sweep tells you which configuration scored highest. A one-factor-at-a-time protocol tells you what each factor does — which is the difference between a result and an explanation.

The two leading configurations were then repeated across five training seeds and five independent stratified splits, separating variation caused by stochastic training from variation caused by the partition.

Results

Selected configuration: 128 and 128 hidden units, dropout 0.0, 10,000-word vocabulary, nine epochs, trained on all training data.

The finding

0.8032 accuracy. 0.5537 macro F1. Balanced accuracy 0.5134.

That gap is the result. Four fifths of newswires are classified correctly, and averaged over the 46 topics the model is barely better than a coin flip. The common classes are handled well; the rare ones are not handled at all.

Reporting 80% alone would tell only half the story. Balanced accuracy of 0.5134 is the number a practitioner needs in order to judge a 46-class tagger, and surfacing it is the point of leading the metric hierarchy with macro F1.

Negative finding: dropout hurt, monotonically

Raising the dropout rate lowered validation macro F1 at every step — from 0.5298 at rate 0.0 down to 0.3792 at rate 0.5 — while narrowing the train-validation accuracy gap from 0.1245 to 0.0646.

Dropout did exactly what it is supposed to do, and the model got worse. The regularisation successfully reduced overfitting and cost more in capacity for minority classes than it recovered in generalisation. The selected configuration therefore uses no dropout, which is not the conventional answer.

Uncertainty

Bootstrap intervals accompany the headline figures rather than bare point estimates. The report gives more than one macro-F1 interval — a plain bootstrap and a stratified bootstrap — so any quotation of an interval has to say which one it is.

Scope

• Multi-hot representation, which discards order and frequency by design and makes the capacity and vocabulary experiments clean.
• Dense layers only, per the study scope.
• Minority classes carry few examples, which is the source of the macro-F1 gap the study set out to expose.
• One well-studied corpus, chosen so the protocol is reproducible.

Where I would take this next

• Class-weighted loss or resampling aimed squarely at macro F1 rather than accuracy.
• A sequence representation, to measure what the order-free encoding costs.
• Per-class error analysis on the weakest topics, separating "too few examples" from "genuinely overlapping" — they call for different remedies.
• Hierarchical classification, if the taxonomy has structure worth exploiting.