When Close Isn't Close Enough

Research questions

Three, and the interesting answer is to the first one.

1. Does the choice of distance metric meaningfully change k-NN performance?
2. How much does feature scaling matter, and equally across datasets?
3. Where does PCA help, and where does it stop?

Implementation

k-NN and the evaluation metrics were implemented from scratch in NumPy rather than called from a library. The implementation was then validated against scikit-learn, reaching roughly 0.9987 mean prediction agreement across the checked settings — high enough to trust the results, and the small residual disagreement is tie-breaking behaviour at equal distances.

Leakage safeguards

This is the part that makes the rest trustworthy. Scaling and PCA are fitted inside every validation fold, never once over the whole dataset.

Fitting a scaler on all the data before cross-validating is one of the most common ways to produce a good number that means nothing: the validation fold has already influenced the transform applied to it. Doing it correctly costs a little accuracy on paper and buys a result that survives contact with a test set.

Tuning was kept entirely separate from a single final evaluation on data untouched until selection was finished.

Results

Wine — Euclidean, k=5, six PCA components: 0.975 test accuracy, 0.975 macro F1.

Digits — Euclidean, k=5, all 64 components retained: 0.967 test accuracy, 0.967 macro F1.

Findings

Scaling matters enormously on Wine. Its features are on wildly different numeric scales, and unscaled Euclidean distance is dominated by whichever feature happens to have the largest range. This is not a subtle effect.

PCA helps Digits early, then plateaus. Efficiency improves substantially in the first components, and accuracy stops improving well before the component count is exhausted. The final selection nonetheless retains all 64 components, because the plateau is flat rather than declining and there was no accuracy reason to truncate.

Negative finding: cosine did not win

Cosine similarity is frequently recommended for high-dimensional data almost as a reflex. It did not win here.

More importantly, the differences between Euclidean and Manhattan stayed within the variability of the evaluation itself. The honest conclusion is not "Euclidean is best" — it is that on these datasets, at this scale, distance-metric choice is not where the performance lives. Scaling and honest validation are.

Reporting a distance-metric winner from differences this small would be reading noise.

Scope

• Two small, clean, well-characterised datasets, chosen so the comparison is reproducible.
• PCA is linear, so the dimensionality results speak to linear structure.
• The study measures accuracy rather than prediction-time cost.
• Tie-breaking at equal distances accounts for the small residual difference from scikit-learn's predictions.

Where I would take this next

• A genuinely high-dimensional dataset, where the curse of dimensionality is sharp enough for distance metrics to separate.
• Confidence intervals on the metric comparison, which would show the "within variability" conclusion rather than argue it.