ZH/zhe hang
← Work index

02 ·text classifier ·2026

sorted(mathematics)

Reads an H2 Mathematics exam question and tells you which chapter it belongs to, through Machine Learning.

Facts

  • 93.1% across 21 chapters
  • 1,451 labelled questions
  • group-aware split, 0 papers leaked
  • 32 tests, mostly about leakage

Built with

  • Python
  • scikit-learn
  • PyTorch
  • Hugging Face
  • PyMuPDF

Links

No public repo yet: it goes up once the code is cleanly separated from the private question data.

sorted(mathematics) takes an H2 Mathematics exam question and returns the chapter it belongs to, out of the 21 chapters I teach.

Inspiration

Building a revision package is two jobs. The interesting one is deciding which questions a student actually needs. The tedious one comes first: sorting a stack of past-year papers into chapters, question by question, before you can start choosing anything. I have done it by hand every year, but the job is one that requires judgement, which is worth handing to a model.

Asking it a question

$ python3 src/predict.py "The mass of a randomly chosen metal disc follows a                          normal distribution with mean 22 grams and                          standard deviation 0.8 grams. 2 metal discs are                          randomly chosen. Find the probability that the                          mass of one of the discs is less than 23 grams and                          the other metal disc has mass more than 23 grams."    62.3%  NORMAL DISTRIBUTION    5.1%  BINOMIAL DISTRIBUTION    4.6%  SAMPLING

That is a real prelim question with the values changed, from a paper the model was never trained on.

93.1% accuracy across all 21 chapters, or 229 of the 246 questions held back from training.

Firstly, the data

Since there is no public dataset readily available, I had to first consolidate all of the training data. In total I classified 1,451 questions after removing duplicates.

sourcespast-year papers
extract1,606 questions
dedupe1,451 rows
splitby paper, not row
TF-IDFwords to numbers
chapter1 of 21

Data split by whole exam papers rather than single questions. This prevents the model from being tested on a paper it has already seen, which ensures independence.

Two approaches

There are two honest ways to sort a question into a chapter. One counts the words. The other hands the sentence to a model that already knows English and asks it to work the answer out. I built both against the same data, because my goal was to find out which approach worked better and which model I could reliably use.

Approach 1: count the words

Term frequency, inverse document frequency, followed by multinomial logistic regression. There is one weight per word per chapter. This results in 21 scores, and the highest probability is the machine’s answer to which chapter it belongs to. This model does not understand the mathematics in the question.

Approach 2: fine-tune a language model

DistilBERT is 66 million parameters that have already read billions of words of English. It reads a sentence in order rather than as a bag of words, so “the line meets the plane” and “the plane meets the line” are two different sentences to this model while they are identical to the model in Approach 1.

Results

Approach 1winner
TF-IDF + logistic regression
Counts words. One weight per word per chapter.
93.1%
test accuracy, 229 of 246
Training time
2.5 seconds
Parameters
10,884 features × 21 chapters
Approach 2
DistilBERT, fine-tuned
Reads the sentence in order. Pre-trained on billions of words.
91.2%
test accuracy, mean of three seeds
Training time
~11 minutes
Parameters
66,000,000

The word counter is more accurate, and it got there in two and a half seconds against about eleven minutes.

Why the simple one won

The training data did not justify the 66 million parameters. Moreover, the signal for which chapter the question belongs to is mostly vocabulary. “Argand”, “significance level” and “position vector” signal Complex Numbers, Hypothesis Testing and Vectors respectively. Finding words that decide a label is precisely what TF-IDF is built to do. A transformer’s advantage is understanding word order and context, and there is very little in this problem that needs it.

What each model gets wrong

The two models are separated by accuracy, but they are separated much more interestingly by what they fail on.

Approach 1
Confuses my own boundaries
Differentiation called Curve Sketching×3
Sequences and Series called APGP×2
Twelve other pairs, noise×1
The mistakes made by the model were due to me trying to split the line between two very similar chapters.
Approach 2
Makes the same mistake, more often
Differentiation called Curve Sketching×5
Differentiation called Equations×2
Normal Distribution called Sampling×2
Sequences and Series called APGP×2
Nine other pairs, noise×1
The same boundary trips both models. It trips DistilBERT nearly twice as often.