Urdu Nastaliq OCR leaderboard
Loading…
Method
- Source
- Farhang-i-Asifiyah vol. 4 (1908), a lithographed Urdu dictionary in Nastaliq. Public domain; archive.org
Farhang-i-afiyah_1908_amaduoft. - Crops
- At the scan's native 400 DPI, each page is cut into a header strip plus 8 overlapping bands per column. Every model gets the same images, labelled the same way.
- Prompt
- The same blind protocol for every model: type exactly what is printed, write one line per printed line, and mark every doubt as ⟦?⟧ or ⟦guess?⟧. Temperature is 0 wherever the API accepts it.
- Truth
- A vetted native expert typed the pages blind. Every spot where the models agree against the expert was re-checked against the scan.
- Scoring
- Text is normalised (Unicode NFC, lookalike letters folded), words are aligned by edit distance, and every difference gets a type. Models are ranked by strict word accuracy: the share of the 2,378 truth words reproduced exactly (a word is wrong if misread, missing, a diacritic is lost or changed, or punctuation, spacing or order differ; invented extra words are counted separately). Ties break on fewer silent errors. † Reading accuracy is a lenient secondary view that counts only misread, missing and invented words.
- Fairness notes
- The Claude Opus 5.5 and Fable 5.1 rows come from the original study: they were read inside an agent session under the same protocol, not as single raw API calls. Open-weight models run on a free Kaggle T4 read one band at a time, with images downscaled to at most 1,600 px so they fit in 15 GB of GPU memory. At about 2,400 words, differences under roughly 1.5 points are not statistically meaningful.
- Routes
- “page” means one request per page with all bands. Band-by-band routes (marked *) can repeat a line where neighbouring bands overlap; repeats count as invented words in reading accuracy† but cannot change strict word accuracy. Routes marked * read one band per request because of an image-count limit or because the model is a document-OCR engine. These use the same crops and scorer but a slightly different protocol.
Example failures
Red: the model made an error and gave no sign of doubt. Amber: the model flagged doubt. The scan band is shown above each example.
Submit your model
Send the model ID and API route (or open weights) through the pilot form. We run it under the same protocol.
This table uses the public sample, which may already be in some training sets. The official ranking runs on a private held-out set that has never been published.
Dataset and full report: vettedlance.com/human-eval-dataset.