Research data
MPIDR: gender inference and scientist migration at research scale
I worked on a reproducible pipeline for eight million Scopus records, resolving researcher identities, detecting migration, and improving gender inference where commercial APIs were confidently wrong.
The system I joined
The study examined scientist migration to and from Germany from 1996 to 2020. Publication records had to be joined into researcher trajectories before affiliation changes could become defensible migration signals.
More than one million researcher records covered names from over 80 countries. Commercial gender-inference APIs agreed most confidently on some of the name groups where their cultural coverage was weakest.
Where I contributed
I worked on the out-of-core data pipeline, author disambiguation, affiliation-change rules, gender-inference ensemble, and reproducible research environment.
The goal was not merely to produce a dataset. Every transformation and modelling choice had to be rerunnable and inspectable by researchers.
Decision trail
Constraints became architecture.
Each decision below connects a concrete limitation to the engineering response and its practical effect.
- 01
Constraint
Eight million publication records exceeded practical in-memory processing on the available hardware.
Decision
Partition the ETL stages with Dask, persist stable intermediate datasets, and make each transformation independently reproducible.
Impact
The full pipeline could resume, validate, and rerun without holding the complete corpus in memory.
- 02
Constraint
Two external gender APIs shared systematic errors on transliterated East Asian and Arabic names.
Decision
Add an XGBoost classifier using character n-grams and country-of-origin features, then ensemble the three sources with calibrated confidence.
Impact
Held-out accuracy reached 89%, with the largest gains on the exact subgroups where the APIs failed.
- 03
Constraint
A changed affiliation could reflect relocation, a sabbatical, or simultaneous appointments.
Decision
Require temporal and publication-sequence evidence before converting an affiliation change into a migration event.
Impact
Migration matrices represented sustained researcher movement rather than every noisy address change.
Outcome
What the work produced
- Eight million publication records became more than one million longitudinal researcher trajectories.
- Migration flows were mapped across more than 80 source and destination countries and 27 subject classifications.
- The custom ensemble reached 89% gender-inference accuracy on held-out data.
- Containerised and scheduled workflows made the analysis reproducible across the research environment.
Technology
When a model becomes a measurement instrument, average accuracy is not enough. Error distribution across cultures and languages has to be part of the scientific result.
Abrar Ahmed