Research data
Eight million publications turned into migration evidence.
I built a reproducible research pipeline for eight million Scopus records, resolving researcher identities, detecting migration, and correcting gender-inference errors that commercial APIs reported with high confidence.
Technology
Every migration claim depended on a defensible identity trail
The study examined scientist migration to and from Germany from 1996 to 2020. Publication records had to be joined into researcher trajectories before affiliation changes could become defensible migration signals.
More than one million researcher records covered names from over 80 countries. Commercial gender-inference APIs agreed most confidently on some of the name groups where their cultural coverage was weakest.
Built the reproducible research pipeline
I owned the out-of-core data pipeline, author disambiguation, affiliation-change rules, gender-inference ensemble, and reproducible research environment.
The goal was not merely to produce a dataset. Every transformation and modelling choice had to be rerunnable and inspectable by researchers.
Architecture
Scientific claims required reproducible transformations
Identity resolution, migration rules, cultural error analysis, and model confidence remained inspectable from raw publication records to the final research tables.
- 01
Constraint
Eight million publication records exceeded practical in-memory processing on the available hardware.
Decision
Partition the ETL stages with Dask, persist stable intermediate datasets, and make each transformation independently reproducible.
Impact
The full pipeline could resume, validate, and rerun without holding the complete corpus in memory.
- 02
Constraint
Two external gender APIs shared systematic errors on transliterated East Asian and Arabic names.
Decision
Add an XGBoost classifier using character n-grams and country-of-origin features, then ensemble the three sources with calibrated confidence.
Impact
Held-out accuracy reached 89%, with the largest gains on the exact subgroups where the APIs failed.
- 03
Constraint
A changed affiliation could reflect relocation, a sabbatical, or simultaneous appointments.
Decision
Require temporal and publication-sequence evidence before converting an affiliation change into a migration event.
Impact
Migration matrices represented sustained researcher movement rather than every noisy address change.
Outcome
A research instrument, not a one-off data export
- Eight million publication records became more than one million longitudinal researcher trajectories.
- Migration flows were mapped across more than 80 source and destination countries and 27 subject classifications.
- The custom ensemble reached 89% gender-inference accuracy on held-out data.
- Containerised and scheduled workflows made the analysis reproducible across the research environment.
When a model becomes a measurement instrument, average accuracy is not enough. Error distribution across cultures and languages has to be part of the scientific result.
Abrar
Start a project
Working with research data that must remain reproducible?
I can help design the data, modelling, validation, and execution layers so results remain inspectable beyond the first run.
Discuss your data platform