Decision Traces: What Multi-System Data Fusion Reveals About Institutional Knowledge in Enterprise Hiring
Saad Bin Shafiq · arXiv:2604.19819 · Econometrics (econ.EM) · Published 18 April 2026 · doi:10.48550/arXiv.2604.19819
This is an arXiv preprint. It has not been through journal peer review, and nothing on this page should be read as implying that it has.
Plain-language abstract
What the study did
Hiring systems record different pieces of the same decision. Applicant tracking systems store profiles and keywords. Behavioral assessments store measured traits. Production systems store what happened after the hire. Each system is incomplete on its own.
This study connected those records for 10,765 hires at a Fortune 500 insurance carrier. The analysis found that no tested ATS keyword predicted sustained production after correction, while combining behavioral and ATS signals improved predictive performance. It also quantified the economic value of reaching production sooner.
The broader result is that institutional knowledge becomes measurable only when the evidence chain crosses system boundaries.
Research question
Does the screening signal survive contact with the outcome?
Enterprise hiring runs on keywords. Résumés are parsed into skills, skills are matched against requisitions, and candidates are ranked before a human reads anything. The assumption underneath that pipeline is that the keywords carry signal about later performance.
That assumption is rarely tested, because the systems that would test it do not talk to each other. The screening record lives in one system, the assessment in a second, and the outcome in a third. The question here is what happens when they are joined: does any screening keyword predict sustained production, and does connecting systems recover signal that no single system holds?
Dataset
Three systems, one population
| Hires analyzed | 10,765 |
|---|---|
| Period | 2022–2025 |
| Setting | Fortune 500 insurance carrier |
| Systems joined | ATS · behavioral assessment · production outcomes |
Source: arXiv:2604.19819. Dataset figures as published April 2026.
Method
How the systems were joined and tested
Records for the same hire were linked across the three systems so that a screening profile, an assessment result, and a production outcome could be read as one chain of evidence rather than three disconnected rows.
Skills were parsed from ATS profiles, producing 8,181 unique terms. Terms with enough observations to test were retained, leaving 3,597 candidate keywords. Each was tested for association with sustained production.
Because thousands of keywords were tested at once, a Bonferroni correction was applied for multiple comparisons. Without that correction, a handful of apparently significant keywords is the expected result of chance alone, which is one reason screening heuristics survive as long as they do.
Predictive performance was then compared across configurations — behavioral assessment alone, and behavioral assessment fused with ATS signal — and reported as area under the ROC curve.
Findings
What the joined data showed
| Measure | Value |
|---|---|
| Unique skills parsed from ATS profiles | 8,181 |
| Keywords with enough data to test | 3,597 |
| Keywords predicting sustained production after Bonferroni correction | 0 |
| Keywords significantly anti-predictive | 30 |
| Behavioral assessment alone, AUC | 0.647 |
| Multi-system fusion, AUC | 0.735 |
| Value of a day earlier to production, unadjusted | $54 |
| Value of a day earlier to production, adjusted | $35 |
Three results matter more than the individual numbers.
Keyword screening did not survive correction. Of 3,597 testable keywords, none predicted sustained production once multiple comparisons were accounted for, and 30 were significantly anti-predictive — that is, associated with worse outcomes than their absence.
Fusion recovered signal that no single system held. The behavioral assessment carried real information on its own at AUC 0.647. Joined with ATS signal, ranking quality rose to AUC 0.735. Neither system contained that combined signal by itself.
Speed to production has a price. Reaching production one day sooner was worth roughly $54 per agent per day unadjusted, or $35 per day after controlling for source channel and tenure. That makes the cost of a slow decision measurable rather than rhetorical.
On AUC. Area under the ROC curve measures how well a model ranks a randomly chosen positive case above a randomly chosen negative one. It is not accuracy, and it should not be reported as a percentage of correct predictions.
Practical implications
What follows for people running these systems
- A screening rule is a hypothesis, not a policy
- Keyword filters encode beliefs about performance that can be tested against outcomes. Where the outcome data exists, the filter should be required to earn its place.
- The evidence chain has to cross system boundaries
- Signal that predicts an outcome may exist in no single system. Joining records is not a reporting convenience; it is what makes the question answerable at all.
- Preserve the reasoning, not only the result
- A decision record that keeps evidence, rationale, action, and outcome together remains analyzable years later. One that keeps only the final result does not.
- Ranking is not deciding
- A model that ranks well still needs a person who can approve, edit, or reject the action, and explain the choice afterward.
Limitations
What this study does not establish
- The deployment is a single enterprise context. Findings may not generalize to every role or industry without recalibration.
- The data are observational. Associations reported here do not establish every causal relationship.
- AUC is a ranking measure, not accuracy, and should not be described as such.
- Absence of evidence for a keyword after correction is not proof that the underlying skill is irrelevant to every role.
- Human oversight remains necessary. Nothing in these results supports removing a person from a consequential hiring decision.
- The paper is an arXiv preprint unless and until its peer-review status changes.
Citation
Cite this paper
APA
Bin Shafiq, S. (2026). Decision traces: What multi-system data fusion reveals about institutional knowledge in enterprise hiring (arXiv:2604.19819). arXiv. https://doi.org/10.48550/arXiv.2604.19819
MLA
Bin Shafiq, Saad. "Decision Traces: What Multi-System Data Fusion Reveals About Institutional Knowledge in Enterprise Hiring." arXiv, 18 Apr. 2026, arxiv.org/abs/2604.19819.
Chicago
Bin Shafiq, Saad. "Decision Traces: What Multi-System Data Fusion Reveals about Institutional Knowledge in Enterprise Hiring." arXiv preprint arXiv:2604.19819 (2026). https://arxiv.org/abs/2604.19819.
BibTeX
@misc{binshafiq2026decisiontraces,
title = {Decision Traces: What Multi-System Data Fusion Reveals About
Institutional Knowledge in Enterprise Hiring},
author = {Bin Shafiq, Saad},
year = {2026},
eprint = {2604.19819},
archivePrefix = {arXiv},
primaryClass = {econ.EM},
doi = {10.48550/arXiv.2604.19819},
url = {https://arxiv.org/abs/2604.19819}
}
arXiv abstract · PDF · DOI
Related evidence
Where this connects to the production system
The paper is independent research. The links below go to the company's own published evidence and to the field notes where I work through the same questions in shorter form.
- Nodes evidence and methodology — the company's published methodology and outcome reporting.
- Why ATS keywords fail — the screening result, restated for practitioners.
- Speed to production — the economic value of reaching production sooner.
- Data fusion and predictive performance — what joining systems adds over any one of them.
- What is institutional knowledge? — the shorter argument behind the paper's framing.
Author note
Why I published this
I built the system this data came from, which is a reason to be careful rather than a reason to stay quiet. The findings are unflattering to a category of software I also sell into: they say that the screening layer most enterprises rely on did not predict the outcome it is bought to predict.
Publishing the method and the correction matters more than publishing the headline. To our knowledge this is the first production-scale decision-trace study of its kind, and a first result should be easy for someone else to check, re-run, and disagree with.
If you want to replicate, challenge, or extend any part of it, write to me with the section in the subject line.