4 jobs in OxSci

Founding team · Full-time · Hybrid (London or Oxford) · Meaningful equity About Us

Science runs on trust, but the machinery meant to produce that trust is failing. Peer review is slow, opaque, inconsistent, and increasingly gamed. And a flood of AI-generated research is about to overwhelm a system that was already straining. No one is the credible, independent arbiter of research quality. We think there should be one.

OxSci is building a credit rating agency for science : a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale. We're early, focused, and well-resourced. The standards we set now (what "quality" even means, and how expert judgment and AI combine to produce a defensible rating) will define the company.

OxSci was founded by Shumiao Ouyang, Associate Professor of Finance at Saïd Business School, University of Oxford, and Fellow in Management at Wadham College. His research spans fintech, digital payments, and AI; he holds a PhD in Economics from Princeton and degrees from Peking and Tsinghua Universities. He started OxSci out of a conviction that the systems for judging scientific quality are overdue for reinvention, and you'd be building it with him directly.

We're looking for a founding editorial mind to shape how OxSci's certification works, and to earn the academic community's trust in it. The quality judgments themselves rest with the expert peers who review; AI is there to make their work faster and sharper, not to replace it. Your leverage is research taste and a deep read of the academic ecosystem. Concretely:

• Build and curate the network of expert reviewers: knowing who the right people are, bringing them in, and earning their trust.
• Shape and continually improve how reviews happen, including how AI supports reviewers rather than overriding their judgment.
• Bring research taste and ecosystem fluency to guide where the standard and the product should go, spotting what's not working and pushing the corresponding changes.
• Represent OxSci to journals, institutions, and researchers, and be a credible voice on research quality.
• Work side by side with the founder and a small technical team to turn all of this into a product, and to shape what the company becomes.

This is a founding role, not a defined box. Much of the work isn't written yet; you'll help write it.

You know the academic publishing world from the inside, and you want to fix it, not administer it.

The strongest fit has spent time on the professional side of scholarly publishing: a journal editor, a Publisher or portfolio lead at a major house, a research-integrity or peer-review-innovation role, or an early or founding seat at a next-generation publishing or metascience venture. Many of the best people we'll meet are researchers who crossed over from the bench, or operators who've already tried to fix peer review from inside the system. Top-publisher experience is a plus, not a gate.

• Research taste and ecosystem fluency. You can tell strong work from weak, editors and academics take you seriously, and you understand how the machinery of publishing really runs.
• You’re a builder. You’ve shipped something new (launched a journal, built a tool, run a pilot, started something), not only managed what already existed.
• A point of view on what’s broken , and you’ve probably said so in public: a talk, an essay, a thread.
• You see AI as leverage, not threat , and you’re excited to define how it and human judgment combine.
• The mission is what pulls you. You’d want to work on this problem before the details were even worked out.

Strong written and spoken English is essential.

A founding seat with meaningful equity, a direct line to the founder, and the rare chance to define a standard the field doesn't yet have. Editorial and publishing work is chronically underpaid; this role isn't. Pay is genuinely competitive, and compensation and equity are discussed openly and structured for a founding-stage role.

Skip the cover letter. Send us a few paragraphs (one page max) answering:

Email it with your CV to We read every one, and a sharp answer gets a fast reply.

Overview

OxSci is building a credit rating agency for science: a certification layer that combines AI with expert peer review, so researchers, institutions, and AI developers can assess research quality quickly and at scale. Webre early, focused, and well-resourced.

The open scientific question at the heart of this company is yours to own: where does the frontier of AI peer review actually stand? What can AI reviewers catch that human experts miss, what do they still get wrong, and how do you prove it rigorously? The standards we set now, including what \"quality\" even means and how we demonstrate our AI reviewers are actually good, will define both the company and, we believe, the field.

The tech side is led by OxSci's cofounder, a senior tech lead from one of the world\'s largest technology companies, with deep experience building and operating systems at global scale. You\'d work with both founders day to day, shaping the evaluation methodology and research culture from the ground up.

• Own the research agenda on AI-reviewer evaluation. Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why.
• Design meta-evaluations that expose weaknesses, not just measure agreement. Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching.
• Run expert-annotation studies at scale. Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience.
• Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change.
• Calibrate the combined rating. Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us.
• Close the loop. Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps.

• A PhD (or near completion) in CS, ML, NLP, or a related field , or an equivalent research track record. You\'re likely already working on LLM evaluation, LLM-as-a-judge, AI for science, automated peer review, or a nearby frontier.
• A fascination with the boundary between AI and human reviewers. What each catches that the other misses, and conviction that mapping it rigorously is how trustworthy peer review gets built.
• A track record of rigorous evaluation of ML/LLM systems: benchmark or eval-framework design, evaluator/judge models, expert-annotation study design, hallucination and factuality measurement, uncertainty quantification, or RAG evaluation. Bonus if you\'ve built evaluations that score individual criticisms rather than just verdicts.
• Fluency in evaluation methodology and statistics: sampling, inter-annotator agreement, significance testing, and the discipline to distinguish a real effect from a lucky prompt.
• Strong Python and hands-on habits. You build the eval harnesses and pipelines yourself, not just spec them, with enough LLM-application fluency (RAG, tool calling, orchestration) to turn a finding into a shipped improvement.
• Bonus: publications in NLP/ML evaluation or automated peer review; open-source benchmarks or evaluator models the community actually uses; experience with scholarly content at scale.

• Founding seat with meaningful equity and a direct line to the founders
• Ownership of a genuinely open research question, with encouragement to publish and present the work
• A standing expert-reviewer network as your annotation infrastructure
• A proprietary, growing dataset of paired human and AI reviews of real submissions
• The rare chance to define the standard by which AI reviewers themselves are judged
• Genuinely competitive pay; equity discussed openly
• Generous LLM token budget for your daily work
• Flexible working hours; fast personal growth with broad ownership from day one

OxSci in London seeks a PhD-level researcher to lead AI-reviewer evaluation, shaping how AI reviewers are judged in science.

You will design meta-evaluations, run large-scale expert-annotation studies, and build a living taxonomy of AI-reviewer failure modes. Join a founding team with equity, flexible hours, and open research questions as you push the frontier of AI-enabled peer review.