Skip to content

Member of Technical Staff, Evals

Magic.DevSan FranciscoRemote

First seen Nov 11 · open 10 days · posted once$200k–$550k/yr

HirQ verdict

79/100 · Strong. Based on age, reposts, pay, the company's hiring and ghost signals.

Low ghost risk
True age
10 days
Reposts
Posted once

See the evidence behind this score.

Pro shows each part of the score, the reasons, the pay percentile and the listing history.

HirQ's ghost-risk estimate and company labels are automated opinions based on what we observe on employers' public job boards: how long a listing has been open, whether it was re-listed, wording in the description, and how often roles on that board close. They are not statements about an employer's intent, and a high-risk listing can still be a real opening. Figures are as of the date shown. How we measure

At a glance

Pay
$200k–$550k/yr
Level
Lead
Type
FullTime
Team
Engineering
Work setup
Remote

“San Francisco”

Pay vs market

Employer-posted ranges for Software Engineering roles in CA in the last 12 months (6,470 postings).

$174k
25th pct/yr
$210k
Median/yr
$253k
75th pct/yr
$140k · 10th90th · $300k

This job: $200k–$550k/yr

Where this pay sits: ••th percentilePro

Magic.Dev hiring

Company page
SteadyHiring50· FairAI & Machine Learning
Open roles
8
New in 30 days
1
Closed in 30 days
0
Hiring trend, 13 weeksPro
Follow-through: ••%Pro

About the role

From Magic.Dev's careers page, checked Nov 21

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pretraining, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal.

ABOUT THE ROLE

The Evals team measures the performance and capabilities of internal and external models, helping teams across Magic make informed decisions. The team supports pretraining, RL, data, inference, and product, and sits on the critical path of many of the company's most important decisions.

As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance.

Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality.

WHAT YOU'LL WORK ON

  • Design and validate new evaluations and benchmarks across model training and deployment
  • Investigate model behavior and identify gaps in existing evaluations
  • Measure dataset quality and identify opportunities to improve training data
  • Improve evaluation correctness, reproducibility, and reliability
  • Audit and improve upon public benchmarks, research papers, and open-source implementations
  • Define metrics that accurately reflect model quality
  • Build tools that make evaluation results accessible and useful across Magic

WHAT WE'RE LOOKING FOR

  • Experience developing model evaluations, benchmarks, or ML experiments
  • Strong programming skills and experience working with machine learning systems
  • Track record of owning technical projects end-to-end
  • Strong understanding of post-training techniques and their impact on model performance
  • Skepticism toward results that cannot be reproduced, validated, or explained
  • Experience analyzing model performance and interpreting experimental results
  • Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
  • Commitment to raising the standard of rigor in AI evaluations

OUR CULTURE

  • Integrity. Words and actions should be aligned
  • Hands-on. At Magic, everyone is building
  • Teamwork. We move as one team, not N individuals
  • Focus. Safely deploy AGI. Everything else is noise
  • Quality. Magic should feel like magic

Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.

COMPENSATION, BENEFITS, AND PERKS (US)

  • Annual salary range between $225K - $550K depending on experience
  • Equity is a significant part of total compensation, in addition to salary
  • 401(k) plan with 6% salary matching
  • Generous health, dental, and vision insurance for you and your dependents
  • Unlimited paid time off
  • Visa sponsorship and relocation support for candidates moving to San Francisco
  • A small, fast-moving, highly collaborative team working on frontier AI systems

Skills mentioned

  • Machine Learning
Apply on Magic.Dev's siteOpens Magic.Dev's own application page.

Similar jobs

Same role or field, nearby or remote, ranked by HirQ score.

More at Magic.Dev

All roles

Keep exploring

Member of Technical Staff, Evals

$200k–$550k/yr · HirQ 79

Apply