Skip to content

Problem Design Engineer

WidgetfactoryUnited StatesRemote

First seen Oct 28, 2025 · open 388 days · posted onceNo pay listed

HirQ verdict

46/100 · Weak. Based on age, reposts, pay, the company's hiring and ghost signals.

Medium ghost risk
True age
388 days
Reposts
Posted once

See the evidence behind this score.

Pro shows each part of the score, the reasons, the pay percentile and the listing history.

HirQ's ghost-risk estimate and company labels are automated opinions based on what we observe on employers' public job boards: how long a listing has been open, whether it was re-listed, wording in the description, and how often roles on that board close. They are not statements about an employer's intent, and a high-risk listing can still be a real opening. Figures are as of the date shown. How we measure

At a glance

Pay
No pay listed
Level
Mid level
Experience
1+ years
Type
PartTime
Team
Software Engineering
Work setup
Remote

“United States”

Pay vs market

Employer-posted ranges for Hardware & Engineering roles in the US in the last 12 months (4,510 postings).

$118k
25th pct/yr
$154k
Median/yr
$185k
75th pct/yr
$90k · 10th90th · $220k

This job: No pay listed

Widgetfactory hiring

Company page
SteadyHiring49· WeakCybersecurity
Open roles
3
New in 30 days
0
Closed in 30 days
0
Hiring trend, 13 weeksPro
Follow-through: ••%Pro

About the role

From Widgetfactory's careers page, checked Nov 21

WHAT WE’RE BUILDING

We create high-quality coding problems for training AI models — drawn from real production repositories, not synthetic puzzles. Your role is to ensure these problems are fair, clear, and technically sound.

THE ROLE

Part-Time Code Quality Reviewer | Remote | $40/hr | 0-25 hrs/week

We’re looking for engineers who can trace the intent behind a code change, understand what changes were made and why, and express that clearly in natural language.

This is judgment work, not mechanical testing. You'll make philosophical decisions about fairness and clarity in technical contexts.

WHAT YOU'LL ACTUALLY DO

  • Review code changes across unfamiliar repositories and tech stacks
  • Assess problem fairness: Could a skilled engineer solve this with only the given context?
  • Evaluate test quality: Do tests properly validate the intended solution?
  • Write clear justifications for your assessments (2-3 sentences per decision)
  • Flag coherence issues: Inconsistent artifacts, unclear requirements, missing context

Time commitment: 1-4 hours per problem, fully async — work whenever fits your schedule.

WHO SHOULD APPLY

Ideal candidates

  • PhD students in CS/computational fields seeking long-term flexible income
  • Open source contributors who care deeply about code quality
  • Undergrad with at least 1 year in CS, ECE, Computational Biology, or related technical programs
  • Professional engineers seeking intellectually interesting part-time work

You'll thrive if you

  • Have strong software engineering fundamentals
  • Can work independently with minimal supervision
  • Write clearly and precisely
  • Enjoy making judgment calls in ambiguous situations
  • Stay focused during 1-3 hour review sessions

REQUIREMENTS

  • 1+ years of software engineering experience (internships, research projects, or professional work)
  • Fluent English; strong writing is a must
  • Responsive, independent, and attentive to detail
  • 0–25 hours per week, flexible

COMPENSATION & GROWTH

  • $40/hour for all review work
  • Performance-based advancement: Top contributors can receive promotions and bonuses
  • Long-term engagement: We're building a scalable team, not a temporary workforce

APPLICATION PROCESS

  • 15-minute async video interview — answer 4 questions about your technical background and work style
  • Sample problem review — demonstrate your assessment skills on a real coding problem
  • Onboarding — join our active problem design engineering community and start reviewing

FAQ

Q: Is this suitable for PhD students?
Yes — many of our top contributors are PhD students. The work is fully async and fits around research schedules.

Q: Do I need to be available during specific hours?
No. Work whenever you want. Some problem design engineers work weekends, others weekday evenings.

Q: What if I'm not familiar with the programming language in a problem?
That's expected. You're assessing problem fairness and clarity, not implementing solutions. Familiarity with general software engineering principles is what matters.

Q: How quickly can I start?
If you pass the video interview and sample review, you can start within 1-2 weeks.

Q: Is this a path to full-time employment?
This is a long-term contract role, not a pipeline to full-time positions. We value contributors who excel at this specific type of work.

ABOUT IDLEXPERTS

idlexperts is the quality assurance division of Idler (YC S25), where we ensure coding problems meet research-grade standards for AI training. Our mission is to build the most reliable QA system for AI coding benchmarks.

Skills mentioned

  • Quality Assurance
Apply on Widgetfactory's siteOpens Widgetfactory's own application page.

Similar jobs

Same role or field, nearby or remote, ranked by HirQ score.

More at Widgetfactory

All roles

Keep exploring

Problem Design Engineer

No pay listed · HirQ 46

Apply