Skip to content

Member of Technical Staff - Distributed Training Engineer

Liquid AISan FranciscoRemote

First seen Jul 29, 2025 · open 479 days · posted onceNo pay listed

HirQ verdict

45/100 · Weak. Based on age, reposts, pay, the company's hiring and ghost signals.

Medium ghost risk
True age
479 days
Reposts
Posted once

See the evidence behind this score.

Pro shows each part of the score, the reasons, the pay percentile and the listing history.

HirQ's ghost-risk estimate and company labels are automated opinions based on what we observe on employers' public job boards: how long a listing has been open, whether it was re-listed, wording in the description, and how often roles on that board close. They are not statements about an employer's intent, and a high-risk listing can still be a real opening. Figures are as of the date shown. How we measure

At a glance

Pay
No pay listed
Level
Lead
Type
FullTime
Team
Research & Engineering
Work setup
Remote

“San Francisco”

Pay vs market

Employer-posted ranges for Member of Technical Staff roles in CA in the last 12 months (74 postings).

$201k
25th pct/yr
$235k
Median/yr
$284k
75th pct/yr
$175k · 10th90th · $307k

This job: No pay listed

Liquid AI hiring

Company page
SteadyHiring46· WeakAI & Machine Learning
Open roles
29
New in 30 days
11
Closed in 30 days
4
Hiring trend, 13 weeksPro
Follow-through: ••%Pro

About the role

From Liquid AI's careers page, checked Nov 20

ABOUT LIQUID AI

Spun out of MIT CSAIL, we build general-purpose AI systems that run efficiently across deployment targets, from data center accelerators to on-device hardware, ensuring low latency, minimal memory usage, privacy, and reliability. We partner with enterprises across consumer electronics, automotive, life sciences, and financial services. We are scaling rapidly and need exceptional people to help us get there.

THE OPPORTUNITY

Our Training Infrastructure team is building the distributed systems that power our next-generation Liquid Foundation Models. As we scale, we need to design, implement, and optimize the infrastructure that enables large-scale training.

This is a high-ownership training systems role focused on runtime/performance/reliability (not a general platform/SRE role). You’ll work on a small team with fast feedback loops, building critical systems from the ground up rather than inheriting mature infrastructure.

WHAT WE'RE LOOKING FOR

We need someone who

  • Loves distributed systems complexity: Our team builds systems that keeps long training runs stable, debugs training failures across GPU clusters, and improves performance.
  • Wants to build: We need builders who find satisfaction in robust, fast, reliable infrastructure.
  • Thrives in ambiguity: Our systems support model architectures that are still evolving. We make decisions with incomplete information and iterate quickly.
  • Aligns with team priorities and delivers: Our best engineers align with team priorities while pushing back with data when they see problems.

THE WORK

  • Design and build core systems that make large training runs fast and reliable
  • Build scalable distributed training infrastructure for GPU clusters
  • Implement and tune parallelism/sharding strategies for evolving architectures
  • Optimize distributed efficiency (topology-aware collectives, comm/compute overlap, straggler mitigation)
  • Build data loading systems that eliminate I/O bottlenecks for multimodal datasets
  • Develop checkpointing mechanisms balancing memory constraints with recovery needs
  • Create monitoring, profiling, and debugging tools for training stability and performance

DESIRED EXPERIENCE

Must-have

  • Hands-on experience building distributed training infrastructure (PyTorch Distributed DDP/FSDP, DeepSpeed ZeRO, Megatron-LM TP/PP)
  • Experience diagnosing performance bottlenecks and failure modes (profiling, NCCL/collectives issues, hangs, OOMs, stragglers)
  • Understanding of hardware accelerators and networking topologies
  • Experience optimizing data pipelines for ML workloads

Nice-to-have

  • MoE (Mixture of Experts) training experience
  • Large-scale distributed training (100+ GPUs)
  • Open-source contributions to training infrastructure projects

WHAT SUCCESS LOOKS LIKE (YEAR ONE)

  • Training throughput has increased
  • Overall training efficiency/cost has improved
  • Training stability has improved (fewer failures, faster recovery)
  • Data loading bottlenecks are eliminated for multimodal workloads

WHAT WE OFFER

  • Greenfield challenges: Build systems from scratch for novel architectures. High ownership from day one.
  • Compensation: Competitive base salary with equity in a unicorn-stage company
  • Health: We pay 100% of medical, dental, and vision premiums for employees and dependents
  • Financial: 401(k) matching up to 4% of base pay
  • Time Off: Unlimited PTO plus company-wide Refill Days throughout the year

Skills mentioned

  • Distributed Systems
  • SRE
  • Machine Learning
  • PyTorch
Apply on Liquid AI's siteOpens Liquid AI's own application page.

Similar jobs

Same role or field, nearby or remote, ranked by HirQ score.

More at Liquid AI

All roles

Keep exploring

Member of Technical Staff - Distributed Training Engineer

No pay listed · HirQ 45

Apply