About the role
The training stack is what lets researchers turn a good idea into a trained model. We build the frameworks that shard parameters, gradients and optimiser state across tens of thousands of chips, keep them fed with data, and recover when something inevitably breaks mid-run.
Pretraining, post-training and reinforcement learning each stress the stack differently, and Frontier AI needs all three to move quickly. Long runs also feed Clexerl's products directly, from reasoning models to the long-memory systems behind Clexerl Go, so a day lost to an unstable job is a day lost for everyone downstream.
We treat a training run as production software. Runs have owners, dashboards, loss-curve alarms and rehearsed recovery procedures, and research engineers sit beside us rather than behind a ticket queue.
What you’ll do
- Build and maintain sharded training with FSDP-style parameter sharding, tensor, pipeline and expert parallelism, and sequence parallelism for long contexts.
- Design checkpointing that is fast, asynchronous and resumable on a different topology from the one that wrote it.
- Improve model FLOPs utilisation by overlapping communication with compute and eliminating pipeline bubbles and stragglers.
- Diagnose silent failures such as loss spikes, numerical divergence and corrupted shards, and build tooling that catches them early.
- Make deterministic, reproducible runs possible so research comparisons are trustworthy.
- Support reinforcement learning workloads where rollout generation and learner updates must be coordinated at scale.
What you’ll bring
- Hands-on experience training large neural networks across many accelerators, with real knowledge of where scaling breaks down.
- Strong Python and PyTorch internals, plus C++ or Rust where performance or reliability requires it.
- Understanding of collective communication such as all-reduce, all-gather and reduce-scatter, and how topology shapes their cost.
- Experience debugging hard distributed problems like hangs, deadlocks, nondeterminism and uneven load.
- Care for data pipelines: streaming, shuffling, deduplication and resumable data loaders at scale.
- A willingness to be on call for a long run and to write down what went wrong afterwards.
Nice to have
- Experience with fault-tolerant elastic training that survives node loss without restarting the whole job.
- Background in mixed-precision and low-precision training numerics.
- Familiarity with asynchronous reinforcement learning systems and off-policy correction.
- Open-source contributions to a distributed training framework.
Work you might take on
- 01
Implement topology-independent checkpoint resharding so a run can resume on a different number of nodes after a hardware loss.
- 02
Add straggler detection that identifies a slow accelerator or link from step-time telemetry and drains it automatically.
- 03
Overlap expert-parallel all-to-all traffic with compute in a mixture-of-experts model and measure the utilisation gain.
- 04
Build a divergence early-warning system that watches gradient norms and loss statistics and pauses a run before it wastes days.
Compensation
$300K – $500K
The US base salary range for this role. Every offer also includes equity in Clexerl and, when it spins out, in the venture you help build. We tell you the reasoning behind your number.
Benefits
Health, fully covered
Medical, dental and vision for you and your dependants, premiums paid in full.
Two kinds of equity
Equity in Clexerl, plus a stake in the venture you help build when it spins out.
22 weeks parental leave
Fully paid, for every parent, however your family arrives.
Rest that sticks
Flexible time off with a four-week minimum we actually check on, plus a company-wide week in December.
Learning and compute
A yearly budget for courses, conferences and books, and compute credits for side projects.
401(k) with a 4% match
Matched from your first paycheque and vested immediately.
How we hire
Apply 5 minutes
Send links to what you have built and a few lines on why this role. No cover letter needed.
Intro call 30 minutes
A conversation with the hiring manager about the work, the team and what you want next.
Work sample 2–4 hours
A paid take-home or a pairing session on a real-shaped problem. You pick the format.
Studio day One day
Meet the team, go deep on your work and ours. Remote or in person.
Offer Within 5 days
A written offer with the full range, the equity and the reasoning behind both.
Opens an email with a short template. Links beat résumés.