Staff / Senior Staff AI Inference Reliability Engineer (Compute Infra)

Join our team as a Staff/Senior Staff AI Infra Engineer, where you'll own the production of our Kubernetes-native control plane that provisions and runs our GPU inference fleet. You'll be responsible for triaging issues, diagnosing faults, and extending our automated self-healing system. You'll also lead incident response, own SLOs and error budgets, and work closely with the inference/ML platform team. This role requires hands-on production-reliability experience with large-scale distributed or GPU-serving systems, as well as a software-engineering mindset.

- Assurer la fiabilité de l'infrastructure d'inférence GPU en production, en prenant en charge la réponse aux incidents et en mettant en œuvre des améliorations systémiques. - Collaborer avec les équipes de plateforme d'inférence/ML pour intégrer les nouvelles technologies en production et améliorer les performances de la pile d'inférence de modèle. - Diriger les initiatives de fiabilité à travers les équipes d'infrastructure, de plateforme ML et de produit, en gérant les parties prenantes lors des incidents et des compromis de feuille de route.

- A demonstrated track record of eliminating operational toil through automation and tooling you built yourself, not just process or documentation - Deep experience owning production reliability for large-scale distributed systems; GPU/ML-serving infrastructure experience is a strong plus - Hands-on experience with Kubernetes controller/CRD patterns, reconciliation loops, and node-level health/remediation systems - Familiarity with the operational realities of multi-tenant model serving (LoRA/adapter-based serving) and with running differentiated SLOs across model pools of different priority - Comfort with event-driven architectures, backpressure/circuit-breaking design, and treating observability (metrics, logs, traces) as something built into a system, not added after an incident - Experience operating a provisioning or host-lifecycle pipeline in production, comfortable troubleshooting stuck or failed state transitions (discovery, bring-up, GPU driver/CUDA validation, health checks, decommission/RMA) at fleet scale, and translating real failure patterns back into improvements to the state machine’s design - A software-engineering mindset first: you write and ship production code, and you think about GPU infrastructure as software to be engineered, not a fleet to be manually operated - Hands-on experience troubleshooting InfiniBand/RDMA networking issues, ideally in a multi-node training or disaggregated-inference context — comfortable reading IB port counters and subnet manager logs, not just application metrics