Member of Technical Staff - Machines

THE ROLE:

We are looking for strong engineers with experience and interest in designing, building, and maintaining the novel, high-performance systems that make up our serverless platform.

Specifically, you’ll be working on Modal’s machines layer: the fleet of bare-metal and cloud hosts that every Function, Sandbox, and training job runs on, and the control plane that provisions, images, monitors, and repairs them.

You’ll automate the integration of new capacity from a growing set of hardware providers, from auditing and benchmarking hosts and clusters to maintaining our machine images, configuring GPUs, RDMA, networking, and storage, and getting machines into production. You’ll build the automation that keeps the fleet healthy without human intervention: detecting bad GPUs, thermals, and disks. You’ll dig into whatever is between the hardware and the software that runs on top of it, whether that is a kernel panic, a broadcast storm during boot, or getting our container runtime to run on new architectures and platforms.

WHAT YOU'LL WORK ON:

- Automate remediation of unhealthy machines (power cycling, reimaging, GPU recovery) to maximize uptime and minimize operator toil

- Automate integration of new CPU, GPU, and storage servers into the fleet while managing hardware and network heterogeneity

- Monitor network health and reliability across many datacenters, and standardize bare-metal network configuration

- Automate hardware acceptance testing and benchmarking (CPU, disk, GPU, interconnect, network)

- Build our custom network bootloader, machine image pipeline, and kernel and firmware management across the fleet

YOU MAY BE A FIT IF:

- You have 5+ years of experience writing high-quality production code

- You’ve operated fleets of physical hardware (bare-metal provisioning, BMC/IPMI, PXE or network boot, firmware) or built the control planes that manage them (the more challenges you’ve worked through, the better)

- You have strong cloud skills

- You have strong knowledge of low-level operating system foundations (Linux kernel, drivers, networking, file systems, containers, etc.)

- You’re effective at debugging across layers, from BGP flapping, Linux RPS, and vBIOS bugs to a Python control-plane service

- You’re willing to step into the thick of it with our on-call rotation and respond to production incidents

NICE-TO-HAVES:

- Experience with GPUs and the NVIDIA software stack in production (drivers, health monitoring, XIDs, RDMA/NVLink)

- Prior experience with Go