GPU Performance Engineer

Join the team building the world's fastest generative AI inference platform. As a GPU Performance Engineer at Fireworks AI, you will be responsible for pushing the boundaries of LLM serving by optimizing low-level CUDA kernels and orchestration logic. Your work will directly impact the latency and throughput of state-of-the-art models used by thousands of developers globally.

- Design and implement highly optimized CUDA kernels for LLM inference, focusing on attention and linear layers. - Develop and maintain a high-performance serving stack that maximizes hardware utilization across multi-GPU environments. - Profile and identify bottlenecks in the inference pipeline, from operator-level execution to host-device communication. - Implement cutting-edge quantization techniques to reduce memory bandwidth pressure without sacrificing model quality. - Collaborate with the research team to integrate new model architectures into the optimized production runtime. - Contribute to the development of custom scheduling and memory management strategies for large-scale model serving. - Benchmark Fireworks AI performance against industry standards to ensure continued leadership in inference speed.

- 5+ years of experience in high-performance computing or systems programming. - Expert-level proficiency in C++ and CUDA kernel development. - Deep understanding of NVIDIA GPU architectures (Hopper, Ampere) and memory hierarchies. - Proven track record of optimizing transformer-based architectures and attention mechanisms (e.g., FlashAttention, PagedAttention). - Experience with profiling tools such as Nsight Systems, Nsight Compute, and architectural simulators. - Strong mathematical foundation in linear algebra and floating-point arithmetic (FP8, INT4, NF4 quantization). - Familiarity with deep learning frameworks like PyTorch and compiler stacks like TVM or MLIR.