S
sarvamBengaluru
SDE2 - API Platform
Posted 24d ago
At a glance
- Posted
- 29 October 2026
- Workplace
- On-site
- Type
- Full-time
- Level
- Mid level
- Country
- India
- Skills named
- dynamodb, fastapi, kubernetes, load testing, mlops, observability, postgresql, python
- Application form
- Ashby
Read from the posting by Shortlist. Pip fills this Ashby form from your profile and shows you every answer before anything is sent — most take under two minutes.
ABOUT THE ROLE
Sarvam's model APIs (LLM, ASR, TTS, Vision, Translate) are how developers and enterprises ship products on our foundation models. Our OpenAI-compatible LLM APIs route every request across multiple model deployments, with streaming, failover, metering, billing, and rate-limiting all part of the same stack. Today it's FastAPI and Python; we're in the middle of rewriting it in Go.
We're hiring a Software Engineer (SDE 2) to own core parts of this platform. The features. The reliability. The performance. This is an IC role.
WHAT YOU'LL DO
- - Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out - The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability - Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects - Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments - Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently - The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves - Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads - Observability (logging, metrics, tracing), so the team can debug production confidently and quickly - Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs - Close partnership with the Inference and MLOps teams on production readiness for new model launches - Design-first, documentation-first engineering culture within the team: RFCs before code, decisions written down, no tribal knowledge
- WHAT WE'RE LOOKING FOR - 3+ years building production backend systems - Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load - Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite - Experience building low-latency, multi-tenant backend services - Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure - Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, and rate limiting - Comfort running services on Kubernetes in production - An ownership mindset: you test before you ship, debug production with metrics, logs, and traces, and write the RCA afterwards
- BONUS POINTS - Experience serving or integrating LLMs in production, especially OpenAI-compatible APIs - Familiarity with inference servers (vLLM, SGLang, Dynamo) and latency metrics such as time to first token and inter-token latency - Experience building metering, billing, or usage-based pricing infrastructure - Time at an early- or growth-stage startup: comfort with ambiguity, speed, and ownership
- WHY SARVAM?
- Sarvam is a fast-moving, high talent-density team building full-stack AI for India, working on problems that push the frontiers of AI with real population-scale impact.
- - Work alongside researchers, engineers, builders, and business leaders who move fast and hold each other to a very high bar - High ownership and high impact, from day one - Everything we do is AI-first, from the way we build and ship to the way we think about problems - You can work on problems that could change how an entire country learns, works, and communicates
- If you want to work on problems at the frontier of AI in India, Sarvam is the place to be.