# Perplexity Details Serving Architecture for Search Embedding Models

The architecture splits workloads across a Rust HTTP frontend, a Rust gRPC batch scheduler called Tulip, and a Python CUDA execution engine called ROSE.

- Tulip uses CUDA graphs to pre-record GPU execution paths and launch tasks in a single call to lower CPU overhead.
- LazyTensors asynchronously tracks results so the CPU prepares the next batch while the GPU executes the current one.
- The system separates bulk batch embedding for indexing from latency-sensitive online embedding for live search.
- Batching targets approximately 512 tokens per batch to saturate GPU runtime for small embedding models.

## Why it matters

Engineers building vector search systems can use these batching and CUDA graph techniques to reduce inference latency and improve GPU utilization.

## Sources

- [Perplexity: Perplexity Details Serving Architecture for Search Embedding Models](https://x.com/perplexity_ai/status/2095984677463191607)

---

Summarized by dstilled on 2026-09-04. https://dstilled.ai/story/35a44df0-c24c-4452-b625-7f896854a782