Perplexity Details Serving Architecture for Search Embedding Models
The architecture splits workloads across a Rust HTTP frontend, a Rust gRPC batch scheduler called Tulip, and a Python CUDA execution engine called ROSE.
- Tulip uses CUDA graphs to pre-record GPU execution paths and launch tasks in a single call to lower CPU overhead.
- LazyTensors asynchronously tracks results so the CPU prepares the next batch while the GPU executes the current one.
- The system separates bulk batch embedding for indexing from latency-sensitive online embedding for live search.
- Batching targets approximately 512 tokens per batch to saturate GPU runtime for small embedding models.
Engineers building vector search systems can use these batching and CUDA graph techniques to reduce inference latency and improve GPU utilization.
