Efficient batch embedding requires controlled batching with API-specific size limits (e.g., OpenAI max 2048 texts per request), rate limiting with exponential backoff, lazy streaming of documents, and persistent caching to avoid redundant work. LangChain's base classes provide chunking, but additional rate control and checkpoint handling must be implemented manually.
Batch embedding large corpora without hitting API rate limits or memory constraints requires three strategies: controlled batch sizing (OpenAI supports up to 2048 texts per request[reference:20], but effective batch size often lower), rate limiting with exponential backoff and token bucket, and streaming/lazy loading of documents to avoid loading entire corpus into memory. LangChain's embed_documents automatically chunks inputs, but lacks built-in rate control, resume capability, or checkpoint handling[reference:21].
OpenAI API limits: maximum 2048 texts per request, 500k tokens per minute (tpm) rate limit[reference:22]
Memory management: Use lazy loading (
.lazy_load()) with generators to avoid holding all documents in memoryToken counting: Pre-compute token counts per document to prevent exceeding per-request token limits
Resume capability: Store processed document IDs with embeddings to resume from failure points
Parallelism: Consider concurrent embedding requests with semaphore control for throughput (stay within RPM limits)
Cost: Monitor token usage; embedding large corpora can be expensive; use token-aware chunking to avoid waste