How does the WiredTiger engine handle concurrency during a massive updateMany operation?

Question

Accepted Answer

WiredTiger, MongoDB's default storage engine, employs a sophisticated concurrency control mechanism to handle massive `updateMany` operations. Unlike older storage engines that used process-wide locking, WiredTiger provides **document-level concurrency** using Multi-Version Concurrency Control (MVCC) and optimistic locking . During a large `updateMany`, WiredTiger must balance throughput, consistency, and resource constraints while potentially affecting thousands of documents. WiredTiger uses **document-level locks** rather than collection or database locks, allowing multiple clients to modify different documents in the same collection simultaneously . For most write operations, WiredTiger employs **optimistic concurrency control**. When two operations conflict on the same document, one will experience a write conflict, causing MongoDB to transparently retry the operation . This approach maximizes throughput while maintaining consistency. When an `updateMany` modifies documents, WiredTiger does not immediately write changes to disk. Instead, it creates new `WT_UPDATE` structures and prepends them to in-memory **update chains** associated with each document . These chains enable MVCC by allowing concurrent readers to see their own consistent snapshot while writers modify the document. Readers traverse the chain from newest to oldest until finding a version visible to their transaction snapshot . This design ensures that readers don't block writers and vice versa . ```javascript
// currentOp output from a massive updateMany operation
{
  op: 'update',
  ns: 'myDb.myCollection',
  query: { fieldId: Long("111") },
  update: { '$set': { fieldId: Long("222") } },
  multi: true,
  numYields: 95113,          // High yields indicate contention
  locks: {
    ParallelBatchWriterMode: 'r',
    FeatureCompatibilityVersion: 'w',
    Global: 'w',              // Global lock held
    Database: 'w',             // Database lock held
    Collection: 'w'            // Collection lock held
  },
  waitingForLock: false,
  flowControlStats: {
    acquireCount: Long("95114"),
    timeAcquiringMicros: Long("143036")  // Time waiting for flow control
  }
}
```

This real-world example from a massive `updateMany` shows over 95,000 yields, indicating frequent pauses to allow other operations to proceed, and significant flow control waiting time due to concurrency management . **Ticket System**: WiredTiger caps concurrent write transactions using a ticket system, defaulting to a maximum of 128 write tickets . During massive `updateMany` operations, write tickets can become exhausted, causing queueing and increased latency. **Yielding (Cooperative Scheduling)**: Long-running `updateMany` operations periodically yield (release locks) to allow other operations to execute. The high `numYields` value in `currentOp` output indicates frequent yielding, which prevents starvation but can extend operation duration . **Intent Locking**: WiredTiger acquires intent locks at the global, database, and collection levels even with document-level concurrency . The `currentOp` example shows global, database, and collection locks held in write mode during the entire multi-update operation . **Flow Control**: In high-write scenarios, MongoDB may activate flow control, throttling writes to keep secondary replication lag within limits. The `flowControlStats` in `currentOp` show time spent waiting for flow control . A critical concurrency consideration for massive `updateMany` operations is the behavior of update chains. WiredTiger has an internal threshold (approximately 1000 updates) for the length of an update chain on a single document . When this threshold is exceeded, WiredTiger performs a **copy-on-write**, rewriting the entire document to a new memory location. During a massive `updateMany` that repeatedly updates the same documents, this can cause significant memory pressure, increased disk I/O, and degraded performance . This is particularly problematic for large documents with frequently updated arrays, where update chains can grow rapidly. When `updateMany` modifies a field with a **unique index**, concurrency handling becomes significantly more complex. WiredTiger must check uniqueness constraints for each document, which can introduce additional locking and validation overhead. Historical data shows that updates on unique indexes were up to 60x slower than on non-unique indexes in some versions, though recent optimizations have narrowed this gap . The overhead is particularly pronounced when updating values to keys that sort before the original value, requiring more expensive index traversal . Concurrent massive updates create significant memory pressure. When readers materialize the latest version of a document by traversing long update chains, they allocate memory outside the tracked WiredTiger cache. This **untracked memory** can grow unbounded with the number of concurrent readers, potentially causing memory spikes and instability . Additi

How does the WiredTiger engine handle concurrency during a massive updateMany operation?

How does the WiredTiger engine handle concurrency during a massive updateMany operation?

Scenario Questions

Follow-up Questions