What is a 'Rollback' in the context of a replica set, and how do you minimize its occurrence?

A rollback is the process of reverting write operations on a former primary when it rejoins a replica set after a failover, ensuring data consistency by discarding un-replicated writes.

In a MongoDB replica set, a rollback is an automatic data-recovery mechanism that occurs when a former primary node rejoins the set after a failover. It reverts write operations that were accepted by the old primary but not successfully replicated to any secondary before the primary stepped down. This process ensures that all nodes in the replica set maintain a consistent view of the data. Rollbacks are designed to be rare events, and when they do occur, they are often the result of network partitions or secondary nodes that cannot keep up with the primary's write throughput.

Detecting and Reading Rollback Data

The root cause of a rollback is the timing gap between when a primary acknowledges a write to the client and when that write is replicated to secondary nodes. With default write concern { w: 1 }, MongoDB only acknowledges the write after it's committed on the primary, regardless of replication status. If the primary fails before secondaries can replicate that write, the data exists only on the failed primary. When that node later rejoins the set, it must roll back those un-replicated writes to align with the new primary's data state.

The most effective way to minimize or prevent rollbacks is to use stronger write concerns that ensure writes are replicated to multiple nodes before acknowledgment.

1. Use Majority Write Concern

1
Configure write concern { w: "majority" } to ensure writes are propagated to a majority of voting nodes before acknowledgment
2
Since MongoDB 5.0, { w: "majority" } is the default write concern for most deployments
3
This guarantees that even if the primary fails, the write is safely stored on other nodes and cannot be rolled back

2. Enable Journaling

1
Run all voting members with journaling enabled to provide crash recovery
2
The writeConcernMajorityJournalDefault setting controls whether majority writes wait for on-disk journaling
3
Setting this to false makes majority writes vulnerable to rollback if a majority of nodes crash and restart

3. Ensure Secondaries Can Keep Up

1
Monitor replication lag to prevent secondaries from falling behind
2
High write throughput that outpaces secondary replication increases both the likelihood and impact of rollbacks
3
Consider upgrading hardware or adding more secondaries to distribute read load

4. Handle Priority Configuration Carefully

1
Be aware that higher-priority primaries can trigger rollbacks when they rejoin after failure
2
If a higher-priority node reconnects and immediately attempts to become primary, it may force rollbacks of writes accepted by the current primary
3
Test failover scenarios with your specific priority settings

Difficulty: 5/10

Topics: Rollback, Write Concern, Replica Set Elections

Scenario Questions

0-2 years experience

1
You have a three‑node replica set and after the primary steps down the new primary is missing recent writes. How would you explain what happened and what would you do to fix it?
2
If you need to guarantee that a rollback never occurs during a deployment, which write concern would you set and why?

2-5 years experience

1
During a rolling upgrade a secondary fell behind, the primary then crashed, and you observed a rollback. Walk me through how you would diagnose the root cause and prevent it in future upgrades.
2
Your application is seeing occasional duplicate documents after a network partition. Explain how a rollback could cause this and what configuration changes could reduce its likelihood.

5-8 years experience

1
Design a strategy for a globally distributed MongoDB deployment that minimizes rollback risk when using sharded clusters across regions. Discuss trade‑offs in write concern, read preference, and election settings.
2
You need to build a monitoring system that alerts before a rollback can happen. Which metrics would you track and how would you act on them to keep the replica set healthy?

8+ years experience

1
Our organization is migrating from a monolithic MongoDB deployment to a microservices architecture with multiple replica sets per service. How would you architect the rollout to avoid rollbacks affecting data consistency across services?
2
Given strict data‑integrity SLAs across several data centers, what long‑term operational practices and architectural decisions would you implement to virtually eliminate rollbacks in the replica sets?

Follow-up Questions

❓What specific replica set settings influence the likelihood of a rollback?
❓How does majority write concern protect against rollbacks?
❓Can you share an incident where you had to recover from a rollback in production?