Skip to main content

5.5 Release Notes

Replication​

remove_node Waits for a Live Node's Writes​

Removing a live node with remove_node could lose writes that only that node held: writes it accepted just before removal, while a peer link was down, or to sharded records it does not own. The node being removed now stops accepting local writes first, and its membership is deleted only after every peer that must hold its writes reports them committed. The wait is bounded by timeout; force: true removes without waiting, and cancel: true stops a removal that timed out. A node that cannot be reached, or the only owner of a shard, now needs force: true. replicatedConfirmation on RocksDB also no longer counts a 5.5 peer that has received a later-timestamped write but not yours; a peer on an earlier version is still counted by timestamp. See Remove Node, harper-pro#1348 and harper-pro#1351.

Own Writes Recovered After an OS Restart​

After an OS crash, power loss or reboot, a node may have lost its most recent commits even though peers confirmed them. It now asks each peer for the writes it originated and applies the ones it is missing. Until every peer and every node in the record lock home map has answered, a cluster record lock on that node that depends on its writes from before the restart waits within its timeout and then answers 503. A restart of Harper alone, with the OS still running, loses nothing and skips this.

  • Every node a node sends a database to must also send it back; on a one-way route such locks stay at 503.
  • A peer that already purged this node's writes from the recovery window cannot return them, as after a node wrote nothing for longer than its peers' retention; such locks then answer 503 until the node is recloned.
  • Writes to a sharded table (one with a residency policy) are not recovered this way, and locks on such a table do not wait for recovery.

Replication Confirmation Defaults​

Two settings make writes wait for peer confirmation without changing each write path:

  • replication.recordLockConfirmation makes a commit that holds a cluster record lock wait for at least that many peers. It is 1 by default while replication.recordLocks is true. Without it, a holder that loses power before its write replicates can lose the write, and the next holder can be admitted without it.
    • While no peer is connected, a locked commit is refused with 503 before anything is written, rather than waiting. A wait whose peers drop ends within about twice pingTimeout (about 2 minutes by default), and its 504 says the write committed on this node but was not confirmed.
    • Each locked commit now waits one replication round trip. Set recordLockConfirmation: 0 to keep the previous behaviour; a single-node cluster with record locks on must, or its locked commits are refused.
    • Nodes with record locks off are unchanged.
  • replication.confirmation, off by default, is the peer count a write waits for when it requests none. A write's own count, including 0, still takes precedence. A node that replicates a database to fewer peers than this count refuses writes to it, so set the key once the cluster has formed.

A write to a table declared replicate: false no longer waits for confirmation, which no peer could give.