Many scalable NoSQL databases like Cassandra, HBase, Mongo, provide tunable consistency in order to define a specific guarantees level for an operation. And what make them scalable make them also vulnerable: in all case the whole cluster must run on synchronized clocks.
It’s quite surprising that, given how important this is, it is not very detailled in the product documentation. One chapter in the HBase documentation, a pararaph in the MongoDB production readiness, a few lines in the Cassandra book… Most of the times it simply refers to a blindly installation of a NTP daemon on each node with a poorly detailled sentence which look like:
It is recommended that you run a Network Time Protocol (NTP) service, or another time-synchronization mechanism on your cluster and that all nodes look to the same service for time synchronization.
Not lying, this is from the HBase documentation.
Time usually just advances. Even if you have istributed applications running on different nodes, time still should even advance if you switch from one node to another. Obviously if one system is ahead of the others, the others are behind that particular one. From the perspective of an external observer, switching between these systems would cause time to jump forward and back, a non-desirable effect indeed. Let’s be clear: time synchronization is one of the first things to check if you see unexplained problems in your cluster. You will find many blog posts by users who got burned by clock drifting. The clocks on a cluster must be synchronized. A small amount of variation is acceptable, but larger amounts of skew can cause erratic and unexpected behavior.
Now known mostly as a computer and printer company, Hewlett-Packard created for many decades the best scientific electronic laboratory instruments in the world. And also, atomic clocks, like the one you can see above. This picture shows the old master clock of Japan Standard Time (JST). You (maybe, and surely) don’t have the budget to buy and run a real atomic clock in your datacenter, but you can find alternative using a GPS receiver with pulse per second output to make a stratum-1 NTP server. Some guys even did it on a Raspberry Pi…
But in all cases, if you are running a computational cluster you should consider running your own NTP server. Letting all instances synchronize their clocks independently leads to poor relative clock synchronization. It is also not considered good netiquette since you are unnecessarily increasing load on public NTP servers. For a bigger, scalable cluster, the best option is to run your own NTP server pool externally synchronized. It gives you perfect relative and absolute clock synchronization, high availability, and easy maintenance.