{"id":2257,"date":"2011-01-25T10:00:00","date_gmt":"2011-01-25T10:00:00","guid":{"rendered":"https:\/\/ikaisays.com\/?p=2257"},"modified":"2021-04-20T10:01:32","modified_gmt":"2021-04-20T10:01:32","slug":"app-engine-datastore-tip-monotonically-increasing-values-are-bad","status":"publish","type":"post","link":"https:\/\/ikaisays.com\/2011\/01\/25\/app-engine-datastore-tip-monotonically-increasing-values-are-bad\/","title":{"rendered":"App Engine datastore tip: monotonically increasing values are bad"},"content":{"rendered":"\n

When saving entities to App Engine\u2019s datastore at a high write rate, avoid monotonically increasing values such as timestamps. Generally speaking, you don\u2019t have to worry about this sort of thing until your application hits 100s of queries per second. Once you\u2019re in that ballpark, you may want to examine potential hotspots in your application that can increase datastore latency.<\/p>\n\n\n\n

To explain why this is, let\u2019s examine what happens to the underlying Bigtable of an application with a high write rate. When a Bigtable tablet<\/em>, a contiguous unit of storage, experiences a high write rate, the tablet will have to \u201csplit\u201d into more than one tablet. This \u201csplit\u201d allows new writes to shard. Here\u2019s a visual approximation of what happens:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

There\u2019s a moment of pain \u2013 this is one of the causes of datastore timeouts in high write applications, as discussed in Nick Johnson<\/a>\u2018s article, \u201cHandling Datastore Errors<\/a>\u201c.<\/p>\n\n\n\n

Remember that for indexed values, we must write corresponding index rows. When values are randomly or even semi-randomly distributed, like, say, user email addresses, tablet splits function well. This is because the work to write multiple values is distributed amongst several Bigtable tablets:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

The problems appear when we start saving monotonically increasing values like timestamps, or insert dictionary words in alphabetical order:<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

The new writes aren\u2019t evenly distributed, and whichever tablet they end up going to end up becoming a new hot tablet in need of a split.<\/p>\n\n\n\n

As a developer, what can you do to avoid this situation?<\/p>\n\n\n\n