Capacity Planning

Adaptive indexing automatically manages the memory index for you, and should generally be used with the default settings, regardless of the the shape of your transaction load.

memory-index-threshold=32m
memory-index-max=512m

If you have an older transactor properties file with different memory-index settings, change them to match the settings above.

The following scenarios may warrant adjusting the object-cache-max setting on the transactor:

The transactor will automatically apply back pressure during sustained heavy write loads such as imports. When the amount of novelty accumulated by the transactor exceeds memory-index-max before an indexing job can be completed, the transactor will radically slow the rate at which it processes transactions.

Import jobs should use transact-async and be willing to wait for several minutes for a transaction to complete. Once the indexing job is finished, transaction speeds will return to normal.

During imports, you can safely ignore [AlarmBackPressure] alarms, since they are expected.

Two practices will help make imports as fast as possible: pipelining and batching.

• To pipeline transactions, use Connection.transactAsync to send more

than one transaction at a time. It is still important to wait on the transaction futures, but you can use e.g. a semaphore to keep ten transactions in flight at once.

• If the groupings of datoms into transactions during an import do not

have semantic significance for your data, you are free to batch data into transaction sizes that are optimal for import data size and speed.

Pipelining 20 transactions at a time with 100 datoms per transaction is a good starting point for efficient imports.

Temporarily provision a larger system than you will later use in production. E.g. when running on DDB, your import will be throttled by your provisioned write volume, so turn it up to go faster.

For the fastest imports, you will want DDB writes set to 1000 or more.

After a large import, call gc-storage to recover storage space.

If your database is very small, you can get extraordinary read performance by setting datomic.objectCacheMax high enough that the entire database fits in the object cache on the peers.

Transactors can take advantage of multiple CPUs, both for transaction processing and indexing. If your monitoring shows that a transactor process is using most of its CPUs, increasing the number of CPUs will improve all aspects of transactor performance.

Transactions must consider the existing state of the database, e.g. to enforce uniqueness constraints and cardinality rules. As soon as your database is larger than memory, Datomic will automatically swap in segments of the database as needed. In a basic configuration, this means reading in segments from storage. When you add memcached and/or valcache to a system, these reads can be an order of magnitude faster.

Prefer squuid for UUID generation.

When a transaction submits a datom whose attribute is unique, Datomic must test it against existing datoms with that attribute. Since many UUID generators are deliberately random, this test has very poor locality, consuming (or even thrashing) a large number of segments in the object cache. (d/squuid) generates UUIDs with some temporal locality.

Many database systems support a variety of workloads, with varying degrees of priority. For example, a common pattern is a system that has an ongoing load of user-facing transactions, plus one or more batch jobs that import data from other systems.

Submit transactions directly only where you need a low-latency, synchronous response. Put all other transactions on a queue, and limit their submission rate so that high-priority transactions achieve better latency and throughput.

Because Datomic stores time information, a common modeling mistake is to assume that everything temporal belongs in Datomic–particularly high volume, non-transactional operational logs.

Datomic is designed as a system-of-record database. Put your records in Datomic, and put your operational logs somewhere else.

If your system stores information whose past values are unimportant, declare those attributes as noHistory.

By default, background indexing is mostly serial. You can request more parallelism in indexing by setting the index-parallelism transactor property, available in 0.9.5981 and later. index-parallelism defaults to 1.

If you have high write volumes, a transactor with plenty of CPUs to spare, and are using a scalable storage service, you can set index-parallelism as high as 8 to speed up indexing jobs:

index-parallelism=8

Only part of the indexing job is parallelized, so you will not see a linear increase in CPU usage when you increase index-parallelism.

The [StorageBackoff] metric measures time spent backing off and retrying storage operations. Most storage will rarely (if ever) trigger this metric, but DynamoDB will trigger it frequently as you near your provisioning limit.

Every [StorageBackoff] implies that an operation is being forced to wait, and that your system could potentially have lower latency and more throughput by provisioning more storage. This does not mean that you should provision a system to eliminate all backoffs.

I/O may not be your bottleneck, so you should correlate storage backoffs with application-level performance, e.g. the transaction metric before deciding to increase provisioning.

The write-concurrency setting in the transactor properties file sets a soft limit on the number of concurrent writes to storage that Datomic will attempt. The minimum write-concurrency is two (2).

When using DynamoDB, match write-concurrency to your write provisioning, with one thread per 200kb/sec of write, e.g. a system with DynamoDB write set to 800 might choose the (default) write-concurrency of 4. write-concurrency should be set commensurate with the lowest expected DynamoDB write provisioning to be used.

If you fail to provision enough DynamoDB capacity, your [StoragePutMsec] maximum times will spike significantly as you hit your throughput limit and the storage library automatically adds exponential backoff and retry to your writes. If your [StoragePutMsec] Maximum time exceeds one second, you are so far past your provisioned limit that the storage library may eventually fail a write, killing your transactor.

If you are regularly being throttled on DynamoDB reads, expand the size of your Memcached cluster to serve reads from there instead.

DynamoDB has its own metrics, including the very useful [ThrottledRequests] Samples, which tell you how many times a system has exceeded provisioned capacity. You can set an alarm on this metric to be notified if your system starts to exceed its provisioned capacity.

By far the fastest, easiest, and cleanest mechanism for reclaiming storage is to simply delete the underlying storage mechanism, e.g. the DynamoDB table, the SQL table, etc. Deleting an entire table is a fast operation, and is not subject to any provisioning limits.

Clean delete and recreate at the storage level is strongly encouraged for all development, testing, and staging use cases.

It may be impossible to delete the underlying storage in a production system, as that storage is also being used for other Datomic databases, or for other uses entirely.

To reclaim deleted databases in a storage that is also under ongoing use, Datomic provides the gc-deleted-dbs command-line API, taking a system URI (a Datomic URI with database name omitted) as an argument. The following command will delete storage associated with any databases that have been deleted on the foo system:

bin/datomic gc-deleted-dbs datomic:ddb://us-east-1/foo

Such garbage collection competes for resources with ongoing use of the system. You can reduce garbage collection's impact by pacing it with the datomic.gcStoragePaceMsec option, e.g.

bin/datomic -Ddatomic.gcStoragePaceMsec=10 gc-deleted-dbs datomic:ddb://us-east-1/foo

The datomic.gcStoragePaceMsec setting defaults to no pacing, allowing GC to go as fast as a single thread can write. If you need to slow the pace of GC down, setting this value to an integer will cause GC to pause that many milliseconds between operations.

You can kill a gc-deleted-dbs process and restart it later with no adverse affects. Note that:

Calling gc-deleted-dbs has two weaknesses when compared to deleting the underlying storage.

• Datomic must walk all the data structures associated with deleted databases, which takes time proportional to the size of the deleted data
• Datomic does not mark all garbage in all cases, so gc-deleted-dbs will reclaim most (but not necessarily all) storage used by deleted databases