Capacity Planning

Transactor memory is used for three key things:

• the memory index holds transaction data that has been logged to disk but not yet committed to a disk index
• the object cache caches recently used values from the database index
• transaction functions can use memory in arbitrary ways

The following system properties in the transactor properties file determine how memory is allocated among the above tasks.

These settings must be included in the transactor properties file. The config/samples directory contains property file examples for both development and production use.

You should uncomment the appropriate settings and then launch bin/transactor with enough memory to cover the sum of memory-index-max and object-cache-max, plus plenty of headroom for the process itself.

The actual amount of memory used by the memory index depends on the recent transaction history. When a system is experiencing high write volume, the memory index can grow up to memory-index-max. Under low write volume, the memory index will range between zero and a little more than memory-index-threshold.

By default, the transactor launches with 1G RAM, which works well with the recommended settings for development use.

bin/transactor my-dev-transactor.properties

The sample production settings are designed to work with 4G of RAM. When launching a transactor for production, you should always explicitly specify the Xmx and Xms flags:

bin/transactor -Xmx4g -Xms4g my-production-transactor.properties

Most applications will never need to change these settings.

There are several things to consider when performing data imports:

Peers need memory for three things:

• their copy of the memory index (up to memory-index-max)
• their own object cache
• application use

For example, with 4GB of RAM on all transactor and peer processes would have the following memory breakdown:

• 2GB (50% of available RAM) for the object cache
• 512MB for memory index max
• 1.5GB remaining for the application

Note that peer processes may use up to the memory-index-max per connected database to hold the live index.

To adjust this balance, set the datomic.objectCacheMax system property before loading the peer library, e.g.

// on startup
System.setProperty("datomic.objectCacheMax", "256m");

Datomic transactions enforce the invariants of a Datomic database. To improve the performance of Datomic transactions, you can

• give Datomic more resources, e.g. more CPUs, memcached, or more memory
• do less work, e.g. prefer squuids, queue transactions, or store what you need

On an ongoing basis, Datomic must perform background indexing jobs to copy recent changes from memory into the persistent index. An indexing job is triggered automatically when the transactor's memory-index-threshold accumulates in memory. (You can also explicitly request an indexing job by calling requestIndex.)

Once indexing has begun, the transactor will continue to process transactions until memory-index-max is reached. This limit protects peers and the transactor from devoting too much memory to the in-memory index. Once this limit is reached, the transactor will radically throttle transactions, and issue [Alarm] and [AlarmBackPressure] metrics once per minute.

To see how well Datomic is keeping up with your system load, follow the [MemoryIndexMB] metric. If this metric reaches the midway point (default 144MB) between memory-index-threshold and memory-index-max during normal operation, your system has enough load that you need to plan carefully to avoid back pressure.

On large systems, you may want to prefer squuids and enable index-parallelism.

When you serve multiple databases with Datomic, you have a choice of provisioning separate transactor pairs per database, or colocating multiple databases on a single transactor pair. Similarly, you can provision peer processes to serve a single database or many, including possibly connecting one peer to multiple databases on different transactors.

When you serve multiple databases from the same (peer or transactor) process, those databases share and compete for that process's resources. In particular:

• A process must hold the sum of every database's memory index. For a single-database process, this will be a small fraction of the total heap, but if there are multiple databases with high write volume, the memory indexes could strain or exhaust the heap.
• All databases compete for space in the object cache to efficiently serve queries.
• Transactions and queries compete for CPUs.
• Garbage collection pauses have a correlated impact across all databases.
• If the process has an operational problem, all databases are impacted.

For all these reasons, you should usually give large and/or mission-critical databases their own dedicated transactor and peer processes. Conversely, it is reasonable to share and conserve resources by hosting many low-volume, non-critical databases on a single system. For example, it is not uncommon to host hundreds of small databases on a staging or continuous integration system.

DynamoDB allows you to independently specify read and write capacity. As a first rule of thumb, you should spend at least as much money on read+write capacity as you do on the transactor instance. The table below shows starter DDB settings for a few common EC2 instance sizes.

Datomic uses all storages similarly - to store log and index segments. These segments range up to about 50kb each. The database is a set of trees of these segments. You can model the leaf segments of the trees as arrays of datoms. How many datoms fit in a segment? It depends first on the value types. Larger value types generally take up more space than smaller ones. Next, the Fressian format has packed encodings of, e.g. small numbers etc, as well as options for semantic compression that we leverage to help mitigate the datom encoding (E, A, and T parts) above and beyond the Values. Finally, all of the segments are compressed (with, e.g. zip), further reducing their size. In practice, for non-large values, we see anywhere from 1,000 to 20,000 datoms/segment.

You can expect every datom to be stored at least 3 times:

• once in the transaction log
• once in an EAVT-sorted index
• once in an AEVT index.

For attributes of ref type, the datom is further stored in a VAET index for reverse lookup. Finally, for attributes with index or unique properties, the datom is also stored in an AVET index. Each index type might have more or less effective compression depending on the redundancy present in the data.

During normal operation, no-longer-referenced (garbage) storage segments accumulate, and you should periodically invoke gcStorage to clean up. The gcStorage operation:

• is intended to be called infrequently, e.g. once a day and/or after transactor restart
• does not lock, block, nor even read any live data segments
• continues in the background on the active transactor until collection is complete or the transactor process terminates
• can easily keep pace with new garbage when run regularly

If you use DynamoDB storage, note that the delete operations performed by gcStorage count against write provisioning, and you may need to increase provisioning accordingly.

The reason that garbage is not deleted immediately on creation of a new tree is that not all consumers will immediately adopt the latest tree. gcStorage should not be run with a current or recent time, as this has the potential to disrupt index values used by long-running processes. Except during initial import, garbage collection (gcStorage) older-than value should be at least a month old.

Note that some storages, like SQL databases, might have their own vacuuming/reclamation facilities that you'll need to run periodically.

Individual storages might add very slight overhead per segment, but in general a db should take similar space in any storage (multiplied, of course, by any redundancy factor in the storage subsystem itself). And storage requirements should increase linearly with more (similar) data. So you can load some non-trivial amount of exemplar data, requestIndex, and backup to get an idea of your basis size.

gc-db runs gcStorage as an independent process.

To invoke the gc-db tool, use the following command:

bin/run -m datomic.tools.gc-db $db-URI $older-than

The tool accepts JVM options. You can also set parallelism (maximum value 128) for Datomic garbage collection with the datomic.deleteConcurrency option. With increased parallelism it is important to set datomic.readConcurrency to double your deleteConcurrency value e.g.

bin/run -m datomic.tools.gc-db -Ddatomic.deleteConcurrency=8 -Ddatomic.readConcurrency=16 -Xms4g -Xmx4g $db-URI $older-than

When you delete a database with deleteDatabase, the database immediately becomes unavailable for use. As a separate step, you may later choose to reclaim all storage associated with deleted databases on a system. There are two ways to do this, corresponding roughly to production and development/testing needs.