Growing Your System

This section only applies to Datomic 990-9202 and lower. Newer versions of Datomic Cloud are free of licensing-related costs, and the user only pays for the hardware that you use to run the system.

You can change your Datomic Cloud instance size at any time by updating the CloudFormation template. As the table below shows, moving up the t3 instance size chart provides more memory and more CPUs.

• Additional memory can improve the performance of queries and transactions
• Increasing the number of CPUs allows Datomic to serve more requests simultaneously, and can also improve the performance of individual queries and transactions

If you need maximum performance, consider i3 instances. They provide valcache, are not subject to unlimited mode usage charges, and require that you raise your AWS account limit.

To develop and test your Datomic applications at no cost, use dev-local. For complete details on the monthly base price, check understanding your AWS bill.

The primary compute group of a system performs all transactions, while queries can be performed by any compute group (primary or query). The primary compute group should not autoscale: it should run a fixed number of instances based on your transaction load and database count.

For a system with a single primary database or with a low total write volume, you should fix the size of the primary compute group at two instances. When you need to scale queries beyond what these two instances can support, add a query group.

If your system serves a high write volume across more than one database, you may want to run more than two instances in your primary compute group. Contact Datomic support if necessary.

When you first launch a Datomic system it has only a single compute group, the primary compute group described above. When you are ready to support higher query volume, you can add a query group. To add your first query group:

• Create a new query group with the same application name used by the original system
• Direct production traffic to this query group instead of the primary compute group

Datomic will automatically coordinate between query groups and the primary group, using the primary group for transactions and indexing.

If the usage of your system is homogeneous, one query group is all you ever need. You can simply expand that single query group to match the increasing query load. But in many cases, a growing system has more than one audience – in those cases, you can create additional query groups for different use cases. Simply start an additional query group for each distinct use case, and direct traffic for that use case to the query group's unique endpoints.

Tasks-specific query groups provide:

• The ability to scale resources independently for a task, without competing with other tasks
• The ability to turn off resources when not in use, for e.g. intermittent or one-off tasks
• Automatic, tuned-to-the-task caching without the need for configuring storage

A typical example of a multi-query group system is the following:

• An "app" query group that supports a production web application
• An "analytics" query group for live analytics, so that expensive analytics operations do not compete with the web application for resources
• A "support" query group for the operations team to query the system, without interfering with other users

Note that each query group can have its own instance size and group size, tuned to its availability, latency, and scalability requirements.

Datomic Cloud automatically manages a multilayer cache that improves the performance of queries and transactions. If your system makes intensive use of a large database, you can further improve performance by enabling Valcache, an optional additional cache tier backed by SSDs on the primary compute group.

Datomic enables Valcache automatically when you choose an instance type in the i3 family. Before switching to i3, increase your AWS i3 instance limit.

Datomic indexes are redundantly stored in S3 and EFS. These high-reliability, high-availability storages are automatically scaled by AWS, with no user configuration required.

Database logs are stored in a DynamoDB table named "[system]". This table is configured with provisioned throughput and a broad range of autoscaling for both read and write. For most systems, these settings are a good sweet spot: when the system is under low transaction load, autoscaling will turn throughput down, saving money. When the load increases, autoscaling will increase throughput to match.

That said, there are two situations where you may want to increase DynamodDB provisioning for the "[system]" table:

• If you have a very bursty transaction load, and your system is intolerant of increased latency before autoscaling kicks in, you can increase the DynamoDB write minimum
• If your system has a very high write volume, and you are seeing regular DynamoDB throttling of writes, you can increase the DynamoDB write maximum

The Datomic catalog is stored in a DynamoDB table named "[system]-catalog", configured with modest provisioned throughput. Catalog operations are very low volume, you should never need to adjust this table.

This section only applies to Datomic 990-9202 and lower. Newer versions of Datomic Cloud are free of licensing-related costs, and the user only pays for the hardware that you use to run the system.

The total cost of running a Datomic system is the sum of the base price and the usage price.

• The base price is the total monthly price of services that are billed per unit time, usually per hour (e.g. EC2 instances) or per month (e.g. KMS keys). A running Datomic system has a well-known base price based on its configuration, regardless of how the system used.
• The usage price is the monthly total prices of services that are billed by some measure of utilization, e.g. data stored, data transferred, or autoscaled DDB provisioning.

All prices below are for us-east-1, and were most recently updated on May 17, 2021.

There are a number of things you can do to reduce the AWS costs for Datomic Cloud:

• Develop and test your code with dev-local, which has no cost
• Choose the smallest EC2 instance size that can handle your performance requirements
• If you intend to run a compute group for a long period of time, purchase AWS Reserved Instances
• If you have a compute group that is used intermittently, turn off the compute instances when they are not in use

If you plan to support multiple environments with Datomic systems, keep the following points in mind:

• The Datomic system is the unit of AWS IAM access control. If different environments need to have different AWS controls, put them in separate Datomic systems.
• Within a Datomic system, the compute group is the unit of ion deployment. So a single system can host many applications simultaneously. These applications could serve entirely separate purposes, or they could be various revisions of the same functionality.
• Transaction functions execute only in the primary compute group. Deploy them there, not to a query group.
• The CodeDeploy application name is the unit of ion push. If you give different Datomic systems the same application name, they will all be able to deploy from the same push. This is probably what you want if you e.g. test an application in one system and release it in another.

The picture below shows an example deployment designed to meet the following dev workflow objectives:

• Keep production code and data separate from all development work
• Support two different user bases that have different performance requirements: web app and analytics
• Demo and test code before deploying to production
• Allow developers to access test data from their laptops