Upgrading

Important:

The AWS Marketplace configuration of Datomic Cloud utilizes a master template in order to facilitate one-click deployment. On your first upgrade you will move to separate cloud formation stacks for storage and compute so you can manage their upgrades and lifecycles independently, per AWS best practice guidelines.

What Versions Am I Running?

After your first upgrade, a running Datomic system will include one or more CloudFormation templates, each running a specific DatomicCFTVersion.

To determine the DatomicCFTVersion a template is running:

  • Select the name of your system stack in the CloudFormation console
  • On the Outputs tab, find the value for the DatomicCFTVersion key.

Choosing to Upgrade

The releases page provides three resources that can help you decide when and how to upgrade:

  • The Critical Notices section contains critical notices for all users. You should always read this section.
  • The Release History table includes a summary column with a brief description of each release.
  • The Release Notes provide a comprehensive list of changes in each release.

How to Upgrade

Before any ugprade, read the critical notices for all releases between your current release(s) and the release(s) you are upgrading to. These notices may override the generic instructions below.

Unless otherwise directed by the critical notices, choose the first row in the table below whose criteria apply to you, and follow the Upgrade Path instructions.

Criteria Upgrade Path
System has never been upgraded before First Upgrade
Upgrading compute template only Compute-Only Upgrade
Upgrading storage template only Storage-Only Upgrade

First Upgrade Ever

Your first upgrade will bring the entire system down and back up. After the first upgrade, other upgrades will typically be rolling upgrades with no downtime.

This upgrade process converts your Master Template Stack to the setup described in the Production Setup documentation.

Remove original stack

This process removes the Master stack, but will reuse the existing storage in a later step to maintain your storage assets.

  1. Select the root stack for your system in the CloudFormation console. The root stack will have a Stack Name that is the same as your system name.
  2. Choose "Delete Stack" from the "Actions" menu. Confirm this in the Delete Stack popup, then wait for the stack deletion to complete. This can take 10 minutes or more.

Create and Upgrade Storage Stack

Creating a new storage stack that utilizes existing storage from the original master template.

  1. Choose "Create Stack", and then paste in the template URL for the Storage Stack you want from the Releases Page.
  2. Click the Next button, and fill in the template's parameter values as specified below.

    StackName
    Enter your system name, i.e. the name of the stack you just deleted.
    Reuse existing storage
    Set this to "True".

    This CloudFormation Template creates a VPC in which to run Datomic Cloud. Configure the settings for the VPC in the VPC Configuration section. See the AWS VPC Guide for details on specifying the CIDR blocks. Unless you know you need to change the CIDR block settings, you should accept the default configuration.

    VPC CIDR block
    The CIDR block to assign to the VPC. Accept the defaults.
    First, Second, and Third CIDR Blocks.
    Datomic will configure three subnets in the VPC. These three CIDR Blocks must be subsets of the VPC CIDR Block, and they must not overlap with each other. Accept the defaults.
  3. Click the Next button to move to the "Options" screen. Leave all settings unchanged.
  4. Click the Next button to move to the "Review" screen. Under "Capabilities", click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names."
  5. Click "Create" to launch the Storage stack.
  6. Wait for the template to report CREATE_COMPLETE. This can take up to 25 minutes. You can refresh the CloudFormation dashboard to see progress.

Create and Upgrade Compute Stack

  1. Choose "Create Stack" a second time
  2. Go to Releases and click click the file icon for the compute release you wish to use. Selection between Solo and Production depends on factors explained in the Architecture document
  3. Click the Next button, and fill in the template's parameter values as specified below.
    StackName
    Use the name "$(System)-compute", where System is your system name.
    SystemName
    Enter your system name.
    Start bastion?
    Set this to "Yes" so that the template will start a bastion for developer internet access.
    AWS EC2 Key Pair
    The key pair to assign to compute nodes. Select a key pair for ssh access to nodes and the bastion host.
    Application Name
    If you supply the name of an existing application, this compute group will be a deployment group of that application. Otherwise, a new application will be created with the provided name.
    Environment Map
    Ion environment map.
    Preload Database
    All compute group instances will load this database when they start.
    Existing IAM managed policy for node
    Optional. Leave this blank.
  4. Click the Next button to move to the "Options" screen. Leave all settings unchanged.
  5. Click the Next button to move to the "Review" screen. Under "Capabilities", click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names."
  6. Click "Create" to launch the Compute stack.
  7. Verify that your upgraded system is available using the dashboard.
  8. (Optional) To allow developers to access the bastion from outside the VPC, you must follow the steps to Allow Inbound Bastion Traffic. You must repeat these steps if you previously did them, as deleting the compute stack removes that access.

Parameter Upgrade

You can upgrade a CloudFormation stack to change template parameters, while leaving the template itself unchanged. To update a stack parameter:

  • Select your stack via the checkbox or radio button.
  • Click the "Actions" button and select "Update Stack"
  • On the "Select Template" screen, choose "Use current template"
  • On the "Specify Details" screen, change your parameter(s).
  • On the "Options" screen, leave all options unchanged.
  • On the "Review" screen, click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names."

Compute-Only Upgrade

Note: If you have multiple compute groups, upgrade the primary compute group first and then all other query groups in any order.

To update a Compute Stack:

  • Open the CloudFormation console
  • Select your stack via the checkbox or radio button.
  • Click the "Actions" button and select "Update Stack"
  • Select "Specify an Amazon S3 template URL:" and enter the CloudFormation template URL for the version that you wish to upgrade to (see Release page for all versions) then click "Next".
  • On the "Specify Details" screen, leave all options unchanged.
  • On the "Options" screen, leave all options unchanged.
  • On the "Review" screen, click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names."

Storage-Only Upgrade

To update your Storage Resources:

  • Open the CloudFormation console
  • Select your stack via the checkbox or radio button.
  • Click the "Actions" button and select "Update Stack"
  • Select "Specify an Amazon S3 template URL:" and enter the CloudFormation template URL for the version you wish to upgrade to (see Release page for all versions) then click "Next".
  • On the "Specify Details" screen, set the "Reuse Existing Storage" option to true:

    reuse-existing-storage-true.png

  • On the "Options" screen, leave all options unchanged.
  • On the "Review" screen, click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names" and click "Update".

Upgrading from Solo to Production

You must raise your i3.large instance limit before launching the Production Topology.

The Production Topology is designed for production use. It provides high availability, horizontally scalable processing, load balanching, and auto scaling. To update your system stack from solo to production, click the name of your solo stack in CloudFormation console. On the stack page, Click "Update Stack":

updatestack.png

On the following page, select "Specify an Amazon S3 template URL:" and enter the CloudFormation template URL for the latest production template you wish to upgrade to (see Release page for the latest production template) and click "Next".

  • On the "Specify Details" screen, leave all options unchanged.
  • On the "Options" screen, leave all options unchanged.
  • On the "Review" screen, click the checkbox stating "I acknowledge that AWS CloudFormation might create IAM resources with custom names."

Wait for the template to report UPDATE_COMPLETE. This can take five or more minutes. You can refresh the CloudFormation dashboard to see progress. In the EC2 Dashboard you'll note that two i3.large instances with your stack name are running and a t2.small instance has been terminated. Your production stack is now ready for use.

Do Not Downgrade

Generally speaking, you should never downgrade Datomic CloudFormation templates.

  • Newer versions are better: they may contain important fixes.
  • Newer versions are compatible: they continue to support the entirety of the API.
  • Older versions may be incapable of correctly handling newer features.
  • Our support team is better able to help if you are running a recent version.

If you have a need to downgrade Datomic, please contact support first, and let us advise you.

Upgrading Datomic Schema

Every Datomic database starts with a base schema, whose identifiers are prefixed with :db. When the Datomic team enhances the base schema, all databases created with new versions of Datomic get the enhancements automatically.

Existing databases do not automatically incorporate enhancements to the base schema. In order to upgrade existing databases to use new base schema, you must first upgrade all compute groups (primary and query) to the minimum version of Datomic required for the base schema features you want:

Feature Minimum Datomic Version Released
tuples 480-8770 June 27, 2019
attribute predicates 480-8770 June 27, 2019
entity specs 480-8770 June 27, 2019
db.type/symbol 480-8770 June 27, 2019

One you have upgraded all compute groups, you can upgrade an existing database to use the latest base schema by passing the :upgrade-schema action to administer-system. For example:

(d/administer-system client {:db-name "movies"
                             :action :upgrade-schema})

Upgrading a schema is an idempotent operation, so it will not harm your data to call it more than once. That said, you should treat administer-system as a potentially expensive operation. Call it only when you actually need it, not before every call to connect.

Once you have used a base schema feature in a particular database, do not run any compute group for that system on versions of Datomic older than the features used. (See table above.)

Why Multiple Stacks?

A Datomic System consists of Storage Resources plus one or more Compute Stacks. These resources have distinct lifecycles and are upgraded separately (in particular Storage upgrades are less common than Compute upgrades) and so each has its own CloudFormation stack per AWS best practice guidelines.

The Marketplace install wraps a Storage template and Compute template in a single master template. This makes the initial setup slightly easier, but is less desirable for ongoing operations. Use the first upgrade instructions convert from the marketplace master stack to separate stacks.

The Master Template setup also obfuscates various errors, making it more difficult to debug issues.

Future Datomic Systems should be created as described in Production Setup.