Setting up Storage Services

Storage service options are listed here:

• Dev mode (dev)
• SQL database (SQL)
• DynamoDB (DDB)
• Cassandra (Cass3)

The steps required to provision them and detailed instructions are included below (links provided here for convenience).

Note that you can move an application from one storage service to another simply by switching the connection string used by peers and the properties file used to start the transactor. All are fully API-compatible.

All of the script commands described in this document must be executed from the root directory of the Datomic distribution.

Datomic depends on various storage client libraries. The recommended versions of all storage client libraries are in the provided scope of the Datomic pom.xml file:

mvn dependency:list -DincludeScope=provided

These jars are included in the Datomic distribution /lib, and are used by the transactor to access the storage systems. Peers will need the jars corresponding to the storage system on their classpath as well.

The dev storage protocol is intended for development. It runs an embedded JDBC server inside the transactor, and uses local disk files for storage.

noslide

bin/transactor config/samples/dev-transactor-template.properties

By default, the embedded storage runs with default passwords and is accessible only by other processes on the same machine. This configuration is intended for interactive development where application peers and the transactor are colocated.

The steps to provision a SQL database as your storage service are:

• Setup a SQL database, or use an existing one
• Create SQL table (datomic_kvs)
• Create SQL user(s), or use an existing one
• Get JDBC connection string
• Add JDBC driver dependencies to your project

There are scripts for doing each of the first three steps with PostgreSQL, MySQL, and Oracle in the Datomic distribution's bin/sql directory. You can run them using their respective command line or GUI admin tools.

For example, for Postgres at the command line:

noslide

psql -f bin/sql/postgres-db.sql -U postgres

psql -f bin/sql/postgres-table.sql -U postgres -d datomic

psql -f bin/sql/postgres-user.sql -U postgres -d datomic

The last script creates a user named 'datomic' with the password 'datomic'. You can use an existing user instead, or modify the script to create a user with a different username or password, if desired.

If you want to use a different SQL server, mimic the table and schema from one of the included databases.

Datomic is not compatible with DynamoDB's Global Tables feature.

The steps to provision DynamoDB as your storage service are:

• Set up an AWS account, or use an existing one.
• Create a DynamoDB table, IAM roles, S3 bucket, permissions, etc. Note that this process can be automated using Datomic's ensure-transactor command.
• Add the AWS DynamoDB dependency to your project.

Datomic supports IAM roles. If you have a transactor that uses IAM user access keys, see Migrating to IAM Roles for information on how to switch to using roles. Check AWS Access Control for information about Datomic's use of IAM roles and IAM user access credentials in general.

If you don't already have an AWS account, sign up using the Amazon Web Services Console.

Once you know everything is working, you may want to run your Transactor on the Amazon cloud. See Running on AWS for instructions.

Using Cassandra as a storage service requires a cluster of at least 3 nodes running Cassandra 3.x+ with a replica factor of at least 3. The Cassandra3 storage (using the `cass3` protocol) uses the Datastax V4 Driver and supports Cassandra-as-a-Service offerings such as AstraDB and AWS Keyspaces. If you are looking for the Cassandra or Cassandra2 legacy storages, look here.

The version compatibility matrix below shows which components from the Cassandra integrate with which Datomic release. Refer to the version of the Cassandra driver in the provided scope in the pom.xml in the Datomic directory and the Datastax Java Driver Compatibility matrix, to verify Datastax/Cassandra compatibility for a specific release of Datomic.

The cluster needs to be configured to support the CQL native transport (this is on by default in modern releases of Cassandra). If you have an existing cluster that meets these requirements, you can use it. Otherwise, you need to configure one following the instructions on the Cassandra site.

Note that Datomic does not support running on a Cassandra cluster that spans datacenters.

Once a cluster is configured, you must provision a keyspace and table (column family) for Datomic to use. You can do this using the CQL scripts provided in the distribution's bin/cql directory. You can execute the scripts using the cqlsh tool provided by Cassandra (use an appropriate superuser username and password, if required):

noslide

cqlsh -f bin/cql/cassandra3-keyspace.cql -u cassandra -p cassandra

noslide

cqlsh -f bin/cql/cassandra3-table.cql -u cassandra -p cassandra

Datomic provides optional support for Cassandra's internal username/password mechanism for authentication and authorization. If your cluster is configured to require authentication and authorization, you must also create a user for Datomic:

noslide

cqlsh -f bin/cql/cassandra3-user.cql -u cassandra -p cassandra

This script creates the user and grants it access to the Datomic keyspace.

Next, setup your transactor properties file:

• Copy the config/samples/cassandra3-transactor-template.properties file to another location and give it a new name, for instance, cassandra3-transactor.properties.
• Set the entry for cassandra-host to refer to a member of the cluster.
• If your cluster uses a non-standard port for the CQL native transport, set cassandra-port.
• If your cluster is configured to require authentication and authorization, set cassandra-user and cassandra-password.
• If your Cassandra cluster is configured with the necessary certificates to support SSL, and verified it works with a simple client (e.g., cqlsh), you can configure the transactor to use SSL by setting cassandra-ssl to true. See the Cassandra docs for more information.
• For advanced configurations, Datomic also supports file based driver configuration via the cassandra-config-file transactor property or by copying the file to ./resources/application.conf within the Datomic distribution directory. Peers similarly need the config file copied to ./resources/application.conf , see the Cassandra Driver V4 Docs for more information.

• You can optionally provide cassandra-session-callback, whose value is a name identifying either a Java static method (e.g. my.app.Cassandra.buildCluster), or a clojure function (e.g. my.app/build-cluster). This method/function takes as its single argument a map containing the above parameters, and which returns an instance of SyncCqlSession. Note that the argument type for this method is Object, e.g.

  import com.datastax.oss.driver.api.core.cql.SyncCqlSession;
  import com.datastax.oss.driver.api.core.CqlSessionBuilder;
  import java.net.InetSocketAddress;
  import datomic.Util;
  import java.util.Map;
  
  public class DatomicCassandra {
    public static SyncCqlSession build(Object args) {
      Map <Object, Object> params = (Map<Object,Object>) args;
      CqlSessionBuilder session = new CqlSessionBuilder();
      session.addContactPoint(InetSocketAddress.createUnresolved(
                            (String)params.get(Util.read(":host")),
                            (Integer)params.get(Util.read(":port"))));
      session.withAuthCredentials((String)params.get(Util.read(":user")),
                                                 (String)params.get(Util.read(":password")));
  
      // Set other session options here
  
      return session.build();
    }
  }
  

• If using AWS Keyspaces with IAM, you will need to add the additional Authentication Plugin dependency below (with the exclusions) and follow the Keyspaces docs:

<dependency>
 <groupId>software.aws.mcs</groupId>
 <artifactId>aws-sigv4-auth-cassandra-java-driver-plugin</artifactId>
 <version>4.0.9</version>
 <scope>provided</scope>
 <exclusions>
  <exclusion>
   <groupId>com.datastax.oss</groupId>
   <artifactId>java-driver-core-shaded</artifactId>
  </exclusion>
  <exclusion>
   <groupId>com.datastax.oss</groupId>
   <artifactId>java-driver-core</artifactId>
  </exclusion>
 </exclusions>
</dependency>

In your peer project, add a dependency for the DataStax Java Driver for Apache Cassandra.

In a tools-deps based project, add the following to the deps section of your deps.edn:

noslide

com.datastax.oss/java-driver-core-shaded {:mvn/version "4.17.0"}

In a maven based build, add the following snippet to the dependencies section of your pom.xml:

noslide

<dependency>
  <groupId>com.datastax.oss</groupId>
  <artifactId>java-driver-core-shaded</artifactId>
  <version>4.17.0</version>
</dependency>

In a leiningen project, add the following to the dependencies section of your project.clj:

noslide

;; in collection under :dependencies key
[com.datastax.oss/java-driver-core-shaded "4.17.0"]

Note that Cassandra, Cassandra2 and Cassandra3 are different storage services! You can use backup and restore to migrate databases from Cassandra or Cassandra2 to Cassandra3 (or between any storage services).