Ions Reference

Every ion project should depend on two libraries from the datomic-cloud maven repo:

• the com.datomic/ion library contains functions that ions use at runtime, and should be added as a direct dependency
• the com.datomic/ion-dev library contains functions used at ion development time, and should be added under a :dev alias

The deps.edn snippet below adds both libraries and the datomic-cloud maven repo, and should be merged into every ion project.

;; add the following to get the latest ion release
{:deps {com.datomic/ion {:mvn/version "0.9.34"}}
 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}
 :aliases
  {:dev {:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.229"}}}}}

Every ion application must have a resource named datomic/ion-config.edn on its classpath. This resource is included in deployment and is used at runtime to configure the application's entry points. ion-config is an edn map with the following keys:

• :app-name is string name of a Datomic application
• :allow is a vector of fully qualified symbols naming query or transaction functions. When you deploy an application, Datomic will automatically require all the namespaces mentioned under :allow.
• :lambdas is a map from lambda names (keywords) to lambda configurations.
• :http-direct is a map configuring HTTP Direct entry points.

The environment map is typically used only to distinguish an environment name, and the environment name is then later used to locate one or more parameters.

You configure an environment map when you first create a compute group. If you want to change the environment map after the group is created, you can perform a parameter upgrade.

For example, the environment map below specifies that this compute group is for a :staging deployment environment:

At runtime, applications need access to configuration data such as the name of a Datomic database. There are several challenges to consider:

• Configuration values have a lifecycle that is independent of application source code. They should not be hard-coded in the application, and should be managed separately from source code.
• Applications need a way to obtain their configuration values at runtime.
• Configuration values may be sensitive, and should not be stored or conveyed as plaintext
• Configuration values may need to be secured at a granular level.

Parameters are a solution to these challenges:

• A parameter is a named slot known to application code that can be filled in with a specific parameter value at runtime.
• The AWS Systems Manager Parameter Store provides an implementation of parameters that supports an independent lifecycle, encryption for sensitive data, and IAM security over a hierarchical naming scheme.
• All Datomic cluster nodes have read permission on parameter store keys that begin with datomic-shared.
• The ion library provides a set of functions for using parameters.

You are also free to use the AWS parameter API directly, and customize Datomic node permissions to match your own idiomatic use.

NOTE The datomic-shared prefix is readable by any Datomic system. If you want more granular permissions, you can choose your own naming convention (under a different prefix!), and explicitly add permissions to the IAM policy for your Datomic nodes.

Local testing of Ions should be performed with the same JVM version and memory settings used by Datomic Cloud:

Datomic runs on Java version 1.8.0

Additionally all instances use the following flags:

-XX:+UseG1GC -XX:MaxGCPauseMillis=50 -Dclojure.spec.skip-macros=true

When you are developing ions locally, you will use a remote connection to a system. But when you deploy ions, you will want the same code to utilize an in-memory implement of client. The :ion server-type implements this behavior, connecting as a client when remote, and providing an in-memory implemention of client when running in Datomic Cloud.

The following example shows how to create a client that can be used both for local development and when deployed in an ion application.

(require '[datomic.client.api :as d])

(def cfg {:server-type :ion
          :region "<your AWS Region>" ;; e.g. us-east-1
          :system "<system-name>"
          :endpoint "http://entry.<system-or-query-group-name>.<region>.datomic.net:8182/"
          :proxy-port <local-port for SSH tunnel to access gateway>})

(def client (d/client cfg))

A lambda ion takes a map with two keys:

• :input is a String containing the input JSON payload.
• :context contains all the data fields of the AWS Lambda context as a Clojure map, with all keys (inluding nested and dynamic map keys) converted to keywords.

A lambda ion can return any of String, ByteBuffer, InputStream, or File; or it can throw an exception to signal an error.

For example, the following echo function simply echos back its invocation:

(defn echo
  [{:keys [context input]}]
  input)

A web ion is a function that takes the following input map:

NOTE datomic.ion.edn.api-gateway/json and datomic.ion.edn.api-gateway/data keys are only present when invoked through api-gateway.

and returns

The input and output maps are mostly compatible with the Clojure Ring Spec, and many web applications should be able to run unmodified as web ions.

The ion-starter project includes an example web ion.

Web ions can be exposed as lambda proxies through the AWS API Gateway. The ion library includes a helper function datomic.ion.lambda.api-gateway/ionize that will convert a web ion into a lambda proxy ion, as demonstrated in the ion-starter project:

(def get-tutorial-schema
  "API Gateway web service ion for tutorial-schema-handler."
  (apigw/ionize tutorial-schema-handler))

Datomic Cloud does not automate API Gateway configuration for lambda proxies, but the ions tutorial includes an example that you can use as a starting point.

The push operation creates a revision of your application in S3. Ion push understands your dependencies at a graular level, so e.g. all revivisions can share the same copy of a common library. This is more efficient than "uberjar" style deployment.

To push an application revision, call datomic.ion.dev with an :op of :push:

clojure -Adev -m datomic.ion.dev '{:op :push (options)}'

Push will ensure that git and maven libs exist under

s3://$(datomic-code-bucket)/datomic/libs

and will create a CodeDeploy application revision located at

s3://$(datomic-code-bucket)/datomic/apps/$(app-name)/$(git-sha|uname).zip

On success, push returns a map with the following keys:

The deploy operation deployes a pushed revision to a compute group.

When you deploy, Datomic will use an AWS Step Machine to

• CodeDeploy the code for the application onto each instance in the compute group.
• Automatically roll back to the previous deployment if the application does not deploy correctly (e.g. if loading a namespace throws an exception.)
• Ensure that active databases are present in memory before routing requests to newly updated nodes.
• Ensure that all the lambdas requested via the ion-config.edn map exist and are configured correctly.

Node deployment happens one node at a time, so a Datomic system will remain available with N-1 members active throughout a deployment.

To deploy an application, call datomic.ion.dev with an :op of :deploy:

clojure -Adev -m datomic.ion.dev '{:op :deploy (options)}'

The time to complete an entire deployment is the sum of

• the sum of times to deploy to each node
• the time to (re)configure lambdas

The time to deploy to a single node is the sum of

• the time to stop and restart the Datomic Cloud process
• the time to load active databases into memory

Deployment to a single node can take from 20 seconds up to several minutes, depending on the number and size of active databases.

You can monitor a deployment from the command line or from the AWS Console.

When you deploy an application with lambda ions, Datomic will create AWS Lambdas named:

$(group)-$(name)

where group is the :group key you used to invoke deploy, and name is the name under the :lambdas key in ion-config.edn.