Ions Reference

Ion applications are described by a standard Clojure deps.edn file. Add:

• com.datomic/ion as a runtime dependency
• com.datomic/ion-dev under a :dev alias
• a "datomic-cloud" entry under your :mvn/repos

;; add the following to get the latest ion release
{:deps {com.datomic/ion {:mvn/version "0.9.28"}}
 :mvn/repos {"datomic-cloud" {:url "s3://datomic-releases-1fc2183a/maven/releases"}}
 :aliases
  {:dev {:extra-deps {com.datomic/ion-dev {:mvn/version "0.9.186"}}}}}

See ion-starter's deps.edn for an example.

An ion function is a plain function, with a signature based on its intended use:

ion type	input args	return
transaction fn	db + data	transaction data
query fn	data	data
lambda	:input JSON, :context map	String, InputStream, ByteBuffer, or File
web service	web request	web response

To write an ion, implement a function with one of these signatures. A single ion application can have any number of ions of any types.

The datomic.ion.starter namespace has examples of each type of ion signature.

An ion application must have a resource named datomic/ion-config.edn on its classpath. This resource tells Datomic how to invoke your application:

• the :app-name key names the CodeDeploy application. If you did not override this name when creating your Datomic compute group, it will default to your system name.
• the :allow key whitelists ions that Datomic is allowed to call. When you deploy an application, Datomic will automatically require all the namespaces mentioned in the :allow section.
• the :lambdas section configures lambda entry points for your application.

The example below is taken from the ion-starter project.

{:allow [;; transaction function
         datomic.ion.starter/inc-attr

         ;; query function
         datomic.ion.starter/modes

         ;; lambda handler
         datomic.ion.starter/echo

         ;; web application
         datomic.ion.starter/get-tutorial-schema    
         ]            
 :lambdas {:echo
           {:fn datomic.ion.starter/echo
            :description "Echos input"}
           :get-tutorial-schema
           {:fn datomic.ion.starter/get-tutorial-schema
            :description "returns the schema for the Datomic docs tutorial"}}
 :app-name "<YOUR-APP-HERE>"}

(defn echo
  [{:keys [context input]}]
  input)

When you are developing ions locally, you will want to connect to a Datomic cloud system as a client. But when you deploy ions, you will want the same code to utilize an in-memory implement of client. The :ion server-type implements this behavior, connecting as a client when remote, and providing an in-memory implemention of client when running in Datomic Cloud.

NOTE Ions support the use of the synchronous Datomic Client API. You should not use the asynchronous Client API in Ions.

The following example shows how to create a client for an ion application:

(require '[datomic.client.api :as d])

(def cfg {:server-type :ion
          :region "<your AWS Region>" ;; e.g. us-east-1
          :system "<system-name>"
          :endpoint "http://entry.<system-or-query-group-name>.<region>.datomic.net:8182/"
          :proxy-port <local-port for SSH tunnel to bastion>})

(def client (d/client cfg))

The datomic.ion.dev main function takes the following arguments for push:

keyword	required	value	example
:op	yes	:push	:push
:uname	no	string	"janes-wip"
:creds-profile	no	string	"janes-profile"
:region	no	string	"us-east-1"

You can specify a Named Profile by passing it as :creds-profile. You can override your environment's default region by providing :region.

As a development convenience, Ions support an unreproducible push, allowing you to deploy work-in-progress code not committed to git. Since there is no git SHA, you must specify an unrepro name (:uname) when pushing. You are responsible for making the uname unique within your org+region+app.

It is ok to push to the same :uname repeatedly, so long as you understand that the last push wins.

Push will ensure that git and maven libs exist under

s3://$(datomic-code-bucket)/datomic/libs

and will create a CodeDeploy application revision located at

s3://$(datomic-code-bucket)/datomic/apps/$(app-name)/$(git-sha).zip

or at

s3://$(datomic-code-bucket)/datomic/apps/$(app-name)/$(uname).zip

if a :uname is specified.

On success, push returns a map with the following keys:

keyword	required	value
:rev	no	git SHA for the commit that was pushed
:uname	no	unreproducible name for the push
:deploy-groups	yes	list of available groups for deploy
:deploy-command	yes	sample command for deploy
:doc	no	documentation
:dependency-conflicts	no	map describing conflicts

Because ions run on the same classpath as Datomic Cloud, it is possible for ion dependencies to conflict with Datomic's own dependencies. If this happens:

• Datomic's dependencies will be used
• The return from push will warn you with a :dependency-conflicts map
• You can add the :deps from the conflicts map to your local project so that you can test against the libraries used by Datomic Cloud.

The Datomic team works to keep Datomic's dependencies up-to-date. If you are unable to resolve a dependency conflict, please contact support.

The datomic.ion.dev main function takes the following arguments for deploy:

keyword	required	value	example
:op	yes	:deploy	:deploy
:group	yes	compute group name	"my-datomic-compute"
:rev	no	output from push	"6468765f843d70e01a7a2e483405c5fcc9aa0883"
:uname	no	input to push	"janes-wip"
:creds-profile	no	string	"janes-profile"
:region	no	string	"us-east-1"

The :group key takes compute group name as a value. If you followed the upgrade instructions and are running separate storage and compute stacks the name of your compute group is the name you gave the compute stack when you created it.

However, if you launched from AWS Marketplace, you are running a nested stack with an automatically generated compute stack name. To find the name of your compute stack, go to the AWS CloudFormation console and look through the list of Stack Names. The name of your compute stack will be $(SystemName)-Compute-$(GeneratedId). Use that whole stack name as the parameter to :group.

You can specify a Named Profile by passing it as :creds-profile. You can override your environment's default region by providing :region.

When you deploy an ion application, Datomic will use an AWS Step Machine to

• CodeDeploy the code for the application onto each instance in the compute group.
• Ensure that active databases are present in memory before routing requests to newly updated nodes.
• Ensure that all the lambdas requested via the ion-config.edn map exist and are configured correctly.

The time to deploy to a single node is the sum of

• the time to stop and restart the Datomic Cloud process
• the time to load active databases into memory

Deployment to a single node can take from 20 seconds up to several minutes, depending on the number and size of active databases.

Node deployment happens one node at a time, so a Datomic system will remain available with N-1 members active throughout a deployment.

The time to complete an entire deployment is the sum of

• the sum of times to deploy to each node
• the time to (re)configure lambdas

keyword	required	value	example
:op	yes	:deploy-status	:deploy-status
:execution-arn	yes	output from deploy	"arn:aws:states:us-east-1:123456789012:execution:datomic-compute:datomic-compute-1526506240469"
:creds-profile	no	string	"janes-profile"
:region	no	string	"us-east-1"

You can specify a Named Profile by passing it as :creds-profile. You can override your environment's default region by providing :region.

Deploy Status returns the status of the overall ion deploy and Code Deploy as a map:

{:deploy-status "SUCCEEDED"
 :code-deploy-status "SUCCEEDED"}

Both the overall ion deploy and Code Deploy status will return with "SUCCEEDED" indicating when each process has successfully completed. Utilize :deploy-status when trying to determine when the next deployment step should occur.

See the Step Functions reference for possible values of the status keys.

You can monitor the deployment in near real-time from the Step Functions Console. Look for a state machine named "datomic-$(group)".

A web ion is a function that takes the following input map:

key	req'd	value	example
:body	no	InputStream	"hello"
:headers	yes	map string->string	{"x-foo" "bar"}
:protocol	yes	HTTP protocol	"HTTP/1.1"
:remote-addr	yes	caller host	"example.com"
:request-method	yes	HTTP verb as keyword	:get
:scheme	no
:server-name	yes	server host name	"example.com"
:server-port	no	TCP port	443
:uri	yes	request URI
:json	no	api-gateway/json	{"resource":"{proxy+}","path":"datomic","httpMethod":"POST" … }
:data	no	api-gateway/data	{"Path": "/datomic","Querystringparameters": null, "Pathparameters": {"Proxy": "datomic"}…

NOTE datomic.ion.edn.api-gateway/json and datomic.ion.edn.api-gateway/data keys are only present when invoked through api-gateway.

and returns

key	req'd	value	example
:body	no	InputStream or String	"goodbye"
:headers	yes	map string->string	{"x-foo" "bar"}
:status	yes	HTTP status code	200

The input and output maps are mostly compatible with the Clojure Ring Spec, and many web applications should be able to run unmodified as web ions.

The ion-starter project includes an example web ion.

Web services are exposed as lambda proxies through the AWS API Gateway. The ion library includes a helper function datomic.ion.lambda.api-gateway/ionize that will convert a web ion into a lambda ion, as demonstrated in the ion-starter project:

(def get-tutorial-schema
  "API Gateway web service ion for tutorial-schema-handler."
  (apigw/ionize tutorial-schema-handler))

The lambda ion is then configured like any other lambda. You can then use the API Gateway to:

• deploy an API to the internet
• manage content encodings and compression
• validate requests
• configure access control
• issue and manage API keys

The ions tutorial includes an example that deploys a web ion to the internet.

The datomic.ion namespace of the com.datomic/ion library provides three functions for reading parameters from Parameter Store:

• get-env returns an environment map that you specify in CloudFormation.
• get-app-info returns information about your application.
• get-params retrieves a collection of params, using the results from get-app-info and get-env to form a unique path.

The deploy-monitor application shows all of these functions in an idiomatic example.

You configure an environment map when you first create a compute group. If you want to change the environment map after the group is created, you can perform a parameter upgrade.

For example, the environment map below specifies that this compute group is for a :staging deployment environment:

The get-env function takes no arguments, and returns the environment map.

For example, the following code retrieves the deployment environment specified above:

(get (ion/get-env) :env)

When running outside Datomic Cloud, get-env returns the value of the DATOMIC_ENV_MAP environment variable, read as edn.

The get-app-info function takes no arguments, and returns a map that will include at least these keys:

• :app-name is your app-name
• :deployment-group is the name of your Ion deployment group

For example, the following excerpt from deploy-monitor gets the app-name:

(get (ion/get-app-info) :app-name)

When running outside Datomic Cloud, get-app-info returns the value of the DATOMIC_APP_INFO_MAP environment variable, read as edn.

The get-params function is a convenience abstraction over GetParametersByPath. get-params take a map with a :path key, and it returns all the parameters under path as a map from parameter name string to parameter value string, decrypting if necessary.

For example, the call to get-params below returns all the parameters under the path /datomic-shared/prod/deploy-monitor.

(ion/get-params {:path "/datomic-shared/prod/deploy-monitor"})

The deploy-monitor application demonstrates the naming convention: /datomic-shared/(env)/(app-name)/(key), where

• datomic-shared is a Parameter Store prefix readable by all Datomic Cluster Nodes.
• env differentiates different environments for the same application, e.g. "ci" vs. "prod".
• app-name is your Ion app-name.
• The various keys are application specific.

NOTE The datomic-shared prefix is readable by any Datomic system. If you want more granular permissions, you can choose your own naming convention (under a different prefix!), and explicitly add permissions to the IAM policy for your Datomic nodes.

The deploy-monitor talks to two external resources: a Datomic database and the Slack API. It needs four parameters:

• a Datomic database name
• a Slack channel
• two encrypted tokens for Slack

Given this convention, the following AWS CLI commands will create the parameters for the "prod" environment:

# actual parameter values not shown
aws ssm put-parameter --name /datomic-shared/prod/deploy-monitor/db-name --value $DB_NAME --type String
aws ssm put-parameter --name /datomic-shared/prod/deploy-monitor/channel --value $CHANNEL --type String
aws ssm put-parameter --name /datomic-shared/prod/deploy-monitor/bot-token --value $BOT_TOKEN --type SecureString
aws ssm put-parameter --name /datomic-shared/prod/deploy-monitor/verification-token --value $VERIFICATION_TOKEN --type SecureString

You could use similar commands to create parameters for additional environments.

NOTE You should not store AWS credentials in the Parameter Store, as Datomic Cloud fully supports IAM roles.

At runtime, deploy-monitor uses get-app-info and get-env to load the information needed to create a Parameter Store path, and then reads all of its parameter values with get-params:

(def get-params
  "Returns the params under /datomic-shared/(env)/(app-name)/, where

env       value of get-env :env
app-name  value of get-app-info :app-name"
  (memoize
   #(let [app (or (get (ion/get-app-info) :app-name) (fail :app-name))
          env (or (get (ion/get-env) :env) (fail :env))]
      (ion/get-params {:path (str "/datomic-shared/" (name env) "/" app "/")}))))