Ions Tutorial

This tutorial will show you how to write applications that run entirely inside Datomic Cloud without any additional servers.

With ions you can build

  • Data-driven applications that are invoked via AWS Lambda
  • Web services accessed via AWS API Gateway for use by other applications or single page web apps

In this tutorial, you will build a simple inventory application with functions to add and query inventory, and then expose this application via both Lambda and AWS API Gateway. Along the way, you will learn how to develop, push, deploy and use your functions via:

The Day of Datomic Cloud and Datomic Ions videos discuss use of an Ion, with the code on Github.

Tutorial Prerequisites

Before this tutorial, you should

Setup

Clone the sample project

git clone https://github.com/Datomic/ion-starter.git

The ion-starter project contains a complete ion-based application. To begin the tutorial, clone the ion-starter project.

Set application name

Ions are deployed with an application name that must match your compute group. Edit the resources/datomic/ion-config.edn file and set the :app-name to your application name.

Install the ion-dev tools

Make sure you have installed the ion-dev tools.

Develop at the REPL

With ions, you can develop your application interactively at the REPL. In this tutorial, the application code is already written, so we will use the REPL to explore the code.

Configure Connection

The ion-starter project keeps connection arguments in a resource file resources/datomic/ion/starter/config.edn, Edit this file to include the connection arguments for your system.

Test Your Connection

The core of the tutorial application lives in the starter.clj namespace. Start a REPL from the root of the project to explore and test these functions.

First, load some namespace aliases:

(load-file "siderail/user.repl")

Verify that you can get a client object:

(def client (starter/get-client))

If this call succeeds, continue to the next step. Otherwise, return to Configure Connection.

Setup DB and load dataset

The ensure-sample-dataset function idempotently creates a database and loads the schema and some sample data. This function must be invoked once before using any of the tutorial data functions.

Review the ensure-sample-dataset function, and then run it at the REPL:

(starter/ensure-sample-dataset)
=> :loaded

Test data functions at the REPL

The inventory application exposes two simple data functions: get-schema returns the database schema, and get-items-by-type returns inventory items by type (:dress, :hat, :pants, or :shirt).

Review the data functions, and then test that these functions work at the REPL:

(def conn (starter/get-connection))


@(def db (d/db conn))
=> {:t 16, :next-t 17, :db-name "datomic-docs-tutorial", :database-id "ab481fe5-b6eb-4c97-8997-7d6a11809b6d", :type :datomic.client/db}
;; Your result may differ
(starter/get-schema db)
=>
(#:db{:id 39, :ident :fressian/tag, :valueType :db.type/keyword, :cardinality :db.cardinality/one, :doc "Keyword-valued attribute of a value type that specifies the underlying fressian type used for serialization."} #:db{:id 73, :ident :inv/sku, :valueType :db.type/string, :cardinality :db.cardinality/one, :unique #:db{:id 38, :ident :db.unique/identity}} #:db{:id 74, :ident :inv/color, :valueType :db.type/keyword, :cardinality :db.cardinality/one} #:db{:id 75, :ident :inv/size, :valueType :db.type/keyword, :cardinality :db.cardinality/one} #:db{:id 76, :ident :inv/type, :valueType :db.type/keyword, :cardinality :db.cardinality/one} #:db{:id 77, :ident :order/items, :valueType :db.type/ref, :cardinality :db.cardinality/many, :isComponent true} #:db{:id 78, :ident :item/id, :valueType :db.type/ref, :cardinality :db.cardinality/one} #:db{:id 79, :ident :item/count, :valueType :db.type/long, :cardinality :db.cardinality/one} #:db{:id 80, :ident :inv/count, :valueType :db.type/long, :cardinality :db.cardinality/one})
;; Your result may differ
(starter/get-items-by-type db :shirt '[:inv/sku :inv/color :inv/size])
=>
[[#:inv{:sku "SKU-0", :color :red, :size :small}] [#:inv{:sku "SKU-4", :color :red, :size :medium}] [#:inv{:sku "SKU-8", :color :red, :size :large}] [#:inv{:sku "SKU-12", :color :red, :size :xlarge}] [#:inv{:sku "SKU-16", :color :green, :size :small}] [#:inv{:sku "SKU-20", :color :green, :size :medium}] [#:inv{:sku "SKU-24", :color :green, :size :large}] [#:inv{:sku "SKU-28", :color :green, :size :xlarge}] [#:inv{:sku "SKU-32", :color :blue, :size :small}] [#:inv{:sku "SKU-36", :color :blue, :size :medium}] [#:inv{:sku "SKU-40", :color :blue, :size :large}] [#:inv{:sku "SKU-44", :color :blue, :size :xlarge}] [#:inv{:sku "SKU-48", :color :yellow, :size :small}] [#:inv{:sku "SKU-52", :color :yellow, :size :medium}] [#:inv{:sku "SKU-56", :color :yellow, :size :large}] [#:inv{:sku "SKU-60", :color :yellow, :size :xlarge}]]

Test lambda entry points at the REPL

A lambda entry point is just a function with a lambda-compatible signature that receives JSON input an returning an arbitrary string of output.

The lambda entry points are in a separate lambdas namespace, so that you can easily see the difference between domain functions and lambda data marshaling.

Review the lambda entry points, and then try them from the REPL.

(lambdas/get-schema nil)
=>
(#:db{:id 39,\n      :ident :fressian/tag,\n      :valueType :db.type/keyword,\n      :cardinality :db.cardinality/one,\n      :doc\n      \"Keyword-valued attribute of a value type that specifies the underlying fressian type used for serialization.\"}\n #:db{:id 73,\n      :ident :inv/sku,\n      :valueType :db.type/string,\n      :cardinality :db.cardinality/one,\n      :unique #:db{:id 38, :ident :db.unique/identity}}\n #:db{:id 74,\n      :ident :inv/color,\n      :valueType :db.type/keyword,\n      :cardinality :db.cardinality/one}\n #:db{:id 75,\n      :ident :inv/size,\n      :valueType :db.type/keyword,\n      :cardinality :db.cardinality/one}\n #:db{:id 76,\n      :ident :inv/type,\n      :valueType :db.type/keyword,\n      :cardinality :db.cardinality/one}\n #:db{:id 77,\n      :ident :order/items,\n      :valueType :db.type/ref,\n      :cardinality :db.cardinality/many,\n      :isComponent true}\n #:db{:id 78,\n      :ident :item/id,\n      :valueType :db.type/ref,\n      :cardinality :db.cardinality/one}\n #:db{:id 79,\n      :ident :item/count,\n      :valueType :db.type/long,\n      :cardinality :db.cardinality/one}\n #:db{:id 80,\n      :ident :inv/count,\n      :valueType :db.type/long,\n      :cardinality :db.cardinality/one})
(lambdas/get-items-by-type {:input (json/write-str "shirt")})
=>
[[#:inv{:sku \"SKU-0\", :size :small, :color :red}]\n [#:inv{:sku \"SKU-4\", :size :medium, :color :red}]\n [#:inv{:sku \"SKU-8\", :size :large, :color :red}]\n [#:inv{:sku \"SKU-12\", :size :xlarge, :color :red}]\n [#:inv{:sku \"SKU-16\", :size :small, :color :green}]\n [#:inv{:sku \"SKU-20\", :size :medium, :color :green}]\n [#:inv{:sku \"SKU-24\", :size :large, :color :green}]\n [#:inv{:sku \"SKU-28\", :size :xlarge, :color :green}]\n [#:inv{:sku \"SKU-32\", :size :small, :color :blue}]\n [#:inv{:sku \"SKU-36\", :size :medium, :color :blue}]\n [#:inv{:sku \"SKU-40\", :size :large, :color :blue}]\n [#:inv{:sku \"SKU-44\", :size :xlarge, :color :blue}]\n [#:inv{:sku \"SKU-48\", :size :small, :color :yellow}]\n [#:inv{:sku \"SKU-52\", :size :medium, :color :yellow}]\n [#:inv{:sku \"SKU-56\", :size :large, :color :yellow}]\n [#:inv{:sku \"SKU-60\", :size :xlarge, :color :yellow}]]

Test HTTP entry points at the REPL

An HTTP entry point is just a function with an HTTP-compatible signature that receives input and output maps that describe web requests and responses.

The HTTP entry points are in separate http namespace. Review the HTTP entry points, and then try them from the REPL.

(http/get-items-by-type {:body (s-edn/input-stream :shirt)})
=>
{:status 200, :headers {"Content-Type" "application/edn"}, :body "[[#:inv{:sku \"SKU-0\", :size :small, :color :red}]\n [#:inv{:sku \"SKU-4\", :size :medium, :color :red}]\n [#:inv{:sku \"SKU-8\", :size :large, :color :red}]\n [#:inv{:sku \"SKU-12\", :size :xlarge, :color :red}]\n [#:inv{:sku \"SKU-16\", :size :small, :color :green}]\n [#:inv{:sku \"SKU-20\", :size :medium, :color :green}]\n [#:inv{:sku \"SKU-24\", :size :large, :color :green}]\n [#:inv{:sku \"SKU-28\", :size :xlarge, :color :green}]\n [#:inv{:sku \"SKU-32\", :size :small, :color :blue}]\n [#:inv{:sku \"SKU-36\", :size :medium, :color :blue}]\n [#:inv{:sku \"SKU-40\", :size :large, :color :blue}]\n [#:inv{:sku \"SKU-44\", :size :xlarge, :color :blue}]\n [#:inv{:sku \"SKU-48\", :size :small, :color :yellow}]\n [#:inv{:sku \"SKU-52\", :size :medium, :color :yellow}]\n [#:inv{:sku \"SKU-56\", :size :large, :color :yellow}]\n [#:inv{:sku \"SKU-60\", :size :xlarge, :color :yellow}]]\n"}

Note that the REPL tests provide only the parts of the request map that are actually needed. This optionality is a benefit over more static systems that might require mocking and stubbing.

Now you are ready to push and deploy your ion to Datomic Cloud.

Push

The push operation creates an application revision in S3 and AWS CodeDeploy that can then be deployed multiple times to separate systems.

Configure entry points

When you push an application revision, the resources/datomic/ion-config.edn file specifies the applications entry points:

  • the :lambda section specifies functions that will be callable via AWS Lambda
  • the :http-direct section specifies functions that will be callable via HTTP
  • the :allow section specifies functions that will be callable from inside Datomic transactions or queries

For this tutorial, the entry points are already specified. Review them before you push.

Commit your changes

By default, push requires a clean git repository, and uses the git SHA to name the application revision in AWS. Such a push is reproducible. To prepare for a reproducible commit, you should git add files that are part of your application, git ignore any extraneous files, and git commit. If you are precisely following the tutorial, you have only changed resource files to set your application name and configure a client.

git add resources
git commit -m 'application name and client config'

To double check that you have a clean commit, you can run git status:

git status
=> ... details elided ...
nothing to commit, working tree clean

Push a revision

Now you are ready to push an application revision!

clojure -A:ion-dev '{:op :push}'

The push operation will report progress on the console. If it succeeds, it will print an edn map that includes a :deploy-command. Congratulations, you have pushed an app!

If push fails, don't worry. Use the push troubleshooting guide to diagnose and fix any problems you encounter.

Deploy

The deploy operation installs your pushed application on a Datomic compute group.

Let's deploy the revision you just pushed. The output of a successful push will include the deploy command under the :deploy-command key. You can copy the value of this key, remove any escaping, and use it to deploy your Ion.

The format for a deploy command is:

clojure -A:ion-dev '{:op :deploy :rev $(REV) :group $(GROUP)}'

Replace $(GROUP) with the name of your compute group, and $(REV) with the :rev output by push.

On success, a deploy will print an edn map that includes a :status-command. Congratulations, your app revision is now deploying!

If deploy fails, don't worry. Use the deploy troubleshooting guide to diagnose and fix any problems.

Monitor Your Deployment

The deploy operation does the minimal work necessary to ensure that your application revision is running on all your compute nodes. This can take anywhere from a few seconds to several minutes.

The output of an Ion deploy includes the command to monitor your deployment. The command is located in the deploy output's :status-command key. You can copy the value of this key, remove any escaping, and use it to monitor your Ion deploy.

The format to monitor deploy-status:

clojure -A:ion-dev '{:op :deploy-status :execution-arn $(EXECUTION_ARN)}'

Replace $(EXECUTION_ARN) with the :execution-arn output by deploy.

This will return the status of the AWS Code Deploy and the overall ion deploy, with "SUCCEEDED" indicating when each process has successfully completed:

{:deploy-status "SUCCEEDED", :code-deploy-status "SUCCEEDED"}

If your deploy-status doesn't reach SUCCEEDED, don't worry. Use the deploy-status troubleshooting guide to diagnose and fix and problems.

Entry Points

There are many entry points for Ions.

This section covers:

Invoke a Lambda

Lambda entry points can be invoked from the AWS CLI. First, call get-schema. In the command below, replace $(GROUP) with the name of your compute group:

aws lambda invoke --function-name $(GROUP)-get-schema --payload '' /dev/stdout

This may return pending while AWS creates or configures external resources.

On success, this call will print your lambda's response, plus some AWS CLI status information:

(#:db{:id 39,
      :ident :fressian/tag,
      :valueType :db.type/keyword,
      :cardinality :db.cardinality/one,
      :doc
      "Keyword-valued attribute of a value type that specifies the underlying fressian type used for serialization."}
 ;; more schema elided
 #:db{:id 80,
      :ident :inv/count,
      :valueType :db.type/long,
      :cardinality :db.cardinality/one})
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

On your first invocation, this call may take a while due to Lambda cold start. Outside of dev, most lambda invocations are warm, so try calling the function a second time to see typical production performance.

To see a function that takes an argument, try get-items-by-type.

If aws --version returns version 2, then add the flag --cli-binary-format raw-in-base64-out to the command:

aws lambda invoke --function-name $(GROUP)-get-items-by-type --payload '"shirt"' /dev/stdout

On success, this will return details about shirts:

[[#:inv{:sku "SKU-28", :size :xlarge, :color :green}]
 ;; more shirt details
 [#:inv{:sku "SKU-56", :size :large, :color :yellow}]]
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Add an Attribute Predicate

You can use ions to deploy functions for use as attribute predicates, entity predicates, transaction functions, or query functions.

When you deployed the tutorial application, your ion-config.edn file allowed a valid-sku? function suitable for an attribute predicate.

(defn valid-sku?
  [s]
  (boolean (re-matches #"SKU-(\d+)" s)))

You can now test this function at the REPL:

(load-file "siderail/user.repl")

;; try the fn at the REPL
(attrs/valid-sku? "SKU-112")
=> true
(attrs/valid-sku? "SKU-1B")
=> false

Then you can install valid-sku as a predicate for :inv/sku:

(def conn (starter/get-connection))

(def tx [{:db/ident :inv/sku
          :db.attr/preds 'datomic.ion.starter.attributes/valid-sku?}])
(d/transact conn {:tx-data tx})

After you install the function, you can use a with-db to prove that the predicate is in effect:

(def with-db (d/with-db conn))
(d/with with-db {:tx-data [{:db/id "should-not-work"
                            :inv/sku "not-a-sku"}]})
=> Entity -9223301668109598141 attribute :inv/sku value not-a-sku failed pred datomic.ion.starter.attributes/valid-sku?

HTTP Direct

Find your IonApiGatewayEndpoint in your compute group's CloudFormation template outputs.

Substitute for your Ion API Gateway Endpoint URL in the following curl command to invoke your Ion through HTTP Direct:

curl https://$(IonApiGatewayEndpoint) -d :hat
=>
#{{:color :red, :type :hat, :size :medium, :sku "SKU-7"}
  ...
  {:color :blue, :type :hat, :size :small, :sku "SKU-35"}}

On success, this will return an EDN representation of inventory data for a particular type.

Conclusion

In this tutorial, you have seen how ions can take ordinary Clojure functions, and install them in Datomic Cloud where they have in-memory access to Datomic data. You have used ions to deliver:

There is a lot more to explore! Here are some things to try: