Ions Tutorial

This tutorial will show you how to write applications that run entirely inside Datomic Cloud without any additional servers.

With ions you can build

  • data-driven applications that are invoked via AWS Lambda
  • web services accessed via AWS API Gateway for use by other applications or single page web apps

In this tutorial, you will build a simple inventory application with functions to add and query inventory, and then expose this application as both a data-driven application and a web service.

Prerequisites

  • Read the Ions overview.
  • Ions require Datomic Cloud Version 397 (May 31,2018) or later. Follow the instructions to create a new Datomic Cloud system, or to upgrade an existing system.
  • Run this tutorial in an environment with proper AWS access keys.
  • This tutorial presumes AWS Administrator permissions.
  • You will need to know the name of your compute stack, see the Ion Deploy reference for details on finding the name of your compute stack.

Setup

The ion-starter project contains a complete ion-based application. To begin the tutorial, clone the ions-starter project.

git clone git@github.com:Datomic/ion-starter.git

Set System Name

The configuration for ion-starter needs one piece of information from you: the CodeDeploy application name for your system (by default, this is the same as your system name). Copy the resources/datomic/ion-config-sample.edn file to resources/datomic/ion-config.edn. Edit the ion-config.edn file and set the :app-name to the name of your Datomic system.

Adding an Item

Your first task is to provide add-item constructor to create a new inventory item. This function will implement the signature for ion lambdas, taking a JSON string in :input and returning whatever you want. (The implementation shown below returns an EDN map with the database t after the item is added.)

(defn add-item
  "Lambda ion that adds an item, returns database t."
  [{:keys [input]}]
  (let [args (-> input json/read-str)
        conn (get-connection)
        tx [(list* 'datomic.ion.starter/create-item args)]
        result (d/transact conn {:tx-data tx})]
    (pp-str {:t (-> result :db-after :t)})))

Note that add-item uses a custom transaction function called create-item. This is a function of data to data, with the current database value as its first argument:

(defn create-item
  "Transaction fn that creates data to make a new item"
  [db sku size color type]
  [{:inv/sku sku
    :inv/color (keyword color)
    :inv/size (keyword size)
    :inv/type (keyword type)}])

Because this is pure function, it is easy to develop and test at the REPL.

Before your can use add-item, you need to push and deploy it.

Push

The push operation creates an application revision in S3 and AWS CodeDeploy that can then be deployed multiple times to separate systems.

Push the project:

clojure -A:dev -m datomic.ion.dev '{:op :push}'

On success, the push command will return the :rev that names this application, which is equal to the git SHA for the current commit.

Deploy

The deploy operation handles everything needed to update an application with zero downtime. Using AWS CodeDeploy and AWS Step Functions, Deploy installs code, configures AWS Lambdas, and cycles processes.

Let's deploy the revision you just pushed. In the command below, replace $(GROUP) with the name of your compute stack, and $(REV) with the :rev output by push.

clojure -A:dev -m datomic.ion.dev '{:op :deploy :rev "$(REV)" :group "$(GROUP)"}'

Monitor Your Deployment

Deploy does the minimal work necessary to move your system to its new configuration, using a step function to CodeDeploy the application and ensure that the lambdas are correctly configured. You can monitor the deployment with the following command, replacing $(EXECUTION_ARN) with the :execution-arn output by deploy:

clojure -A:dev -m datomic.ion.dev '{:op :deploy-status :execution-arn "$(EXECUTION_ARN)"}'

This will return the status of the Code Deploy and the overall ion deploy, with "SUCCEEDED" indicating when each process has successfully completed:

{:deploy-status "SUCCEEDED", :code-deploy-status "SUCCEEDED"}

Try It

Now you are ready to invoke add-item from the AWS CLI. In the command below, replace GROUP with the name of your compute stack:

aws lambda invoke --function-name $(GROUP)-add-item --payload '["SKU-12345", "small", "red", "hat"]' /dev/stdout

On success, the basis-t following the completion of the transaction will be echoed to stdout:

{:t 16}
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

That is all there is to it. A Datomic Cloud system can implement an arbitrary number of lambdas that you develop interactively, in Clojure, at a REPL.

Querying the Database

Users of the system want to query for items by type. The items-by-type lambda returns information about items of a particular type:

(defn items-by-type*
  "Returns info about items matching type"
  [db type]
  (d/q '[:find ?sku ?size ?color ?featured
         :in $ ?type
         :where
         [?e :inv/type ?type]
         [?e :inv/sku ?sku]
         [?e :inv/size ?size]
         [?e :inv/color ?color]
         [(datomic.ion.starter/feature-item? $ ?e) ?featured]]
       db type))

(defn items-by-type
  "Lambda ion that returns sample database items matching type."
  [{:keys [input]}]
  (-> (items-by-type* (d/db (get-connection))
                      (-> input json/read-str keyword))
      pp-str))

Note that items-by-type lambda uses a custom query function, feature-item?, to determine if an item is currently being featured. Such a query function is ordinary Clojure code, developed at a REPL:

(defn feature-item?
  "Query ion exmaple. This predicate matches entities that
should be featured in a promotion."
  [db e]
  (let [{:keys [inv/color inv/size inv/type]} (d/pull db {:eid e :selector [:inv/color :inv/size :inv/type]})]
    (and (= (:db/ident color) :green)
         (= (:db/ident size) :xlarge)
         (= (:db/ident type) :hat))))

Try It

If you ran the push and deploy steps for add-item above, then items-by-type is also deployed and ready to go. (Push and deploy work with an entire application that can include many ions.)

The following invocation of the lambda will query for all hats in the database. Replace $(GROUP) in the command below with the name of your compute stack:

aws lambda invoke --function-name $(GROUP)-items-by-type --payload \"hat\" /dev/stdout

On success, the results of the query will be echoed to stdout:

#{["SKU-19" 28499341391953988 23934169113428033 false] 
  ...
  ["SKU-55" 46139905947992133 32330039903125571 false]}
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Creating a Web Service

The inventory system must also support a single-page webapp that displays inventory by type. You can use AWS API Gateway to turn an ion lambda into a web service.

Create a Web Service Ion

The lambda for a web service is slightly different from a data-driven lambda. A web ion takes a web request and returns a web response. Here is the web version of items-by-type:

(defn items-by-type-web*
  "Lambda ion that returns sample database items matching type."
  [{:keys [headers body]}]
  (let [type (some-> body read-edn)]
    (if (keyword? type)
      {:status 200
       :headers {"Content-Type" "application/edn"} 
       :body (-> (items-by-type* (d/db (get-connection)) type)
                 pp-str)}
      {:status 400
       :headers {}
       :body "Expected a request body keyword naming a type"})))

For API Gateway, this web function must be converted to the signature for Lambda Proxies. The ion support library provides the helper function api-gateway/ionize for this:

(def items-by-type-web
  "API Gateway web service ion for items-by-type"
  (apigw/ionize items-by-type-web*))

If you ran the push and deploy steps for add-item above, then items-by-type-web is also deployed and ready to go. Let's expose it via API Gateway.

Create an API in API Gateway

In this step you will create an API backed by an AWS Lambda Proxy

  • Go to the AWS API Gateway console to create a new AWS API Gateway.
    • If this is the first AWS API Gateway you are creating, choose Get Started, then click OK and choose thew New API radio button.
    • If you have existing AWS API Gateways, choose Create API.
  • Give your API a unique name, e.g. "ion-get-schema". Leave the other default settings.
  • Click Create API in the bottom right.
  • Under the Actions dropdown, choose Create Resource.
  • Check the box "Configure as proxy resource." Other fields will be updated with defaults. Leave these as is.
  • Click Create Resource.

Connect API Gateway to Datomic Cloud

To connect API Gateway to items-by-type-web:

  • Set the Lambda Function to ${GROUP)-items-by-type-web and click Save. Note that the autocomplete does not always work correctly on this step, so trust and verify your own spelling.
  • Choose OK to give API Gateway permission to call your lambda.
  • Under your API in the left side of the UI, click on the bottom choice Settings.
  • Choose Add Binary Media Type, and add the */* type, then Save Changes.

Deploy

NOTE This step of the tutorial will expose an ion on the public internet.

  • Click on your API name, and choose Deploy API under the Actions dropdown.
  • Choose "New Stage" as the Deployment Stage, add the Stage Name "dev", then choose Deploy.

The top of the Stage Editor will show the invoke URL for your deployed app. Call your web service via curl to make sure everything looks ok:

curl https://$(obfuscated-name).execute-api.us-east-1.amazonaws.com/dev -d :hat

On success, this will return an EDN representation of inventory data for a particular type:

#{["SKU-19" 28499341391953988 23934169113428033 false] 
   ... 
  ["SKU-55" 46139905947992133 32330039903125571 false]}

Conclusion

In this tutorial, you have seen how ions can take ordinary Clojure functions, and install them in Datomic Cloud where they have in-memory access to Datomic data. You have used ions to deliver

  • a data-driven application via AWS Lambda
  • a web service via API Gateway

all hosted out of a single Datomic Cloud system with no additional servers.