Ions Tutorial

This tutorial will show you how to write applications that run entirely inside Datomic Cloud without any additional servers.

With ions you can build

  • Data-driven applications that are invoked via AWS Lambda
  • Web services accessed via AWS API Gateway for use by other applications or single page web apps

In this tutorial, you will build a simple inventory application with functions to add and query inventory, and then expose this application via both Lambda and AWS API Gateway. Along the way, you will learn how to develop, push, deploy and use your functions via:

Tutorial Prerequisites

Before this tutorial, you should

  • Read the Ions overview.
  • Know how to work with a git repository.
  • Install Datomic Cloud Version 477-8741 or later. Follow the instructions to create a new Datomic Cloud system, or to upgrade an existing system.
  • Run in an environment with proper AWS access keys.
  • Have AWS Administrator permissions.
  • Know the name of your compute stack. See the Ion Deploy reference for details on finding the name of your compute stack.

Setup

git clone https://github.com/datomic/ion-starter

The ion-starter project contains a complete ion-based application. To begin the tutorial, clone the ions-starter project.

Set System Name

The configuration for ion-starter needs one piece of information from you: the CodeDeploy application name for your system (by default, this is the same as your system name). Copy the resources/datomic/ion-config-sample.edn file to resources/datomic/ion-config.edn. Edit the ion-config.edn file and set the :app-name to the name of your Datomic system.

Adding an Item

Our first task is to provide a function to create a new inventory item. This function should implement the signature for ion lambdas, taking a JSON string in :input and returning whatever you want. The implementation shown below, which can be found in the datomic.ion.starter namespace, returns an EDN map with the database t after the item is added.

(defn add-item
  "Lambda ion that adds an item, returns database t."
  [{:keys [input]}]
  (let [args (-> input json/read-str)
        conn (get-connection)
        tx [(list* 'datomic.ion.starter/create-item args)]
        result (d/transact conn {:tx-data tx})]
    (pp-str {:t (-> result :db-after :t)})))

Note that add-item uses a custom transaction function called create-item. This is a function of data to data, with the current database value as its first argument:

(defn create-item
  "Transaction fn that creates data to make a new item"
  [db sku size color type]
  [{:inv/sku sku
    :inv/color (keyword color)
    :inv/size (keyword size)
    :inv/type (keyword type)}])

Because this is pure function, it is easy to develop and test at the REPL.

Before your can use add-item, you need to push and deploy it.

Push

The push operation creates an application revision in S3 and AWS CodeDeploy that can then be deployed multiple times to separate systems.

If you have followed the tutorial to this point and cloned the ion-starter project, please use .gitignore to specify any intentionally untracked files. Ion push Requires a Clean git Commit and using .gitignore would ensure a clean git commit. You will need to list any files you have changed, including the ion-config.edn file, the target directory, and the .gitignore file itself. If you have not configured .gitignore and have any untracked files the push command below will error.

Push the project:

clojure -A:dev -m datomic.ion.dev '{:op :push}'

On success, the push command will return the :rev that names this application, which is equal to the git SHA for the current commit.

Deploy

The deploy operation handles everything needed to update an application with zero downtime. Using AWS CodeDeploy and AWS Step Functions, Deploy installs code, configures AWS Lambdas, and cycles processes.

Let's deploy the revision you just pushed. In the command below, replace $(GROUP) with the name of your compute stack, and $(REV) with the :rev output by push.

clojure -A:dev -m datomic.ion.dev '{:op :deploy :rev $(REV) :group $(GROUP)}'

Monitor Your Deployment

Deploy does the minimal work necessary to move your system to its new configuration, using a step function to CodeDeploy the application and ensure that the lambdas are correctly configured. You can monitor the deployment with the following command, replacing $(EXECUTION_ARN) with the :execution-arn output by deploy:

clojure -A:dev -m datomic.ion.dev '{:op :deploy-status :execution-arn $(EXECUTION_ARN)}'

This will return the status of the Code Deploy and the overall ion deploy, with "SUCCEEDED" indicating when each process has successfully completed:

{:deploy-status "SUCCEEDED", :code-deploy-status "SUCCEEDED"}

See deploy status for more information on monitoring deployments.

Try It

Now you are ready to invoke add-item from the AWS CLI. In the command below, replace GROUP with the name of your compute stack and optionally add --profile <name> with <name> being the credentials profile name for the system:

aws lambda invoke --function-name $(GROUP)-add-item --payload '["SKU-12345", "small", "red", "hat"]' /dev/stdout

On success, the basis-t following the completion of the transaction will be echoed to stdout:

{:t 16}
{
    "StatusCode": 200,
}

That is all there is to it. A Datomic Cloud system can implement an arbitrary number of lambdas that you develop interactively, in Clojure, at a REPL.

Querying the Database

Users of the system want to query for items by type. The items-by-type lambda returns information about items of a particular type:

(defn db-centric-items-by-type
  "Returns db-centric (entity ids instead of idents) info about items matching type."
  [db type]
  (d/q '[:find ?sku ?size ?color ?featured
         :in $ ?type
         :where
         [?e :inv/type ?type]
         [?e :inv/sku ?sku]
         [?e :inv/size ?size]
         [?e :inv/color ?color]
         [(datomic.ion.starter/feature-item? $ ?e) ?featured]]
       db type))

(defn get-items-by-type
  "Lambda ion that returns db-centric info about items matching type."
  [{:keys [input]}]
  (-> (db-centric-items-by-type (d/db (get-connection))
                                (-> input json/read-str keyword))
      pp-str))

Note that items-by-type lambda uses a custom query function, feature-item?, to determine if an item is currently being featured. Such a query function is ordinary Clojure code, developed at a REPL:

(defn feature-item?
  "Query ion exmaple. This predicate matches entities that
should be featured in a promotion."
  [db e]
  (let [{:keys [inv/color inv/size inv/type]} (d/pull db {:eid e :selector [:inv/color :inv/size :inv/type]})]
    (and (= (:db/ident color) :green)
         (= (:db/ident size) :xlarge)
         (= (:db/ident type) :hat))))

Try It

If you ran the push and deploy steps for add-item above, then items-by-type is also deployed and ready to go. (Push and deploy work with an entire application that can include many ions.)

The following invocation of the lambda will query for all hats in the database. Replace $(GROUP) in the command below with the name of your compute stack:

aws lambda invoke --function-name $(GROUP)-items-by-type --payload \"hat\" /dev/stdout

On success, the results of the query will be echoed to stdout:

#{["SKU-19" 28499341391953988 23934169113428033]
  ...
  ["SKU-55" 46139905947992133 32330039903125571]}
{
    "StatusCode": 200,
}

Creating a Lambda Proxy Web Service

The inventory system must also support a single-page webapp that displays inventory by type. You can use AWS API Gateway to turn an ion lambda into a web service.

Create a Web Service Ion

The lambda for a web service is slightly different from a data-driven lambda. A web ion takes a web request and returns a web response. Here is the web version of items-by-type:

(defn items-by-type
  "HTTP handler that returns self-describing info about items matching type."
  [{:keys [headers body]}]
  (let [type (some-> body read-edn)]
    (if (keyword? type)
      {:status 200
       :headers {"Content-Type" "application/edn"}
       :body (-> (self-describing-items-by-type (d/db (get-connection)) type)
                 pp-str)}
      {:status 400
       :headers {}
       :body "Expected a request body keyword naming a type"})))

For API Gateway, this web function must be converted to the signature for Lambda Proxies. The ion support library provides the helper function api-gateway/ionize for this:

(def items-by-type-ionized
  "Ionization of items-by-type for use with AWS API Gateway lambda proxy integration."
  (apigw/ionize items-by-type))

If you ran the push and deploy steps for add-item above, then items-by-type-ionized is also deployed and ready to go. Let's expose it via API Gateway.

Create an API in API Gateway

In this step you will create an API backed by an AWS Lambda Proxy

  • Go to the AWS API Gateway console to create a new AWS API Gateway.
    • If this is the first AWS API Gateway you are creating, choose Get Started, then click OK and choose thew New API radio button.
    • If you have existing AWS API Gateways, choose Create API.
  • Give your API a unique name, e.g. "ion-get-schema". Leave the other default settings.
  • Click Create API in the bottom right.
  • Under the Actions dropdown, choose Create Resource.
  • Check the box "Configure as proxy resource." Other fields will be updated with defaults. Leave these as is.
  • Click Create Resource.

Connect API Gateway to Datomic Cloud

To connect API Gateway to items-by-type-ionized:

  • Set the Lambda Function to $(GROUP)-items-by-type-ionized and click Save. Note that the autocomplete does not always work correctly on this step, so trust and verify your own spelling.
  • Choose OK to give API Gateway permission to call your lambda.
  • Under your API in the left side of the UI, click on the bottom choice Settings.
  • Choose Add Binary Media Type, and add the */* type, then Save Changes.

Deploy

NOTE This step of the tutorial will expose an ion on the public internet.

  • Click on your API name, and choose Deploy API under the Actions dropdown.
  • Choose New Stage as the Deployment Stage, add the Stage Name "dev", then choose Deploy.

The top of the Stage Editor will show the "Invoke URL" for your deployed app. Append "/datomic" to the URL, and call your web service via curl to make sure everything looks ok:

curl https://$(Invoke URL)/datomic -d :hat

On success, this will return an EDN representation of inventory data for a particular type:

#{{:color :red, :type :hat, :size :medium, :sku "SKU-7"}
  ...
  {:color :blue, :type :hat, :size :small, :sku "SKU-35"}}

Creating an HTTP Direct Web Service

HTTP Direct requires that you are running the Production Topology due to the need for an AWS NLB.

HTTP Direct allows your web request to go straight to your handler-fn rather than through a lambda proxy. HTTP Direct offers better performance and simpler operation than AWS Lambda proxies.

Configuration

The ion-config must contain an :http-direct key containing a handler-fn that takes a web request and returns a web request.

:http-direct {:handler-fn datomic.ion.starter/items-by-type}

Create an API Gateway

To create an API Gateway associated with a specific VPCLink:

  • Ensure that the VPC Link that you created in the previous step shows "Status: Available"
  • Go to the AWS API Gateway console
    • If this is the first AWS API Gateway you are creating, choose Get Started, then click OK and choose the New API radio button.
    • If you have existing AWS API Gateways, choose Create API.
  • Give your API a unique name, leave the other default settings.
  • Click Create API in the bottom right.
  • Under the Actions dropdown, choose Create Resource.
  • Check the box "Configure as proxy resource." Other fields will be updated with defaults. Leave these as is.
  • Click Create Resource.
  • Select VPC Link as the "Integration Type".
  • Check the box "Use Proxy Integration".
  • Select the desired VPC Link target for this API Gateway from the dropdown box
  • Enter your http://$(NLB URI):port/{proxy} as the "Endpoint URL". This NLB URI can be found in the Outputs tab of your compute or query group CloudFormation Template entry under the "LoadBalancerHttpDirectEndpoint" key:

    ions-api-gw-proxy.png

  • Click Save

Deploy the API Gateway

NOTE This step of the tutorial will expose an ion on the public internet.

  • Click on your API name, and choose Deploy API under the Actions dropdown.
  • Choose New Stage as the Deployment Stage, add the Stage Name "dev", then choose Deploy.

The top of the Stage Editor will show the "Invoke URL" for your deployed app. Append "/datomic" to the URL, and call your web service via curl to make sure everything looks ok:

curl https://$(Invoke URL)/datomic -d :hat

On success, this will return an EDN representation of inventory data for a particular type:

#{{:color :red, :type :hat, :size :medium, :sku "SKU-7"}
  ...
  {:color :blue, :type :hat, :size :small, :sku "SKU-35"}}

Cleanup (Optional)

The API Gateway is an external connection point, not managed by Datomic. If you created an API Gateway in the previous steps, you can select and delete it in the console.

You do not need to delete the AWS Lambdas managed by Datomic Ions in order to control cost. AWS Lambda pricing is entirely usage-based, so if you do not invoke Lambdas they cost nothing. If you want to delete these Lambdas anyway, you can push and deploy an application with an empty lambdas map in ion-config.edn.

Conclusion

In this tutorial, you have seen how ions can take ordinary Clojure functions, and install them in Datomic Cloud where they have in-memory access to Datomic data. You have used ions to deliver:

  • a data-driven application via AWS Lambda
  • a web service via API Gateway
  • a web service via an http-direct endpoint

All hosted out of a single Datomic Cloud system with no additional servers.