Monitoring
AWS CloudWatch provides a powerful set of tools for monitoring a software system running on AWS:
- You can collect and track CloudWatch Metrics – variables that measure the behavior of your system.
- You can configure CloudWatch Alarms to notify operations or take other automated steps when potential problems arise.
- You can monitor, store, and search CloudWatch Logs across all your AWS resources.
- You can create CloudWatch Dashboards that provide a single overview for monitoring your systems.
Datomic is fully integrated with all of these AWS monitoring tools. On the producing side, Datomic creates metrics and logs; and on the consuming side, Datomic organizes metrics in custom dashboards. This document shows how to manage and monitor your Datomic system.
Topics
Finding Datomic Resources by Tag
Datomic tags resources it creates with the following tag:
| Tag Name | Tag Value | Resources Tagged |
|---|---|---|
| datomic:system | System Name | all taggable resources |
You can use this tag to locate all the resources associated with a Datomic system. From the Tag Editor:
- Under Regions, Enter one or more AWS regions
- Under Resource types, choose "All resource types"
- Under Tags, choose the name "datomic:system" for tag key
- Under the same row in Tags, enter a specific System Name for tag value, or leave the tag value blank to find resources for all Datomic Systems.
Searching CloudWatch Logs
To view the CloudWatch Logs for a Datomic system:
- Open the CloudWatch Logs in the AWS Console
- Click on the Log Group named "datomic-{System}", where System is your system name
Each EC2 instance will create a separate log stream. Usually you will want to search across all log streams:
- click the "Search Log Group" button
- click to the right of the filter box to choose an appropriate time window
- enter a metric filter pattern to scope your search
Finding Alerts
The search pattern "Alert - Alerts" will find the text of every alert produced by Datomic. The "Alert" matches each alert, and the "- Alerts" filters redundant summary messages.
The example below demonstrates reviewing a week's worth of alerts. The single event that matches is a transient DynamoDB request failure, so not a problem.
Finding Logs By Message
The image below shows using the CloudWatch
Filter and Pattern Syntax to find all logs whose Msg is
"CodeDeployEvent".
- The brackets
{}identify the query as a JSON metric filter - the
$is a placeholder for log entry as a whole - the dot syntax scopes the search to the individual key
Msg
Metrics Produced by Datomic Cloud
Datomic cluster nodes write Cloudwatch Metrics as follows:
- The namespace is DatomicCloud
- The dimensions include system, cluster, and group.
In the table below
- {CacheOp} can be Get or Put
- {DdbOp} can be Read or Write
- {Outcome} can be Succeeded or Failed
- {RefOp} can be Get, Scan, Update
- {StoreOp} can be Create, Get, Exists, Delete
| Metric | Subsystem | Units | Description |
|---|---|---|---|
| Alerts | Log | count | number of alerts written to Cloudwatch logs |
| CacheKmsEkMap | Cache | count | 0 = miss, 1 = hit, average = hit %, KMS Encrypted Key Map |
| CacheKmsKey | Cache | count | 0 = miss, 1 = hit, average = hit %, KMS Secret Key |
| Datoms | Db | count | number of datoms in a database, reported when indexer checks for work |
| Ddb{DdbOp}{Outcome}Msec | Storage | msec | latency for a ddb operation |
| Ddb{Op}RetryCount | Storage | msec | backoff count for a ddb op |
| Ddb{Op}RetryDelay | Storage | msec | backoff latency for a ddb op |
| Efs{CacheOp}{Outcome}Msec | Cache | msec | latency for an EFS cache operation |
| EfsHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| EfsRepair | Cache | count | count of repaired EFS cache hits |
| Events | Log | count | number of events written to Cloudwatch logs |
| HttpEndpointAsyncTimeout | Web | count | count of requests failed by http async timeout |
| HttpEndpointPings | Web | count | count of healthcheck pings received from e.g. load balancer |
| HttpEndpointOpsPending | Web | count | count is total number of requests started |
| HttpEndpointThrottled | Web | count | requests rejected because server too busy |
| ObjectCacheHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| IndexClaimRootConflict | Index | count | conflict attempting to claim index root, will retry |
| IndexJobMsec | Index | msec | duration of index job |
| IndexJobDatoms | Index | count | number of datoms added by an index job |
| IndexJobStart | Index | count | start of an index job |
| IndexJobWrites | Index | count | number of segments written by an index job |
| IndexMemMb | Index | mb | total size of the in-memory indexes on a node |
| IndexWriteQueueCount | Index | count | count of index write queue |
| JvmFreeMb | - | mb | free JVM memory |
| KmsGenerateDataKey | Log | count | count of data keys generated |
| LogCatchupBatches | Log | count | number of tx batches read to catchup the log |
| LogCatchupBytes | Log | count | number of tx batches read to catchup the log |
| LogCatchupMsec | Log | count | number of tx batches read to catchup the log |
| NanoImplFailedAuthns | Auth | count | number of failed auth'n AND auth'z attempts |
| NextForwardCount | Web | count | number of paginated requests forwarded from one node to another |
| NodeDbCount | - | count | number of databases being server by node |
| Ref{RefOp}Msec | Ref | msec | latency for a ref op |
| S3{StoreOp}{Outcome}Msec | Storage | msec | latency for a store op |
| S3{StoreOp}{Outcome} | Storage | count | number of times current S3 Op has failed, including retries |
| S3{StoreOp}RetrySucceedded | Storage | count | S3 op succeeded on a retry |
| S3StoreCreateBytes | Storage | bytes | number of bytes written to S3 |
| TxBatchCount | Tx | count | number of txes batched into a storage write |
| TxBatchBytes | Tx | bytes | number of bytes batched into a storage write |
| TxBatchStoreVal | Tx | count | number of large tx batches written as vals |
| Tx{Outcome}Msec | Tx | msec | time from transaction dequeued to notifications enqueued |
| TxBatchProcessedMsec | Tx | msec | time from first tx dequeued to all notitications enqueued |
| TxCacheGetMsec | Tx | msec | time to retrieve recent tx from the cache |
| TxCacheHits | Tx | count | 0 = miss, 1 = hit, average = hit % |
| TxForwardCount | Web | count | number of tx requests forwarded from one node to another |
| Valcache{CacheOp}{Outcome}Msec | Cache | msec | latency for a memcached cache operation |
| ValcacheHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| ValcacheRepair | Cache | count | count of repaired memcached cache hits |
Note In order to reduce cost, the Solo Topology reports only a small subset of the metircs listed above: Alerts, Datomic, Events HttpEndpointOpsPending, JvmFreeMb, and HttpEndpointThrottled.
Logs Produced by Datomic Cloud
Datomic writes Cloudwatch Logs as follows:
- The "Log Group" name is the name of the Datomic System
- Each EC instance will create a log stream named by the convention {system}-{group}-{instance-id}-{timestamp}
| Subsystem | Msg Key | Details |
|---|---|---|
| Cache | "Create object cache" | size of object cache |
| Index | "Starting indexer" | indexer settings |
| Index | "Root map loaded" | contents of an index root |
| Node | "Active database ids" | db ids being served by this node |
| Node | "Cluster node starting" | includes all configuration for the cluster |
| {various} | "Daemon thread started" | thread name identifies subsystem and db |
| {various} | "Daamon thread completed" | thread name identifies subsystem and db |
Datomic Cloud Dashboards
When you launch a Datomic Cloud system, it automatically creates a
dashboard named datomic-(system)-(region) that you can view in the
AWS Console.
The Solo Topology creates a basic dashboard:
The Production Topology creates a much larger dashboard, suitable for viewing on a large ops monitor screen:
Each dashboard widget tells a story. For example, the DynamoDB Usage widget tracks DynamoDB AutoScaling of reads (left axis) and writes (right axis). In the image below you can see DynamoDB write provisioning (the green line) scaling up for a four-hour period during a batch load, and then scaling back down to almost nothing.
The metrics tracked in the dashboards are a subset of the metrics produced by Datomic and described in the table above.