Monitoring
Finding Datomic Resources by Tag
Datomic tags resources it creates with the following tag:
| Tag Name | Tag Value | Resources Tagged |
|---|---|---|
| datomic:system | System Name | all taggable resources |
You can use this tag to locate all the resources associated with a Datomic system. From the Tag Editor:
- Under Regions, Enter one or more AWS regions
- Under Resource types, choose "All resource types"
- Under Tags, choose the name "datomic:system" for tag key
- Under the same row in Tags, enter a specific System Name for tag value, or leave the tag value blank to find resources for all Datomic Systems.
CloudWatch Metrics
Datomic cluster nodes write Cloudwatch Metrics as follows:
- The namespace is DatomicCloud
- The dimensions include system, cluster, and group.
In the table below
- {CacheOp} can be Get or Put
- {DdbOp} can be Read or Write
- {Outcome} can be Succeeded or Failed
- {RefOp} can be Get, Scan, Update
- {StoreOp} can be Create, Get, Exists, Delete
| Metric | Subsystem | Units | Description |
|---|---|---|---|
| Alerts | Log | count | number of alerts written to Cloudwatch logs |
| CacheKmsEkMap | Cache | count | 0 = miss, 1 = hit, average = hit %, KMS Encrypted Key Map |
| CacheKmsKey | Cache | count | 0 = miss, 1 = hit, average = hit %, KMS Secret Key |
| Datoms | Db | count | number of datoms in a database, reported when indexer checks for work |
| Ddb{DdbOp}{Outcome}Msec | Storage | msec | latency for a ddb operation |
| Ddb{Op}RetryCount | Storage | msec | backoff count for a ddb op |
| Ddb{Op}RetryDelay | Storage | msec | backoff latency for a ddb op |
| Efs{CacheOp}{Outcome}Msec | Cache | msec | latency for an EFS cache operation |
| EfsHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| EfsRepair | Cache | count | count of repaired EFS cache hits |
| Events | Log | count | number of events written to Cloudwatch logs |
| HttpEndpointAsyncTimeout | Web | count | count of requests failed by http async timeout |
| HttpEndpointPings | Web | count | count of healthcheck pings received from e.g. load balancer |
| HttpEndpointOpsPending | Web | count | count is total number of requests started |
| HttpEndpointThrottled | Web | count | requests rejected because server too busy |
| ObjectCacheHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| IndexClaimRootConflict | Index | count | conflict attempting to claim index root, will retry |
| IndexJobMsec | Index | msec | duration of index job |
| IndexJobDatoms | Index | count | number of datoms added by an index job |
| IndexJobStart | Index | count | start of an index job |
| IndexJobWrites | Index | count | number of segments written by an index job |
| IndexMemMb | Index | mb | total size of the in-memory indexes on a node |
| IndexWriteQueueCount | Index | count | count of index write queue |
| JvmFreeMb | - | mb | free JVM memory |
| KmsGenerateDataKey | Log | count | count of data keys generated |
| LogCatchupBatches | Log | count | number of tx batches read to catchup the log |
| LogCatchupBytes | Log | count | number of tx batches read to catchup the log |
| LogCatchupMsec | Log | count | number of tx batches read to catchup the log |
| NanoImplFailedAuthns | Auth | count | number of failed auth'n AND auth'z attempts |
| NextForwardCount | Web | count | number of paginated requests forwarded from one node to another |
| NodeDbCount | - | count | number of databases being server by node |
| Ref{RefOp}Msec | Ref | msec | latency for a ref op |
| S3{StoreOp}{Outcome}Msec | Storage | msec | latency for a store op |
| S3{StoreOp}{Outcome} | Storage | count | number of times current S3 Op has failed, including retries |
| S3{StoreOp}RetrySucceedded | Storage | count | S3 op succeeded on a retry |
| S3StoreCreateBytes | Storage | bytes | number of bytes written to S3 |
| TxBatchCount | Tx | count | number of txes batched into a storage write |
| TxBatchBytes | Tx | bytes | number of bytes batched into a storage write |
| TxBatchStoreVal | Tx | count | number of large tx batches written as vals |
| Tx{Outcome}Msec | Tx | msec | time from transaction dequeued to notifications enqueued |
| TxBatchProcessedMsec | Tx | msec | time from first tx dequeued to all notitications enqueued |
| TxCacheGetMsec | Tx | msec | time to retrieve recent tx from the cache |
| TxCacheHits | Tx | count | 0 = miss, 1 = hit, average = hit % |
| TxForwardCount | Web | count | number of tx requests forwarded from one node to another |
| Valcache{CacheOp}{Outcome}Msec | Cache | msec | latency for a memcached cache operation |
| ValcacheHits | Cache | count | 0 = miss, 1 = hit, average = hit % |
| ValcacheRepair | Cache | count | count of repaired memcached cache hits |
Cloudwatch Logs
Datomic Compute Resources write Cloudwatch Logs as follows:
- The "Log Group" name is the name of the Compute Resources
- Each EC instance will create a log stream named by the convention {system}-{group}-{instance-id}-{timestamp}
| Subsystem | Msg Key | Details |
|---|---|---|
| Cache | "Create object cache" | size of object cache |
| Index | "Starting indexer" | indexer settings |
| Index | "Root map loaded" | contents of an index root |
| Node | "Active database ids" | db ids being served by this node |
| Node | "Cluster node starting" | includes all configuration for the cluster |
| {various} | "Daemon thread started" | thread name identifies subsystem and db |
| {various} | "Daamon thread completed" | thread name identifies subsystem and db |