Metrics
The Terraform Cloud Agent emits numerous metrics describing the agent's performance.
Metric naming conventions
All metrics emitted by the Terraform Cloud Agent follow some general naming conventions which convey useful context about the measurements.
- All metrics are prefixed by
tfc-agent.to distinguish them from any other metrics in your environment. - When a metric requires a unit in order to be understood, an un-abbreviated
unit will be the last component of the metric name. For example,
.bytes, or.milliseconds. - All timing metrics (other than runtime metrics) are measured in milliseconds.
- All data size metrics are measured in bytes.
- Underscores are used to separate words in metric names when submitted to the collector over the OTLP protocol.
Metric data types
Terraform Cloud agents emit three types of metric data:
- Gauges submit an absolute number value at the end of a measurement period.
- Counters accumulate number values during a measurement period. The sum of the numbers is recorded at the end of a measurement period and the value is reset to zero for the next period.
- Timers measure the time taken to complete a task.
Core metrics
The following metrics are generated by the Terraform Cloud Agent core program,
and are related to generic operations performed regularly by all agents. All
metrics in this section are prefixed by tfc-agent.core..
| Meric name | Type | Description |
|---|---|---|
status.busy | Gauge | Number of agents in busy status. |
status.idle | Gauge | Number of agents in idle status. |
register.milliseconds | Timer | Time to register the agent with Terraform Cloud. |
fetch_job.milliseconds | Timer | Time to complete a job dequeue request. |
update_status.milliseconds | Timer | Time to send a status update over from the agent to Terraform Cloud. |
Runtime metrics
The Terraform Cloud Agent produces a number of metrics which are generated by
the application runtime, and are primarily useful in debugging the TFC Agent.
These metrics are emitted periodically throughout the entire agent process
lifecycle. It is important to note that these metrics do not represent a
complete picture of resource utilization by the agent. The agent may fork child
processes (such as the Terraform binary or other programs) which maintain their
own distinct runtimes and consume resources independently of the agent. To
monitor resource utilization comprehensively, consider monitoring VM or
container metrics. All metrics in this section are prefixed by
tfc-agent.core.runtime..
| Metric name | Type | Description |
|---|---|---|
go.mem.heap_alloc.bytes | Gauge | Memory allocated to heap objects by the Go runtime. |
go.mem.heap_idle.bytes | Gauge | Memory allocated to the heap which are unused. |
go.mem.heap_inuse.bytes | Gauge | Memory allocated to the heap which are in use. |
go.mem.heap_sys.bytes | Gauge | Memory obtained from the OS for the heap. |
go.mem.heap_released.bytes | Gauge | Memory returned to the OS during GC. |
go.mem.heap_objects.count | Gauge | Number of allocated heap objects. |
go.mem.lookups.count | Gauge | Number of pointer lookups. |
go.mem.malloc.count | Gauge | Cumulative count of heap objects allocated. |
go.mem.free.count | Gauge | Cumulative count of heap objects freed. |
go.gc.count | Gauge | Number of completed GC cycles. |
go.goroutines.count | Gauge | Number of running goroutines. |
go.gc.pause_total.nanoseconds | Timer | Cumulative time spent in stop-the-world pauses. |
uptime.milliseconds | Timer | Cumulative time since the agent started. |
Terraform component metrics
The following metrics are emitted by the terraform component, which is
responsible for handling Terraform operations like plans and applies. All
metrics in this section are prefixed by tfc-agent.core.terraform..
| Metric name | Type | Description |
|---|---|---|
configure_terraform_cli.milliseconds | Timer | Time spent configuring the Terraform CLI utility prior to execution. |
handle_signal.milliseconds | Timer | Time spent handling a signal from Terraform Cloud. |
execute.milliseconds | Timer | Time spent handling a Terraform operation. |
output_stream.upload_chunk.bytes | Gauge | Size of a chunk of Terraform output uploaded. |
output_stream.upload_chunk.milliseconds | Timer | Time spent uploading a single chunk of Terraform output. |
output_stream.upload_full.bytes | Gauge | Size of a full Terraform log uploaded. |
output_stream.upload_full.milliseconds | Timer | Time spent uploading the full Terraform log. |
output_stream.close.milliseconds | Timer | Time spent finalizing a Terraform output stream. |
override_sensitive_variables.milliseconds | Timer | Time spent generating a configuration file to override variable sensitivity. |
override_sensitive_variables.count | Gauge | Number of variables in a run of which Terraform Cloud modified the sensitivty flag. |
persist_filesystem.milliseconds | Timer | Time spent packing and uploading a filesystem image. |
persist_filesystem.pack.bytes | Gauge | Size of a packed up filesystem image. |
persist_filesystem.pack.milliseconds | Timer | Time spent packing the contents of a filesystem. |
persist_filesystem.upload.milliseconds | Timer | Time spent uploading a packed up filesystem image. |
plan_json.generate.bytes | Gauge | Size of a generated JSON-formatted plan. |
plan_json.generate.milliseconds | Timer | Time spent generating a JSON plan. |
plan_json.upload.milliseconds | Timer | Time spent uploading a JSON plan. |
provider_schemas_json.generate.bytes | Gauge | Size of a generated JSON-formatted provider schemas file. |
provider_schemas_json.generate.milliseconds | Timer | Time spent generating the provider schemas document. |
provider_schemas_json.upload.milliseconds | Timer | Time spent uploading a provider schemas document. |
restore_filesystem.milliseconds | Timer | Time spent downloading and unpacking a filesystem image. |
restore_filesystem.download.bytes | Gauge | Size of a downloaded filesystem image. |
restore_filesystem.download.milliseconds | Timer | Time spent downloading a filesystem image. |
restore_filesystem.unpack.milliseconds | Timer | Time spent unpacking the contents of a filesystem image. |
run_meta.additions | Gauge | Number of resources added or proposed to be added in a Terraform operation. |
run_meta.changes | Gauge | Number of resources changed or proposed to change in a Terraform operation. |
run_meta.destructions | Gauge | Number of resources destroyed or proposed to be destroyed in a Terraform operation. |
setup_backend.milliseconds | Timer | Time spent configuring Terraform CLI for Terraform Cloud. |
setup_ssh_key.milliseconds | Timer | Time spent configuring an SSH key for Terraform to use while downloading modules. |
setup_ssh_key.check_git_version.milliseconds | Timer | Time spent ensuring the local version of "git" is adequate for leveraging SSH auth. |
setup_terraform_binary.milliseconds | Timer | Time spent downloading and unpacking a Terraform Community release. |
setup_terraform_binary.download.bytes | Gauge | Size of a downloaded Terraform Community version. |
setup_terraform_binary.download.milliseconds | Timer | Time spent downloading a Terraform Community release. |
setup_terraform_binary.unpack.bytes | Gauge | Size of an unpacked Terraform Community release. |
setup_terraform_binary.unpack.milliseconds | Timer | Time spent unpacking a Terraform Community release. |
setup_terraform_config.milliseconds | Timer | Time spent downloading and unpacking a Terraform configuration. |
setup_terraform_config.download.bytes | Gauge | Size of a downloaded Terraform configuration. |
setup_terraform_config.download.milliseconds | Timer | Time spent downloading a Terraform configuration. |
setup_terraform_config.unpack.milliseconds | Timer | Time spent unpacking a downloaded Terraform configuration. |
setup_terraform_config.verify.milliseconds | Timer | Time spent verifying a Terraform configuration. |
setup_terraform_variables.milliseconds | Timer | Time spent configuring Terraform variables provided by TFC. |
setup_terraform_variables.write_file.bytes | Gauge | Size of a tfvars file, generated by TFC-provided input variables. |
terraform_init.milliseconds | Timer | Time spent running terraform init. |
terraform_apply.milliseconds | Timer | Time spent running terraform apply. |
terraform_plan.milliseconds | Timer | Time spent running terraform plan. |
terraform_version.milliseconds | Timer | Time spent running terraform version. |
Policy component metrics
The following metrics are emitted by the policy component, which is
responsible for handling OPA policy enforcement operations. All metrics in this
section are prefixed by tfc-agent.core.policy..
| Metric name | Type | Description |
|---|---|---|
execute.milliseconds | Timer | Time spent handling a policy operation. |
policy_set.download.bytes | Gauge | Size of a downloaded policy set. |
policy_set.download.milliseconds | Timer | Time spent downloading a policy set. |
policy_set.unpack.milliseconds | Timer | Time spent unpacking a policy set. |
generate_opa_input_file.milliseconds | Timer | Time spent generating the OPA input file. |
parse_opa_config.milliseconds | Timer | Time spent parsing the OPA configuration. |
plan_json_download.milliseconds | Timer | Time spent downloading the Terraform JSON plan document. |
plan_json_download.bytes | Gauge | Size of a downloaded Terraform JSON plan document. |
run_opa_eval.milliseconds | Timer | Time spent evaluating OPA policies. |
setup_policies.milliseconds | Timer | Time spent setting up individually managed policies. |
setup_policy_engines.milliseconds | Timer | Time spent setting up policy runtimes. |
setup_subjects.milliseconds | Timer | Time spent setting up subjects for policy enforcement. |