Alerts

Download schema

Endpoints for the metric alert rules of the team and the alerts they raise. Rules are checked every minute and notify the team when a metric of a server stays above or below a threshold.

List alert rules

GET

https://server.chief.app/api/v1

/alert-rules

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

List the team's metric alert rules, oldest first. Rules watch the servers the team owns, not servers shared with it.

List alert rules › query Parameters

query

string | null · maxLength: 255

Search results by name.

per_page

integer · min: 1 · max: 100

Number of results per page.

Default: 50

page

integer · min: 1

Page number to return.

Default: 1

List alert rules › Responses

Paginated set of AlertRule

AlertRule[] · required

object · required

GET/alert-rules

curl https://server.chief.app/api/v1/alert-rules \
  --header 'Authorization: Bearer <token>'

Example Responses

{
  "data": [
    {
      "id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "metric": "cpu_used",
      "condition": "above",
      "threshold": 90,
      "baseline": "time_of_day",
      "minimum_difference": 10,
      "unit": "%",
      "duration_minutes": 10,
      "scope": "all",
      "exclude": true,
      "tag": {
        "id": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
        "name": "production",
        "color": "zinc"
      },
      "servers": [
        {
          "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
          "name": "web-01"
        }
      ],
      "instance": "/var",
      "enabled": true,
      "description": "Disk space used above 90% for 10 minutes",
      "scope_description": "Servers tagged production",
      "firing_alerts_count": 1,
      "created_at": "created_at",
      "updated_at": "updated_at"
    }
  ],
  "links": {
    "first": "first",
    "last": "last",
    "prev": "prev",
    "next": "next"
  },
  "meta": {
    "current_page": 1,
    "from": 1,
    "last_page": 1,
    "links": [
      {
        "url": "url",
        "label": "label",
        "active": true
      }
    ],
    "path": "path",
    "per_page": 0,
    "to": 1,
    "total": 0
  }
}

json

application/json

Create an alert rule

POST

https://server.chief.app/api/v1

/alert-rules

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

Create a metric alert rule: when the metric stays above or below the threshold for the whole duration on a server the rule watches, an alert fires and the team members who want metric alerts are notified. Rules are checked every minute.

Create an alert rule › Request Body

SaveAlertRuleRequest

metric

string · enum · required

What the rule watches.

Enum values:

cpu_used

cpu_iowait

cpu_steal

load_per_core

memory_used

swap_used

disk_space_used

inodes_used

Example: disk_space_used

condition

AlertCondition · enum · required

Fire when the metric stays above or below the threshold.

Enum values:

above

below

Example: above

threshold

number · min: 0 · max: 1000000000 · required

Threshold in the metric's unit: a percentage for percentages, Mbit/s for network traffic, MB/s for disk throughput and minutes for uptime. For a baseline rule, the percentage above or below the normal level instead.

Example: 90

duration_minutes

integer · min: 1 · max: 1440 · required

Minutes the metric must stay above or below the threshold, 1 to 1440.

Example: 10

baseline

string | null · enum

Compare to the metric's normal level instead of a fixed value: time_of_day is the average at the same time of day over the past week. Null for a fixed threshold. Not for disk space, inodes and uptime.

Enum values:

time_of_day

Default: null

minimum_difference

number | null · min: 0 · max: 1000000000

For a baseline rule: how much the value must at least differ from normal, in the metric's unit, so a metric that is normally close to zero does not alert on a small change. Null for any difference.

Example: 10

scope

string · enum

Which of the team's servers the rule watches: all, the servers with tag, or the chosen servers.

Enum values:

all

tag

servers

Default: all

exclude

boolean

For tag and servers rules: watch the servers without the tag, or all servers except the chosen ones.

Default: false

tag

string | null · maxLength: 64

Formatted ID of one of the team's tags, required for tag rules.

Example: tag_01j8q8zk2f4m6n7p9r1s3t5v7x

servers

string[] · maxItems: 500

Formatted IDs of the team's servers, required for servers rules.

Example: ["server_01j8q8zk2f4m6n7p9r1s3t5v7x"]

instance

string | null · maxLength: 128 · pattern: ^[^\x00-\x1F\x7F]*$

The mount point, disk or interface to watch, for metrics that have them. Without one, disk space and inode rules watch every mount point separately, and disk and network rules the host total.

Example: /var

enabled

boolean

Whether the rule is evaluated. Turning it off closes its open alerts.

Default: true

Create an alert rule › Responses

The alert rule was created.

AlertRule · required

POST/alert-rules

curl https://server.chief.app/api/v1/alert-rules \
  --request POST \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <token>' \
  --data '{
  "metric": "disk_space_used",
  "condition": "above",
  "threshold": 90,
  "baseline": null,
  "minimum_difference": 10,
  "duration_minutes": 10,
  "scope": "all",
  "exclude": false,
  "tag": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
  "servers": [
    "server_01j8q8zk2f4m6n7p9r1s3t5v7x"
  ],
  "instance": "/var",
  "enabled": true
}'

Example Request Body

{
  "metric": "disk_space_used",
  "condition": "above",
  "threshold": 90,
  "baseline": null,
  "minimum_difference": 10,
  "duration_minutes": 10,
  "scope": "all",
  "exclude": false,
  "tag": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
  "servers": [
    "server_01j8q8zk2f4m6n7p9r1s3t5v7x"
  ],
  "instance": "/var",
  "enabled": true
}

json

Example Responses

{
  "data": {
    "id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
    "metric": "cpu_used",
    "condition": "above",
    "threshold": 90,
    "baseline": "time_of_day",
    "minimum_difference": 10,
    "unit": "%",
    "duration_minutes": 10,
    "scope": "all",
    "exclude": true,
    "tag": {
      "id": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "name": "production",
      "color": "zinc"
    },
    "servers": [
      {
        "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
        "name": "web-01"
      }
    ],
    "instance": "/var",
    "enabled": true,
    "description": "Disk space used above 90% for 10 minutes",
    "scope_description": "Servers tagged production",
    "firing_alerts_count": 1,
    "created_at": "created_at",
    "updated_at": "updated_at"
  }
}

json

application/json

Get an alert rule

GET

https://server.chief.app/api/v1

/alert-rules/{rule}

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

Get one of the team's metric alert rules.

Get an alert rule › path Parameters

rule

string · required

Get an alert rule › Responses

AlertRule

AlertRule · required

GET/alert-rules/{rule}

curl https://server.chief.app/api/v1/alert-rules/:rule \
  --header 'Authorization: Bearer <token>'

Example Responses

{
  "data": {
    "id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
    "metric": "cpu_used",
    "condition": "above",
    "threshold": 90,
    "baseline": "time_of_day",
    "minimum_difference": 10,
    "unit": "%",
    "duration_minutes": 10,
    "scope": "all",
    "exclude": true,
    "tag": {
      "id": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "name": "production",
      "color": "zinc"
    },
    "servers": [
      {
        "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
        "name": "web-01"
      }
    ],
    "instance": "/var",
    "enabled": true,
    "description": "Disk space used above 90% for 10 minutes",
    "scope_description": "Servers tagged production",
    "firing_alerts_count": 1,
    "created_at": "created_at",
    "updated_at": "updated_at"
  }
}

json

application/json

Update an alert rule

PATCH

https://server.chief.app/api/v1

/alert-rules/{rule}

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

Change an alert rule, or turn it off or on with enabled. Fields left out keep their value. A change closes the rule's firing alerts, which no longer say what the rule says; they fire again at the next check if the new rule is breached.

Update an alert rule › path Parameters

rule

string · required

Update an alert rule › Request Body

SaveAlertRuleRequest

metric

string · enum · required

What the rule watches.

Enum values:

cpu_used

cpu_iowait

cpu_steal

load_per_core

memory_used

swap_used

disk_space_used

inodes_used

Example: disk_space_used

condition

AlertCondition · enum · required

Fire when the metric stays above or below the threshold.

Enum values:

above

below

Example: above

threshold

number · min: 0 · max: 1000000000 · required

Example: 90

duration_minutes

integer · min: 1 · max: 1440 · required

Minutes the metric must stay above or below the threshold, 1 to 1440.

Example: 10

baseline

string | null · enum

Enum values:

time_of_day

Default: null

minimum_difference

number | null · min: 0 · max: 1000000000

For a baseline rule: how much the value must at least differ from normal, in the metric's unit, so a metric that is normally close to zero does not alert on a small change. Null for any difference.

Example: 10

scope

string · enum

Which of the team's servers the rule watches: all, the servers with tag, or the chosen servers.

Enum values:

all

tag

servers

Default: all

exclude

boolean

For tag and servers rules: watch the servers without the tag, or all servers except the chosen ones.

Default: false

tag

string | null · maxLength: 64

Formatted ID of one of the team's tags, required for tag rules.

Example: tag_01j8q8zk2f4m6n7p9r1s3t5v7x

servers

string[] · maxItems: 500

Formatted IDs of the team's servers, required for servers rules.

Example: ["server_01j8q8zk2f4m6n7p9r1s3t5v7x"]

instance

string | null · maxLength: 128 · pattern: ^[^\x00-\x1F\x7F]*$

The mount point, disk or interface to watch, for metrics that have them. Without one, disk space and inode rules watch every mount point separately, and disk and network rules the host total.

Example: /var

enabled

boolean

Whether the rule is evaluated. Turning it off closes its open alerts.

Default: true

Update an alert rule › Responses

AlertRule

AlertRule · required

PATCH/alert-rules/{rule}

curl https://server.chief.app/api/v1/alert-rules/:rule \
  --request PATCH \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <token>' \
  --data '{
  "metric": "disk_space_used",
  "condition": "above",
  "threshold": 90,
  "baseline": null,
  "minimum_difference": 10,
  "duration_minutes": 10,
  "scope": "all",
  "exclude": false,
  "tag": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
  "servers": [
    "server_01j8q8zk2f4m6n7p9r1s3t5v7x"
  ],
  "instance": "/var",
  "enabled": true
}'

Example Request Body

{
  "metric": "disk_space_used",
  "condition": "above",
  "threshold": 90,
  "baseline": null,
  "minimum_difference": 10,
  "duration_minutes": 10,
  "scope": "all",
  "exclude": false,
  "tag": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
  "servers": [
    "server_01j8q8zk2f4m6n7p9r1s3t5v7x"
  ],
  "instance": "/var",
  "enabled": true
}

json

Example Responses

{
  "data": {
    "id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
    "metric": "cpu_used",
    "condition": "above",
    "threshold": 90,
    "baseline": "time_of_day",
    "minimum_difference": 10,
    "unit": "%",
    "duration_minutes": 10,
    "scope": "all",
    "exclude": true,
    "tag": {
      "id": "tag_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "name": "production",
      "color": "zinc"
    },
    "servers": [
      {
        "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
        "name": "web-01"
      }
    ],
    "instance": "/var",
    "enabled": true,
    "description": "Disk space used above 90% for 10 minutes",
    "scope_description": "Servers tagged production",
    "firing_alerts_count": 1,
    "created_at": "created_at",
    "updated_at": "updated_at"
  }
}

json

application/json

Delete an alert rule

DELETE

https://server.chief.app/api/v1

/alert-rules/{rule}

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

Delete an alert rule and close its firing alerts. Its past alerts stay in the alert history with what the rule said.

Delete an alert rule › path Parameters

rule

string · required

Delete an alert rule › Responses

The alert rule was deleted.

string

List alerts

GET

https://server.chief.app/api/v1

/alerts

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

List the alerts the team's rules raised, firing alerts first, then the newest first. Alerts stay after their rule changed or was deleted, and go when their server is deleted.

List alerts › query Parameters

status

string | null · enum

Only return firing or resolved alerts, both when omitted.

Enum values:

firing

resolved

server

string | null · maxLength: 64

Only return alerts of this server.

rule

string | null · maxLength: 64

Only return alerts of this rule, including those it raised before it changed.

metric

string | null · enum

Only return alerts of this metric.

Enum values:

cpu_used

cpu_iowait

cpu_steal

load_per_core

memory_used

swap_used

disk_space_used

inodes_used

per_page

integer · min: 1 · max: 100

page

integer · min: 1

List alerts › Responses

Paginated set of Alert

Alert[] · required

object · required

GET/alerts

curl https://server.chief.app/api/v1/alerts \
  --header 'Authorization: Bearer <token>'

Example Responses

{
  "data": [
    {
      "id": "alert_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "status": "firing",
      "server": {
        "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
        "name": "web-01"
      },
      "rule_id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "rule_description": "Disk space used above 90% for 10 minutes",
      "metric": "cpu_used",
      "condition": "above",
      "duration_minutes": 10,
      "instance": "/var",
      "unit": "%",
      "baseline_value": 120.5,
      "threshold_value": 90,
      "value": 93.2,
      "peak_value": 95.1,
      "description": "Disk space used on /var is 93.2%",
      "fired_at": "fired_at",
      "resolved_at": "resolved_at",
      "closed_reason": "closed_reason"
    }
  ],
  "links": {
    "first": "first",
    "last": "last",
    "prev": "prev",
    "next": "next"
  },
  "meta": {
    "current_page": 1,
    "from": 1,
    "last_page": 1,
    "links": [
      {
        "url": "url",
        "label": "label",
        "active": true
      }
    ],
    "path": "path",
    "per_page": 0,
    "to": 1,
    "total": 0
  }
}

json

application/json

Get an alert

GET

https://server.chief.app/api/v1

/alerts/{alert}

OpenID Connect

Bearer (ctp_*)

Bearer (ctt_*)

Get one of the alerts the team's rules raised.

Get an alert › path Parameters

alert

string · required

Get an alert › Responses

Alert

Alert · required

GET/alerts/{alert}

curl https://server.chief.app/api/v1/alerts/:alert \
  --header 'Authorization: Bearer <token>'

Example Responses

{
  "data": {
    "id": "alert_01j8q8zk2f4m6n7p9r1s3t5v7x",
    "status": "firing",
    "server": {
      "id": "server_01j8q8zk2f4m6n7p9r1s3t5v7x",
      "name": "web-01"
    },
    "rule_id": "rule_01j8q8zk2f4m6n7p9r1s3t5v7x",
    "rule_description": "Disk space used above 90% for 10 minutes",
    "metric": "cpu_used",
    "condition": "above",
    "duration_minutes": 10,
    "instance": "/var",
    "unit": "%",
    "baseline_value": 120.5,
    "threshold_value": 90,
    "value": 93.2,
    "peak_value": 95.1,
    "description": "Disk space used on /var is 93.2%",
    "fired_at": "fired_at",
    "resolved_at": "resolved_at",
    "closed_reason": "closed_reason"
  }
}

json

application/json

Metrics Tags