Watcher is an Elasticsearch feature that you can use to create actions based on conditions, which are periodically evaluated using queries on your data. Watches are helpful for analyzing mission-critical and business-critical streaming data. For example, you might watch application logs for performance outages or audit access logs for security threats.

Watcher is provided as part of x-pack license. Details on x-pack settings to enable watcher-

https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html

For details on watcher and how to get started with creating alerts-

https://www.elastic.co/guide/en/kibana/current/watcher-ui.html

Now coming back to the problem statement — How to configure watcher alerts for multiple slack channels?

Consider there are 2 Slack channels in different workspaces where you want to send the alerts.

You need a Slack webhook URL to configure a Slack Account. To create a webhook URL, set up an an Incoming Webhook Integration through the Slack console:

1. Log in to slack.com as a team administrator.
2. Go to https://my.slack.com/services/new/incoming-webhook.
3. Select a default channel for the integration.

4. Click Add Incoming Webhook Integration.

5. Copy the generated webhook URL so you can paste it into your Slack account configuration in elasticsearch.yml.

Refer-

https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-slack.html#configuring-slack

As one webhook integration can forward to one slack channel only so you will have to create two webhook integrations, one for each channel.

Add webhook URLs to ElasticSearch

Now lets add these two webhook URLs to elasticsearch-keystore.(In older versions of elasticsearch this configuration used to go in elasticsearch.yml but now it has to be in elasticsearch-keystore)

1. Login to elasticsearch servers and run the below commands. This will prompt for URL. Paste the webhook URL to add the entry in elasticsearch-keystore.

bin/elasticsearch-keystore add xpack.notification.slack.account.account1.secure_url

bin/elasticsearch-keystore add xpack.notification.slack.account.account2.secure_url

2. Now add the slack notification attributes in elasticsearch.yml file. When there are more than one slack accounts, then providing a default account becomes mandatory.

xpack.notification.slack:
  default_account: account1
  account:
    account1:
      message_defaults:
        from: testelk
    account2:
      message_defaults:
        from:testelk

Refer below link for full set of notification attributes- (optional)

https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#slack-account-attributes

Creating Watcher alert with slack integration

Lets create an example advanced watcher to send alerts to both slack channels using the settings created above-

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "logs-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                {
                  "term": {
                    "status": {
                      "value": "500"
                    }
                  }
                }
              ],
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "from": "now-30m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "notify-slack": {
      "slack": {
        "account": "account1",
        "message": {
          "from": "testalert",
          "to": [
            "#slackchannel1"
          ],
          "text": "Sample 500 alert",
          "attachments": [
            {
              "color": "danger",
              "title": "Alert Details",
              "text": "{{ctx.payload.hits.total}} events in last 30 mins with 500 status code."
            }
          ]
        }
      }
    }
  },
"notify-slack": {
      "slack": {
        "account": "account2",
        "message": {
          "from": "testalert",
          "to": [
            "#slackchannel2"
          ],
          "text": "Sample 500 alert",
          "attachments": [
            {
              "color": "danger",
              "title": "Alert Details",
              "text": "{{ctx.payload.hits.total}} events in last 30 mins with 500 status code."
            }
          ]
        }
      }
    }
  }
}

As you can see above, both the notify-slack actions have different accounts configured so that corresponding webhook url can be referred and used to send alerts.

Hope this helps. Do comment for any questions/issues or feedback.

The post How to send ElasticSearch watcher alerts to Multiple Slack Channels appeared first on TechManyu.