Collect data and send it to your Splunk instance. Slack is a common way to communicate with other Splunk users outside of user groups, the annual conference (.conf), or other Splunk and industry events. This form of chat enables anyone to communicate with the greater Splunk Community—customers, partners, and Splunkers—worldwide.
Splunk Alerts on Slack. For those using log tools such as Splunk, you can setup alerts. These will run queries every so often and trigger actions when conditions are met e.g. Count of events surpasses a threshold. This post is about pushing those alerts from Splunk to Slack. The Slack Add-on for Splunk uses the Slack Audit Logs API to fetch Slack Enterprise Grid Audit Logs into Splunk. Getting Started The Audit Logs API is for monitoring the audit events happening in an Enterprise Grid organization to ensure continued compliance, to safeguard against any inappropriate system access, and to allow you to audit.
Slack is a common way to communicate with other Splunk users outside of user groups, the annual conference (.conf), or other Splunk and industry events. This form of chat enables anyone to communicate with the greater Splunk Community—customers, partners, and Splunkers—worldwide.
Use Slack to ask for help with issues, learn from other users in real time, and to network outside of in-person events. Experience Splunk Community culture and discover what makes us unique!
Join us on Slack
Anyone can submit a request to join the team called splunk-usergroups on Slack. Go to splk.it/slack. There are over 100 channels in the splunk-usergroups that you can participate in. There are channels for specific Splunk products and apps, specific industries or issues, specific cities or regions, and a few channels just for fun.
For a list of helpful channels, you can enter !channels or !channels-advanced in slack for a current list.
For example:
You can also ask for direction in #_where_do_i_ask
The Splunk Community team approves requests to join the Slack team for Splunk. This helps us to block recruiters and sales people so that they don't disrupt the community vibe. We want to ensure that you are connecting with other Splunk users to learn and share experiences.
You can use the Slack website or download the free Slack client from http://slack.com.
Submit a request to join Slack
All community members, employees, and partners should use the same form to join the Splunk user group Slack:
- Submit a request through splk.it/slack. Specify your splunk.com user ID in the last field on the form to speed up the review process.
- The request is reviewed for approval by someone on the Splunk Community team. You will receive an email inviting you to join the Splunk team on Slack. Please allow one business day to process your request.
- Access the Slack website or free Slack client from http://slack.com.
For those using log tools such as Splunk, you can setup alerts. These will run queries every so often and trigger actions when conditions are met e.g. count of events surpasses a threshold.
This post is about pushing those alerts from Splunk to Slack. When an alarm is triggered, a JSON request is made by Splunk to a URL you provide. This gives the benefit of your own customisation of messages.
Reasons
You can find applications on the Splunk app store to post alerts to Slack. However you may have issues when running such apps on a cluster. Or you may have permission issues in an enterprise environment. In my situation, I ran into both problems.
Flow
- Splunk invokes an AWS API Gateway endpoint.
- The AWS API Gateway endpoint invokes an AWS Lambda function.
- The AWS Lambda function sends a message to a Slack inbound web hook (and delivers the message your own desired Slack channel).
Step 1 - Create Slack Inbound Webhook
Go to the following page, whilst logged-in to your Slack workspace:
Or alternatively, from Slack’s website:
- Configure apps
- Custom integrations (sidebar)
- Incoming WebHooks
More information about inbound web hooks can be found here:
Once you’ve setup an inbound webhook, copy the Webhook URL for the next step. Let’s pretend it’s:
Step 2 - Setup AWS Lambda Function
Create a new Lambda function from scratch:
For this example, we’ll call it splunk alert.
Then paste in the following, but edit the request options with parts of the Webhook URL from the previous step:
Towards the end is the JSON object message, which is the Slack message written to the Slack inbound webhook. This can be heavily customised.
Docs on Slack message format:
Once you’re happy with your Lambda, publish it. At the top, go to Actions and select Publish new version.
Step 3 - Create AWS API Gateway Trigger
On your AWS Lambda function page, go to the Triggers tab and add a trigger.
You will then see a dotted box, click it and select API Gateway.
Then will in the form with an API name (such as splunk-alerts) and set the Security to Open (although you may want to change this later):
Splunk Slack Alert
You should now have API Gateway available as a trigger. Click the arrow icon to show the endpoint’s URL.
Step 4 - Test Trigger
Let’s say the trigger URL is:
Just make a POST request with the following test payload:
In Chrome I use the Postman app, but this is simple enough to achieve with cURL as well:
Step 5 - Splunk Alert
Splunk Slack Channel
Run a query on the Splunk search application. Once it has finished loading, select Save As and then Alert.
Configure the alert as needed; useful docs:
After the alert is setup, add a Webhook action and set the URL to the endpoint created earlier.
Summary
You should now have monkeyboy to save the day: