The global cybersecurity talent shortage is real, and with 72 percent of CISOs claiming that their teams are facing alert fatigue, there’s not a lot of room for error when it comes to getting accurate, context-rich alerts in front of under-resourced teams.
Traditional approaches to managing security alerts have often driven teams into a reactive mode where they’re overwhelmed by huge volumes of alerts or spend too much critical time gathering information and digging through log files. If this proliferation of data can be transformed into actionable intelligence, however, teams can become significantly more proactive and reduce risk over time.
Today, we’ll look at five must-read Threat Stack blog posts that provide excellent advice on how you can move away from reactive, ad hoc tactics toward a more structured and proactive approach by making alerts a key element of your overall information security strategy.
1. Lessons Learned From Lola: Demonstrating PCI Compliance in a Cloud-Native, Containerized Environment
While the title of the post doesn’t refer to “alert fatigue,” the post itself contains five key lessons the company learned when demonstrating PCI Compliance in a Cloud-Native, Containerized Environment, and tips on how to avoid alert fatigue by surfacing high-priority, actionable information figure prominently. Specifically, Katie advises organizations to make sure they are logging everything, but only alerting when they need to:
“To be compliant, you need to maintain a complete record of what is happening in your system. Operationally, you need the ability to surface and deal with critical issues (Sev 1 alerts) in a timely manner, without subjecting your team to alert fatigue and information overload. And remember, you need to make sure your system is behaving the way you want the auditor to see it so you can actually demonstrate that you are logging all events and have also implemented a well-defined workflow for triaging, triggering, and responding to alerts.”
The post also contains great advice on how to set priorities and implement effective workflows that distinguish and separate business critical issues from everything else.
Read the full post here. (Note: This post contains a link to the complete Threat Stack / Lola webinar.)
2. Ending Alert Fatigue: Threat Stack and VictorOps on Modern-Day Security and Incident Management
Based on a webinar we presented with VictorOps (acquired by Splunk in 2018), this post points out that alert fatigue occurs when Security and DevOps teams become so desensitized to alerts through the “normalization of deviance” that even truly anomalous activities may be ignored.
It’s difficult to justify tolerating or avoiding these problems, so this post outlines seven best practices for combatting alert fatigue:
- Make All Alerts Contextual and Actionable: Without context you can’t determine a course of action.
- Reduce Redundant Alerts: Cut inefficiency and overload by reducing and consolidating alerts whenever possible.
- Designate Alerts to a Single Source or Timeline: Unify security alerting by streamlining security functions into a single place like Threat Stack’s Cloud Security Platform®, and use an open communication channel like Slack to stream alerts, provide team wide visibility, and allow for open discussions to resolve issues.
- Adjust Anomaly Detection Thresholds: Remember to fine tune your baselines on a regular basis. This will dramatically cut down on the number of alerts about nothing.
- Ensure That the Correct Individuals and Teams are Alerted: Make sure all your team members have the right access to the right alerts so they can take effective action without wasting time and expertise.
- Customize Personal Notifications: If it’s not a high-severity alert, don’t let it trigger a page that wakes you up in the middle of the night.
- Revisit and Adjust Regularly: As you know, security is never a one and done activity. Keep fatigue at bay by continuing to make alerts meaningful and honing processes so they let you remediate effectively.
Read the full post here. (Note: This post contains a link to the complete Threat Stack / VictorOps webinar.)
3. How to Use Alerts to Become More Proactive About Security
While many companies continually react to alerts, true security maturity means using actionable alerts to proactively become more effective. This post discusses best practices for developing a more proactive approach to alerting:
- Eliminate Ad Hoc Alert Handling: Ad hoc processes waste resources and lengthen Mean Time To Know. It is always a best practice to replace these with standardized, repeatable, automated processes for triage and response.
- Optimize Alert Management: As we pointed out earlier, it can be difficult to determine the cause of a problem if you don’t have context. One of the most effective ways of adding context is to integrate your organization’s security tool with the incident management and ChatOps tools that your Ops and Dev teams already use (such as PagerDuty and Slack) so they can view all the data about an event in a single place.
- Perform a Response Audit: Defining, streamlining, and automating your alert processes is a great start, but you need to conduct response audits to confirm that you’re both following these processes and strengthening them through continual process improvement. With the results of an audit in hand, you can improve your strategy and make your entire alerting process more proactive over time, while reducing your organization’s overall risk.
Read the full post here.
4. Threat Stack Introduces Alert Trends
This post describes the “Alert Trends” view in Threat Stack’s Cloud Security Platform. The name says it all: Alert Trends is the visualization of alert history over time.
Instead of showing a single point in time, the feature lets you see alert trends so you can identify the areas you need to focus on. Alert Trends enables you to:
- Quickly detect peaks and valleys in alerts, navigate to anomalies, identify trends, and take action either by creating a rule or a suppression to handle the identified behavior
- Immediately narrow the time window of alerts to minutes, hours, or days
- Reduce the time needed to review alerts and therefore drive faster resolution times
Read the full post here.
5. Threat Stack Introduces Rapid Baselining — Transforming Data Into Actionable Intelligence
This post gives a detailed introduction to Rapid Baselining — a powerful upgrade to the way that Security and DevOps professionals find and identify areas of risk within their cloud infrastructure. Rapid Baselining groups alerts based on the associated rule and the metadata within the alert. The goal is to give you enhanced insight into alert trends and more actionable data from which you can continuously improve your security posture.
To maximize the value of Threat Stack’s Cloud Security Platform, Rapid Baselining provides a way to more effectively leverage the data at hand and tune alerts in your unique environments. By grouping alerts into logical subsets, you can create stronger workflows to:
- Investigate Anomalous Behavior: Rapidly establish a baseline and obtain much clearer insight into the activity on your workloads. By being able to create groupings, this feature also allows you to highlight unique or rare alerts, which are often indicators of malicious or unsafe activity.
- Tune Alerts: Once a baseline is established, you can quickly tune rules so they surface behaviors that are important to you.
- Strengthen Security Posture: Leverage alert metadata and organize it in a new way to easily find unhealthy behavioral patterns across the infrastructure. Catching these behaviors early and remediating the underlying causes will help you continuously improve the overall security posture of your organization.
With Rapid Baselining, Threat Stack gives you a way to investigate alerts and find patterns in your infrastructure. Regularly acting on those findings, tuning your alerts appropriately, and remediating unsafe behaviors will significantly reduce the risk that your organization is exposed to.
Read the full post here.
Looking Ahead . . .
If you’re suffering from alert fatigue, or you’re not getting value out of your alerts because you don’t have a structured approach to managing them, it makes sense to replace your current ad hoc tactics with a more proactive approach. We hope the five blog posts discussed above offer the motivation and information you need to start bringing greater control and management to your alerting processes.
If you’re uncertain about where you stand in terms of proactive security, take our 3-minute Cloud SecOps Maturity Assessment. And if the prospect of setting up proactive alerts seems overwhelming because of constrained resources or lack of expertise, you don’t need to go it alone. The Threat Stack Cloud SecOps Program℠ lets you leverage our security experts to help you monitor and triage alerts proactively to stay ahead of threats.