A post based on the talk I just gave at SOURCE Boston 2017
If you answer Yes to one or more of the following questions, you probably have agent fatigue! Do not worry, I’m here to help and we can work through this.
- Do you often find yourself booting into safe mode?
- Do you regularly look for programs in the taskbar to kill?
- Do you look for reasons why your computer seems so sluggish after IT did something to it?
- Do you wonder why you even pay for that thing on your computer?
- Do you have employees who complain about installed software?
- Do you look for ways to meet compliance requirements in software?
- Do you care about security?
First, a disclaimer: I am not an independent observer! I lead the agent team here at Threat Stack. I have a horse in this race, but I still think I make some good points. After all, I use other people’s agents too! So let’s get going on the path to acceptance:
- Recognize the fatigue
- Look at what causes agent fatigue
- See why we need agents
- Observe the choices in agent types
- Learn how to evaluate agents
- Accept the situation and move forward
What is Agent Fatigue?
A Google search for “security fatigue” comes up with a lot of articles. Just this past fall, NIST published a study on how “security fatigue” can cause computer users to feel hopeless and act recklessly. Why does that matter? That study describes “normalization of deviance.” Normalization of deviance is cited as one of the reasons for the Challenger shuttle explosion in 1986. Leading up to that catastrophe, engineers discovered a problem with the putty used to seal the O-rings on the rocket boosters. They then analyzed the putty to determine its limits, repeatedly reinterpreting failure as within the bounds of acceptable risk. Acceptance of that risk led to the explosion. Security fatigue leads to normalization of deviance.
That brings us to agent fatigue. Here’s a good quote to sum it up:
“The term agent fatigue is widely used to describe this phenomenon on the desktop. Are viruses a problem? Here is an antivirus solution. Is command and control communication the problem? Here is a Host-based Intrusion Detection System (HIDS). Need to keep track of all the software and versions installed on a system? Here is a compliance agent. The list of agents goes on and on. Each agent serves a different purpose, communicates to a different control server, and is managed by a different group within the organization.” — Building an Intelligence Led Security Program by Allan Liska
Causes of Agent Fatigue
Let us consider the causes of agent fatigue. The most obvious cause is cost, and not just money. If you purchase an agent-based solution, then it’s going to come with some licensing fees, but what about the implication for the compliance regulations? A new piece of software can introduce latency to established workflows, or maybe even full-on road blocks. Additional software also represents a greater attack surface, and the potential to interfere with host behavior. If we do consider the money, consider the money beyond just the licensing fees. An agent-based solution needs human resources to deploy the agents, upkeep the agents, interpret the data from the agents, and act based on that interpretation. Plus, an agent will eat compute resources.
In my opinion, the biggest cause of agent fatigue results from people buying features beyond their preparation or commitment to use them. The user can end up frustrated. It’s not because the tool isn’t good, but that it wasn’t the appropriate tool for the game being played.
The Need for Agents
We need agents. Agents provide data we cannot get any other way. The network cannot give us this data, but the hosts can. Since the hosts actually have the valuable assets we care about, it makes sense that we’d want to provide some protection on the host. You may have been burned by particularly bad experiences in the past. That does not indicate future failure! We must learn to judge between them for the best fit.
Defense on the network is not sufficient. The leaks from Edward Snowden have significantly escalated the adoption of SSL/TLS. This makes Network Intrusion Detection Systems blind to 70–80% of encrypted traffic. It’s true that specialized Network Processor hardware can break security, but that’s not always practical or even a good idea. It’s not even an option with cloud providers. This white paper from NSS Labs documents the situation.
What’s more, the idea of a defensible perimeter has vanished. Gone are the days when we can imagine a Maginot Line that will protect our assets; we need defense in depth. The trend of Bring Your Own Device means that an employee can bring the threat inside the perimeter. Even without BYOD, the employee could bring company assets outside of the perimeter. It really is a porous membrane. A quick note about trying to hide your cloud provider as a defense: They don’t accept the responsibility, so do not try to give it to them. Your cloud provider should not be a Single Point of Failure.
There’s huge potential on the host for both defenders and attackers. Linux provides tools to look at applications, hardware, memory, I/O, and just about everything else. Windows provides similar tools. You have to be present on the box to get any of it. Attackers want these compute resources. They can use them for creating botnets, mining bitcoin, stealing credentials, ransoming data, or leaking intellectual property. The network is important, but it’s only important because it’s the path to the hosts!
Types of Available Agents
Once you accept the need for an agent, you quickly see the almost intimidating variety of choices available:
- Build vs buy: Consider the total cost of ownership, something I’ll revisit later in this post. Do you want to be a security company, or a secure company?
- Open-source vs proprietary: Honestly evaluate the arguments here. Most of the reasons for one or the other have evaporated in the last few years.
- Cloud, server, workstation, IoT: On the one hand: they’re all computers. On the other hand: Do you really want the same agent on your servers as your iPhone?
- Kernel vs user: With great power comes great responsibility. Does the agent need the power that comes in the kernel, and are you willing to accept that risk?
- Visibility vs prevention: Not necessarily mutually exclusive, but my experience indicates different design and implementation philosophies.
Selecting an Agent
With so many types of agents provided from so many different sources with so many different features, how does one decide? In addition to considering the previous attributes, here are some additional criteria to consider:
- Cost: Make sure you consider total cost of ownership. That includes all the “ilities:” Availability, Scalability, Reliability, etc. Do you want your talent working on security or other business goals? That includes the caring and feeding of your security solution. Make sure you’re prepared to pay the price of actually deploying the solution, looking at the results, and tuning the performance.
- Service: The actual agent-based solution may only be the tip of the iceberg. Do you want to manage your own security, or outsource it? Either way, do you need training on the solution? Make sure you consider what support you want and need for the service. It’s fine if the only support is stackoverflow, as long as you know that going in and that’s all you need.
- Benchmarks: It’s really easy to do benchmarks wrong. Everybody with an agent will say it’s “lightweight.” That’s a marketing requirement. They’re not lying, but the phrase is meaningless. You must figure out what resource utilization matters to you and how to measure it. Make sure the agent doesn’t do some kernel trickery and hide its computation in the OS. Make sure you measure resident set size and not the entire virtual memory usage. Do the legwork to figure it out for your environment, because you’re the only one who can.
- Sensors: Consider what the agent will monitor. Will a certain agent actually monitor multiple things, obviating the need for several other agents? In addition to what it senses, consider how often it senses it. If you need a continuous monitoring solution, then do not settle for something that scans.
- Actuators: An agent can take actions based on what it senses. It has to be more than logging. John Strand said “Right now, logging in the cloud is an absolute complete unmitigated train wreck, as far as finding out where your data is.” Start off by checking for the alerting the solution provides. Make sure those alerts provide severity for prioritization as well as context for analysis. Additional actions beyond alerting include taking actions on the host to change behavior and possibly prevent attack escalation.
- Integration: The real power of a solution will come from integration. You already use ChatOps, DevOps, SecOps . . . all the Ops. Make sure you can integrate your agent-based security solution into your existing workflows.
Final Words . . .
I admit that agent fatigue exists, and it exists for valid reasons. We need to get over it. Agents provide critical value. They provide vision to the assets instead of only around them. Attackers want the hosts, not your network. You must choose solutions that match your needs. I may be biased, but I think Threat Stack makes a wise choice in most cases.
The Threat Stack Agent offers multiple deployment options, so you can add security without slowing DevOps velocity, and it is well positioned to capture:
- User privilege escalations
- Modification of the kernel at run time
- Processes running from /tmp or other special system directories
- Audit trails of critical filesystem changes
- Monitoring of host system network services
To learn more, contact us today to set up a free demo and to discuss your security and compliance needs.