One of things we like at Threat Stack is magic. But since magic isn’t real, we have to come up with the next best thing, so we’ve hired one of the libevent maintainers Mark Ellzey Thomas (we like to call him our ‘mad kernel scientist’) to make our agent the best in its class.
Many of the more savvy operations and security people that use our service are blown away by the types of information we can collect, correlate, and analyze from Linux servers. They say something to the effect of, “I’ve tried to do this with (Red Hat) auditd, with little to no success… how do you guys do it?”
The Linux audit subsystem is a very powerful way to collect information about system calls and other security-relevant activity. The best part: no kernel module is required to enable this detailed level of auditing since it’s built right into Linux. You have the option to write a log any time a particular system call happens, whether that be unlink or getpid. Since the auditing operates at such a low level, the granularity of information is incredibly useful.
Traditionally, people have used the userland daemon ‘auditd’ built by some good Red Hat folks to collect and consume this data. However, there are a couple of problems with traditional open source auditd and auditd libraries that we’ve had to deal with ourselves, especially when trying to run it on performance-sensitive systems and make sense of the sometimes obtuse data that traditional auditd spits out. To that effect, we’ve written a custom audit listener from the ground up for the Threat Stack agent (tsauditd).
We’ve asked our agent engineer, Mark, to make a video that highlights the performance, parsing fixes, and other changes we’ve put in the Threat Stack agent by demonstrating a ‘showdown’ with Red Hat’s audit. You can view it in all of its glory here (Warning: Mark tends to ramble):
Here are a couple of highlights from the video, along with some other facts that make our audit listener special:
1. Performance Enhancements
Many people have tried to use traditional Red Hat auditd in production to do very detailed auditing of user, process, and network syscall activity, but have failed due to the performance impact. We’ve ensured that our agent is responsible with resource utilization through our unique parsing model. In fact, while benchmarking a web server, we saw auditd consume 120% of the CPU. Threat Stack’s agent CPU consumption was only 10%!
2. Output Enhancements
This Linux audit system is special, and by special, I mean it will output many different lines across disparate events into syslog, which you then have to correlate later via your ingestion engine or your log management system. The key-value format is also cumbersome to parse, and values are often encoded into hex randomly. We’ve decided that all related events should be grouped together and have conveniently parsed everything correctly for you. We then transformed that to a JSON output format that is much simpler to read and parse.
3. Network Tooling (“src/dst port”)
Tracking network connections across multiple hosts can be a manual and painful process when trying to connect across boxes. To make it easier, our agent adds metadata to network connection events to determine where the connection is originating from and where it is going. Our backend is then able to correlate these network events to determine the originating process and potential user activity that caused that network event, so long as the agent lives on both the source and destination server.
This is especially useful for tracking SSH sessions across your environment and debugging what servers are speaking to one another and why.
4. User Activity Auditing
Digging around the server logs to see where a user on your system went is not an easy job. You’d need to manually find the agent and session that a user connected to, yet all the kernel gives us is a nasty hex encoded string representing the connection address in the traditional auditd logs. On top of that, most of the information logged by auditd is not really relevant, and hard for the human eye to parse. To correct that, we’ve designed Threat Stack to keep storage of events, activity, and commands associated with a logged in user, and automatically reconstruct this information into a clean, compact, and readable timeline.
Stay tuned to read about some of the other engineering feats we are accomplishing at Threat Stack!