How to Track Agent-Based User Activity

See Threat Stack in Action

Threat Stack secures your cloud infrastructure workloads. See how.

Book Your DemoDemo

More often than not we’ll need to go beyond a Severity 1 alert to figure out what a user (including a potentially malicious attacker) was doing on a system. Host events in particular only show a small part of the picture, and a single alert can’t always give you the context necessary to make an escalation decision. This blog post explains how to pivot from a Host event to a user’s session and how to move from a single user-related alert to the user’s session using the data provided by your intrusion detection system.

Host Event Pivoting

Alerts that are triggered by a user failing some sort of authentication are common. The most common occurs when a user fails sudo authentication. (See the example event below.) Others could be triggered by a user failing to SSH into the system or a brute force attempt.


Example 1

In this example, I purposefully failed typing in my password so I could trigger an event. However, there is almost no way for you to know the reason behind a login failure, and you will need to investigate further in order to make a decision. Luckily the Host event gives you enough information to begin your search. You’re going to want to craft a search with the server (agent_id) and the user
while keeping note of the time the event occurred. This will look something like agent_id = "$agent_id" AND user = "$user" and may return a lot of results because it will show all the activity from that user on the server. Depending on your time parameters, it may pick up activity from before the login failure.

Note: If it’s an SSH failure or brute force, you’re probably not going to find very much. Brute force attempts often try accounts that don’t exist, and an SSH failure probably won’t have a command associated with it.

Example 2

If you’re lucky, you might be able to spot the command that caused the authentication failure immediately and be able to make a decision from there. However, if that’s not the case (and I bet that nine times out of ten it won’t be), then you have more than enough information to continue crafting searches. As you can see in the example above, I didn’t capture the command that prompted the authentication failure, but I did capture other activity by my user, which gives me some clues. The most important things you’ll want to pull out are server, session, and user although you should already have two of these from the first search.

If you see 4294967295 as the Session ID, this means a loginuid is not set, and this is most likely system activity or some sort of web-based session (like Teleport). Searching on this Session ID will likely yield more noise than helpful information.

Depending on how busy the user is on the system and the time between when the alert came in and when you started the investigation, you might have to dig around a little bit to find the right session. This is where the timestamp from the Host event comes in: You can either add it to your search and hope you get lucky or use it to narrow your timeframe using the query string parameters while searching. Once you have what seems to be the right session, add it to your search. Your search string will now be agent_id = "$agent_id" AND user = "$user" AND session = "$session", which will return similar results to the previous search but will not include Host events. Other options for pivoting include things like syscall, type, command, arguments, or exe. Some examples are syscall = "execve" or type = "connect" or command = "httpd" or arguments = "/bin/bash".

You can begin an investigation in a number of ways. Starting with syscall = "execve" is an effective way to see commands run by the end user because this will remove forks, clones, connects, and accepts. It is also recommended that you step through the pages to get a better picture of the user’s activity on the system instead of focusing on single events. Focusing on just one user in the session will not give you the whole picture, especially if sudo or su are involved. If an external authentication system (LDAP/AD/Kerberos/etc) is used, then this will show conflicting information. Because pam_unix will report a local authentication failure, they could still succeed with the external authentication service. Additionally, even when using local authentication, the user has three attempts to get their password right with sudo; if they fail the first two times and succeed on the third, there will still only be one Host event for the failure.

You have a few options on how to proceed in your searching:

  • First, you can filter out the current user to see events from other users in the session: agent_id = "$agent_id" AND user != "$user" AND session ="$session".
  • Second, you can show events from all users in the session (useful for seeing a user running sudo and then the event from root right after): agent_id = "$agent_id" AND session = "$session".
  • Third, and final, you can show all successful elevations to root: agent_id = "$agent_id" AND event_type = "host" AND sigid = "5402".

Any one of these will give you the information you need to make an informed decision on escalation. Examples of each are shown below:

agent_id = "$agent_id" AND user != "$user" AND session = "$session"

Example 3

agent_id = "$agent_id" AND session = "$session"

Example 4

agent_id = "$agent_id" AND event_type = "host" AND sigid = "5402"

Example 5

From here you should have a good idea of whether or not the user is actually allowed to sudo to root and what commands they ran. Additionally, it will give you the context necessary to decide whether to escalate to the customer and enough information to give to the customer if you are escalating.

User Alert Pivoting

The second scenario that may cause you to pivot into a user’s session occurs when you get an alert that doesn’t give you enough information about what the user is doing. A good example is User Activity (Privilege Escalation) when a user runs sudo -s or sudo su to elevate to root and run commands. Based solely off the alert, you don’t have much information to go on aside from the fact that they elevated to root. An example is shown below:


Example 6


Example 7

Thankfully you have more information to go on and can quickly dive into the events surrounding this alert. You can do this using the four search parameters I mentioned earlier:

  • agent_id = "$agent_id" AND user = "$user" AND session = "$session" — Shows you events from the specified user in the session.
  • agent_id = "$agent_id" AND user != "$user" AND session ="$session" — Shows you events from users other than the specified user in the session.
  • agent_id = "$agent_id" AND session = "$session" — Shows you events from all users in the session.
  • agent_id = "$agent_id" AND event_type = "host" AND sigid = "5402" — Shows you successful elevations to root.

Below is an example of agent_id = "$agent_id" AND session = "$session" with an addition of syscall = "execve":


Example 8

In this case, the activity is both immediately apparent and obviously suspicious. How much additional investigation you do is entirely up to what you find during your searches and your methodology. From here you can trace back specific commands, narrow down your search by time or indexed fields, or just step through the user’s activity from login to logout. However you do it, make sure you have enough information to take to the customer if you decide to escalate the alert.

Wrapping Up . . .

You can use the tips and search parameters given above to investigate a wide variety of alerts, and they may come in handy when you are attempting to establish context around an alert or event.

See Threat Stack in Action

Threat Stack secures your cloud infrastructure workloads. See how.

Book Your DemoDemo