“Yt? Seeing something odd in the logs…”

A tale of a suspicious Linux process (with a dash of dog food thrown in)

The other day my coworker informed me, “Hey, there’s a weird process making network connections on your box.” A dreaded string of words if there ever were any for the security-conscious developer.

This is what he pastes to me:

/usr/bin/curl –G —location —max-time 60 —silent —write-out %{http_code} —data since=2009-12-312023%3A0%3A0 —data program=prof —data count=1 —output /dev/null http://131.159.28.73/popcon.php;

Now, I fully expected this to be some normal activity observed and misinterpreted, but as I saw this in the logs, I began to grow distressed. Curl POSTing to some random IP address and popcon.php?

The computer in question was not a production box but my development VM environment. This didn’t do much to ease my worries though. Developers are often the weakest links in the security chain. They have access to all the source code, the ability to push to repos, and often direct access to production boxes. If my dev environment was hacked, I would not be having a fun time of it.

Now, you’re probably asking yourself: How did he know this was running? Great question!

At Threat Stack, Inc, we kind of make it our business to know what exactly is running on your boxes, and alerting you about strange activity. New network activity from a process, well, that falls in that category. Like any good company, we dogfood our product on our dev and production boxes. My coworker saw the email alert about the strange curl process, gave me the heads up, and I proceeded right away to investigate it by diving into our user interface:

logs

Right away, I noticed a few weird things: It’s new activity (which I observed by viewing the lonely bar in the executable history diagram), and the user that ran it was ‘root’, which was even more alarming. Looking at the process tree, I observed that its parent process was a shell script.

ProcessTree

In Linux, all new processes are created by fork() and execve() from a parent process. We designed our UI so you can walk up the process tree to the parent process all the way up to init (pid=1), just by clicking on the parent process in the tree. So, I did exactly that. What launched this shell script? Was it some insidious piece of malware running active in memory? A new init script?

Log2

It turns out it was a perl script, launched by cron, located at /usr/share/pp-popularity-contest/cron.daily:

log3

Hmm. On my box, I took a look at the contents of /usr/share/pp-popularity-contest/cron.daily, and I confirmed execution of the curl command in question:

log4

I looked at the rest of the perl script, which searches for a bin/usr/bin/pp_popcon_cnt. The script appeared to post the results of this program up to 131.159.28.73 using curl.

The script didn’t look particularly malicious, although I had no idea what/usr/bin/pp_popcon_cnt was. A quick google showed me that it was part of a valid package, ‘pp-popularity-contest’:

log5

Further investigation indicated that I installed this and a bunch of other random packages earlier while performing some file modification testing! Doh!

So, despite the questionable perl script, and despite the fact I was participating in a ‘popularity contest’ I didn’t know I was participating in, it turned out this was really something benign. And I was able to determine that with < 2 minutes of investigation, which was pretty cool.

What have we learned here?

Well, for one, that my coworkers and I may be unhealthily paranoid. 😉 Certainly, 99.9% of the processes making network activity on your machines are benign, but how can you make that judgement unless you even know what is happening?

Screen_Shot_2014-10-07_at_3.11.01_PM

At my previous job, we observed companies with enormous security budgets that still didn’t know they were hacked for months and months. Even once they were aware of it, they still had a really tough time figuring out what boxes the attackers actually accessed, what credentials they stole, and what malware they installed, etc.

The thing is, you can save yourself a lot of headaches down the line if you assume that this kind of breach can happen and prepare for it proactively. No one wants to think their access controls will fail. But at the same time, you don’t want to hear a merchant saying, “sorry, we did everything we could to protect your credit card numbers, but it looks like someone got them anyway and was using them for months and months, without anyone knowing.” A little logging and a little security monitoring go a long way.