At Threat Stack we write a lot of software around NSM, data collection and processing at a large scale to support our cloud based incident response system. We recently decided to write our own custom unified2 spooler and eventually decided to open source it as a Barnyard 2 replacement.
We designed Pigsty (https://github.com/threatstack/pigsty) to be very extensible with a focus of performance. We wanted that have the ability to quickly support new data stores, communication protocols and preprocessing of data post-storage. Pigsty was our answer to that issue and we are happy to announce 0.1.0 is now available to the public under the GPLv3.
Pigsty is very similar to barnyard 2 when it comes to the CLI. Below are the current Pigsty options available.
Pigsty also comes with a configuration file which can be used to store output plugin settings and paths to your reference files.
Building a new output plugin is really simple. Please see the example plugin (https://github.com/threatstack/pigsty-example-plugin) hich simple logs the event to console. We have examples of outputting to MongoDB, SocketIO, and others right on our GitHub page.
We built an example application using Pigsty and the Mysql/Websocket output plugins. This example application (http://snorby.org:3009/) is connected to the IDS sensor for http://demo.snorby.org and clicking `send attack` in the top right corner will trigger rules to fire for demo purposes. All events are streamed in realtime.