FINRA takes control of its big data challenges with Splunk Cloud
- Summary:
- The US Financial Industry regulator initially began using the tool to tackle its security challenges – but its use has now began to extend to other areas of the business.
The US Financial Industry Regulatory Authority aka FINRA, is no stranger to big data. According to senior director of information security, Gary
This is unsurprising when you consider that the regulatory body is responsible for all of the data that relates to trades occurring in the United States. To put that into perspective, this equates to dealing with more data than either Twitter or Visa heft on a daily basis. We are talking petabytes of data.
All this data has to be kept, so that if FINRA needs to deal with a complaint regarding something nefarious that has taken place, then it can trawl back and inspect it. As a result, earlier this year the regulator began searching for new security information and event management (SIEM) tools that could help it more efficiently search and inspect its trading data.
This led Mikula and his team to Splunk – a real time operational intelligence tool. Mikula said:
SIEMs typically only know about data that they know about. In other words, when an event comes in, they have to match it to a database, and if that database doesn't exist, they throw it into a bit bucket. Well, that's not good enough for us - SIEM vendors can't keep up with the other vendors, so you start losing events.
We wanted the ability to be able to pull in anything we wanted, which Splunk is very good at. It became a really good fit.
FINRA hosts all its trade data in the cloud, via Amazon Web Services. This may sound surprising given that we are talking about financial data, but all of FINRA's data is already in the public domain. This makes security clearance to get data hosted in the public cloud a whole lot easier.
As a result, Mikula took the decision not to implement Splunk Enterprise on premise and rather to make use of Splunk Cloud (which is also hosted on AWS) - as he believes that this better and allows FINRA to add value to the tool. He explains:
We were one of the first people to use Splunk Cloud. Our company in general has a good presence in cloud and we are one of the bigger customers of AWS.
We wanted to build a SIEM, but we needed huge amounts of data storage and data volume storage, and if we were going to do it on premise then we would need to buy and manage this box and that box – it ends up that you spend a greater amount of time on maintenance of these things than you do adding value.
We thought it was a good idea to offload that to Splunk - we are happy with the cloud. Let them own the base layer and we will start adding value on top of that. With cloud I can put an exact dollar amount on data. I can say that it is going to cost us $10,000 a year to give you that information – is it worth it? It makes those type of decisions a little bit easier.
Related articles
- Trade Me bids for better business insight with Splunk.
- John Lewis is weaving operational intelligence into the fabric of the business - not just IT
- Cloud comes last under the US government's Cloud First policy regime
However, FINRA struggled to identify data that had been targeted in the past. So, for example, if it was made aware of a new virus it could protect future data, but it struggled to see if previous trades had been impacted. Splunk solves that problem for FINRA, as it has the ability to quickly query and analyse all of the data held by the organisation. Mikula said:
This happens all the time with zero day malware viruses, because you don't have the signature of what that malware looks like. But when that signature comes out ten days later, how do I go back and look at that data to see if I was breached? All that you do when you push out those new signatures is you check to see if something is happening from that point on, it stops the bleeding – but you don't know if I've already been hacked.
We wanted the ability to go back through lots and lots of mounds of data.
However, having recognised Splunk's capabilities with security and events management, Mikula wanted to make the tool a corporate asset. FINRA is currently in the process of extending the use of Splunk Cloud outside of security and into applications teams, operations teams, audit teams and customer service teams.
It has been extremely well received and the user base continues to grow.We created a little dashboard, where we pulled in active directory data and user data, so that we can go and see how many unique users we have each week and see what department they work in. You can easily see that it is not just security people looking at this, you can see that there's lots of different groups of people using it.
For example, employees that have to troubleshoot stuff from the call centre, what they intended to do was upload all this data themselves and build their own app. But now they can build that in Splunk and get it delivered to the call centre. So now if a broker calls up with a problem, the call centre can go and get into the data with Splunk. The business side is just starting to see what they can do with the tool.
Mikula advises any company that is planning to roll out Splunk across their organisation to sanity check the data before you start uploading it. Splunk charges customers based on how much data they upload, and so it can quickly become an expensive tool to use. Mikula said this doesn't necessarily have to be the case if you are clever with how you use your data.
Create a test environment and scrub your data before you go into production. People complain about cost. We were previously pulling in 100 gigabytes of data a day on our old SIEM. We cut that in half with Splunk.
If you take time to figure out what problems you are trying to solve - then move the data up once this is done. Splunk is more than happy to suck up your data, but they give you the tools to be smart with it so you can keep your costs down.
