It is time cybersecurity deals with its demons. The infrastructure we protect has changed, and security needs to change too. Business solutions have always treated cybersecurity as protecting the network in order to protect users. The brand network and security companies today are continuing down this path of logic, leading their company and their clients into a losing scenario. A combination of cloud capabilities and verbose cloud-managed endpoint solutions was already disrupting security operations. The rise in attacks contributed to work-from-home users shows that while security is designed to protect people in an office, business is no longer there. Worse, while security architects keep trying to bring a network infrastructure to the business cloud world, attackers have moved on to reality.
We need to have a reality check. The office network no longer encompasses the companies infrastructure. Tools like Zscaler, which try to create a virtual controlled network, fail on many levels. The most obvious issue is that of cloud services, platforms (containers), and infrastructure. Other technology issues of Cloud infrastructure make security information event management (SIEM) insufficient. The use of endpoint detection and response (EDR) could solve much of the endpoint problem, but the amount of data from these systems when actually turned on is too much for SIEM technology.
The obvious solution of collecting data from all the SaaS, IaaS, PaaS, EDR, and legacy network infrastructure means you must deal with the three demons of big data analytics. The three D’s are:
Diversity: Each service, platform, and infrastructure element has its own audit log. Some audit is flat, others are nested objects. All have variants of field names, types, and meanings. Providing understanding requires inherent knowledge of each service and how they relate.
Distribution: Collecting the data can be an issue all to itself, as data is spread over the internet, and the means to collect the data can be different. While syslog and RESTful APIs are common, data density and timeliness have some services using webhooks and Kubernetes.
Density: The amount of data by the user is exploding. Once analyzing firewall and flow data was the big hurdle for security analysts. Companies normally do not analyze flow data due to its density. Raw EDR data, needed for security work-at-home users, is ten (10) times denser. An average user produces almost a gigabyte a day of raw logs, for some EDRs. This is the main reason EDR companies limit the data retention to 7 to 30 days. A far cry from the one-year data retention required by most regulations.
Now your 500 person company produces half a terabyte of audit data a day. Pretend your legacy SIEM can handle that, now ask yourself "How do I analyze all this data? In all its formats? With all the different source types?" This is not a Splunk issue, Snowflake, or an Elastic one. This is an issue of being able to perform two basic functions:
Detect behavioral events at data ingest speeds before its written to the database
Be able to search a year’s with a complex search in a reasonable short period of time
At Fluency, we have spent eight (8) years solving these problems. The short answers are: streaming analytics to detect and a custom big data database designed for streaming data to store and search. These are the two basic features you must have in security operations. The ability to detect new events, and the ability to analyze historical ones.
Fluency has handled in real operations ingests and storage of 2.8 million events per second with a three machine cluster. It performs user entity behavioral analytics (UEBA) functions at 400k EPS per node. It performs complex Lucene searches of petabytes of yearly data in minutes. Your network is not going to stress Fluency, but it will stress your current SIEM and other network tools.
If you are serious about security, then Fluency is a tool you should look at. There is much more to Fluency than the ability to handle data loads and perform streaming analytics. But if your current solution cannot handle the three D’s, there is already a reason to take a look.