Security information and event management (SIEM) software provides a way to track, integrate, and analyze the vast amount of log data that comes from an organization’s applications and network hardware. Its original value came from compliance reporting, but it is becoming an increasingly valuable tool for security.

Log files are everywhere

In any organization, end-user devices, servers, network equipment, firewalls, and a host of other components continually generate log files, time-stamped lists of software actions and events. All events generate a record, but which events indicate something anomalous or malicious is often not immediately clear.

SIEM systems gather data from multiple locations to permit the detection of trends and patterns that might indicate a problem. SIEM grows out of earlier software called SIM (security information management) and SEM (security event management), and its first generation was back in the 1990s.

SIEM’s origins in compliance

The original impetus for a SIEM solution was not security per se, but compliance. Organizations needed a way to meet the monitoring and reporting requirements of mandates such as HIPAA, Sarbanes Oxley, and particularly the Payment Industry Data Security Standard (PCI DSS). Collecting and analyzing the events captured by log files provided reports on non-compliant activities and policy violations, time-ordered event data for compliance auditing, and evidence of how well threat prevention efforts were working.

SIEM’s use for threat response has become relatively more important in recent years, but compliance reporting remains a key function.

Sources, formats, and integration of log files

Log files were developed on a case by case and system by system basis, and so lack a common standard. They are largely plain text, and are written to be readable by humans, not by computers. But the volume of them is far too large to be usefully read by anyone.

So one important SIEM process is to break all log files into common components and normalize them into a database table. SIEM vendors typically list the hundreds of log file formats they can automatically incorporate, updating the list regularly. A few allow for general integration of all log sources, at the cost of a lot of upfront work on the part of the administrator.

The increasing security uses of SIEM tools

Many SIEM offerings now incorporate threat intelligence feeds (often from third parties) and provide additional security analytics to track network behavior as well as user behavior to distinguish between benign and malicious activity.

And the analytics included in SIEM packages are increasingly based on machine learning, statistical analysis, and other more sophisticated big data capabilities.

SIEMs generally offer automated response capabilities to block malicious activities in real time. These need to be configured by the organization to match its own implementation and environment.

The human side of SIEM

Software, even sophisticated software, is only as effective as the underlying processes and workflows that it is tracking, supporting, and reporting on.  In general, such software should be used to monitor and automate lower-level processes so that staff can focus on analysis, response, and planning—and in providing executive decision makers with a clear view of what is going on overall.

So anyone implementing a SIEM must focus on what the organization plans to accomplish. A clear understanding of how SIEM will help it accomplish those goals will give the software acquisition greater value. Smaller organizations should recognize their limited resources and identify the infrastructure essential to keeping the business running, and focus on those specific log files.

SIEMs make incident handling more effective. SIEMs often provide data visualizations and search functions to support this. They are a tool to managing existing security systems, and allow for a single view of network activity, over time providing a skilled operator with a more intuitive sense of system state.

SIEM’s place in the market

SIEM users have typically been large enterprises, because of its cost and the cost of the skilled talent needed to get the full use of it. But its capabilities can be attractive to smaller organizations as well, as long as they can focus on business-critical functions for a cost-effective implementation.