The History of Log Management
Compliance mandates that were developed as a result of corporate scandals in the late 1990’s. The initial legislation really didn’t have a lot of traction as far as enforcement or repercussions. Compliance requirements that came out later took more of a best practice approach which was an improvement. When it comes to log management, passing security audits were fairly simple. Organizations had to find a way to collect log data for historical purposes, but they didn’t really use it as a significant source of information. Things have started to change. The latest revisions of compliance requirements are a rather serious attempt to help organizations increase security.
This is a double-edge sword because if a serious incident occurs involving customer information, serious assessments and after-the-fact investigations take place. These investigators are determined to look for negligence on the part of the corporation. If they are found to be at fault, large fines, lawsuits, loss of accreditation and large contracts could follow.
What can be attained from Log Management?
As a result of data theft incidents, privacy laws and new security threats the log management landscape has changed. Customer data breaches have made revised compliance and security initiatives mandates more stringent. PCI Compliance and SOX legislation have forced organizations to seriously change the way they view log management. If they don’t change in a hurry they could face serious fines from regulatory bodies, public outrage from data theft incidents, lawsuits and loss of revenue due to the ill effects on the company’s reputation. Companies are realizing that log data can be used to improve several key areas.
Organizations Use Log Data for 3 Key Value Propositions:
- Achieve & Maintain Compliance
- Increase Network & Data Security
- Uptime & Operational Performance
Achieve & Maintain Compliance
Compliance enforcement has changed. Depending on the compliance mandates, auditors are charged with deeming organizations with meeting the mandated requirements set forth for each business vertical. Mandates like SOX, PCI, FISMA and HIPAA are all getting more stringent and more challenging to achieve and maintain.
Common Requirements Include:
- Full Audit Trail of All Activity
- Log Retention from 1 year to 7 Years
- Log Integrity protection
- Regular Review of Reports
- Data Security and Monitoring
These stringent hurdles require companies to centrally aggregate log data protect event logs from abuse and maintain log data for several years. IT professionals also have to generate in many cases weekly monthly reports on specific areas of focus. All of this requires automation.
Don’t Overlook the “BIG PICTURE”
Many organizations make the fatal mistake with being ONLY concerned about finding a central log aggregator without looking at the big picture. Although the log management and log retention requirements are challenging, they are but the tip of the iceberg. According to the SANS 2008 Survey of over 650 IT professional, they stated that log servers alone were not enough to help identify security incidents, meet compliance or increase operational uptime.
Common complaints from the respondents were the "log volume was overwhelming to find any useful information" "correlating log activity is too difficult" and “manual efforts to produce reports is next to impossible” were among the comments. These comments imply that more intelligent methods including automation are required to make Log Management projects a success.
What are Other Logging Solutions Missing?
All other log management solutions on the market follow the same methodology of log aggregation of all event logs without any log research intelligence. Just to clarify, other logging solutions do provide normalization of log data. Normalization is not the same as intelligent. Normalization simply means to put all the logs into the same uniform format for easier reading purposes. This does not solve the underlying problems that crop up when customers try to gain any real data from logs.
Many commercial logging solutions attempt to cover up this fact, by spending most of their efforts on summary reports. At first glance, this tactic can mask the truth about their lack of filtering. The truth is, reporting is only as good as the data it was derived from. These generic methods are simply not good enough for today’s standards. As enterprises have evolved and threats have gotten more challenging to determine, IT professionals need innovation. The mindless log aggregation is at the heart of the problems that IT professionals are facing.
LogClarity® has a completely different approach.
The LogClarity® Enterprise Solution analyzes log data in real-time based on log research. The Log research knowledge base is built right into the collection engine so that LogClarity® can parse the data to filter and correlate logs based on their known definitions. This technology is known as the LCDF Technology. What this means is LogClarity® has intelligent log aggregation capabilities which are unlike any other logging solution.
The LogClarity® Enterprise Solution can also provide the automation tied in with the intelligence. All log grooming, log protection and compliance reporting is all built into LogClarity. This makes meeting compliance and maintaining the long-term log management lifecycle possible.
Increase Network, Domain & Data Security
In addition to the compliance mandates, security threats are a major reason for logging. Attacks towards networks, domains and most importantly, to customer private data have emerged. The variety and complexity of these threats are pushing the envelope for intelligent log use.
Perpetrators are moving the focus away from network attacks towards stealing valuable data. These rampant data theft incidents are taking place both from external points of entry and internal ones. Identity theft has become a huge revenue stream for the black market. As a result, data theft has increased dramatically since 2007 and is not going away anytime soon.
Types of Customer Information that are Targeted:
- electronic Private Health Records
- Cardholder Data
- HR and Bank Account Information
- Corporate Intellectual Property
- Social Security Numbers
All business sectors are affected by this problem. The data theft phenomenon is driving a much more intelligent use of log data. Today, log data needs to be used in a proactive manner rather than languishing after-the-fact investigation procedures.
The LogClarity® Enterprise Solution provides a full audit trail of event logs across all sources, including databases. This is unique in the industry. All other logging solutions stop logging user/application activity once they have logged into the database. They cannot track user activity at the most critical point; Data Access. LogClarity® provides unmatched logging and monitoring of user interactions with critical database tables. All user activity is audited, including privileged user activity. This enables customers to gain true accountability and the entire audit trail of information.
LogClarity® also provides a proactive approach to detecting and mitigating security incidents, compliance violations, including network and system issues. LogClarity® comes with predefined alerts including response measures that can be used right out of the box or customized for specific needs.
Built-in alerts for over 375 incidents ranging from:
- Network problems
- Security Policy Infractions
- Privilege abuse
- Data theft Activities
- Unauthorized Access
The LogClarity® Incident Management Component provides a powerful framework which encompasses network activity, domain activity, database activity and data activity on servers. LogClarity® enabled security response measures to be used universally across the enterprise without limitation.
Uptime & Operational Performance
Administrators are tasked with a variety of duties. Network, domain, and system uptime is one of their highest priorities. Rather than waiting for “help desk calls of distress” administrators can utilize log data as a valuable source of troubleshooting operational problems across the enterprise. This provides a more efficient and expedient method for handling trouble. Everybody can win by implementing a solid strategy of reliance on log data.
The problem most administrators are contending with when it comes to log data is two fold. First, the sheer volume of log data is overwhelming. Millions of logs are generated even within a medium size enterprise. It is virtually impossible to expect administrators to search through logs manually to look for potential problems. The need for Incident Management processes is clearly evident for early notification of system incidents. Administrators can utilize the same Incident Management framework for operational problems as well.
The LogClarity® Incident Management Features:
- Built in Alerting
- Powerful Response measures
- Incident Status Tracking
Incident management needs to be applied across the entire enterprise to be successful.
Alerts can be defined to identify violations of compliance, security activity, and network and system trouble. Security teams, enterprise administrators and dbas will no longer have to feebly attempt to manually sift through millions of event logs to find the needle in the haystack. LogClarity’s Incident Management will dramatically help mitigate security threats, increase network and system health and improve the overall usability of log data.
LogClarity® provides organizations with a common security/operational Incident management and response framework that can be used enterprise-wide
