Information Systems Record Events On Log Files

An event log should be maintained and record any exceptional, out-of-routine security events, including intelligence and other guidelines relevant to security,

The analysis of big data is the process of organizing, collection, analyze and examining the large volume of data to find patterns, market trends and useful information. This analysis helps organizations to better understanding about the information within data, and helps analyst to make better

Currently, there doesn’t seem to be a good method for collecting, maintaining, and correlating cyber-incidents. As US-CERT provides the standard vulnerabilities database, does other entities offer databases for cyber-incidents? This topic will examine available data on the Internet and how academia could combine various repositories to build and maintain such an environment. Therefore, will be investigating and mapping incident repositories availability for academic cyber security research.

This report presents a brief introduction to the Big Data concept and Operating Systems. The definition, use, quantification, and sources of big data are covered. This report further presents the challenges facing Big Data in the current business world and relates Big Data to Operating Systems. Also,it is presents the solution of Big data with some tool that used in the solution which is hadoop. The application of modern technologies in Big Data management such as internet, social media, and cloud computing are discussed.

One of the most fundamental questions concerning Big Data is the question of how big the data should be in order to be treated as Big Data (Fingar, 2011). Probably the best answer is that it depends. The basic answer is whenever the system needs to span horizontally in order to handle the data, that probably there is a need to shift towards Big Data solutions (Fingar, 2011). Also, the size of the dataset is itself a significant part of the problem. Typically in modern discussions experts are operating on terabytes and zettabytes.

Many companies have already implemented continuous logging and centralized secure log server with restricted access to very few people. This could be used to detect and investigate changes that occur infrequently such as changes to operating system files, scripts and executable services. All these changes could come in the form of software patches that could be pushed down either from the server done by the system administrator or possibly malicious insiders who tried to modify the system to their own needs. As some computer system contain sensitive data, such as the client database and company portfolio, companies actually exercise control of the transfer of data from the system to removable storage devices such as flash drive and portable hard disk by coming up with a logging system. Another way is to do auditing on various areas and keeping records of them. Possible things that companies audit on their secure systems include the following:

Big data is nothing but collecting of datasets. Organizations in current world demands data to be broken down which can used to get more high effectiveness and benefit. Big data refers to the large amounts of data which collected from various devices such as mobiles, sensors and social media etc. Generally, large amount of data have been regenerating by IT industry such as satellite data, mobile devices and etc. This data is being growing rapidly day by day and it would be referred as Big Data.

In a fast paced, business ordinated technological world the overall welfare of a company is tied to the success or failure to make the tough decisions. On one instance a company’s CEO might be able to make the choices based on experience, advice, or simple gut instinct. However, this is not the only skill one needs. There is a great deal of information to be found in being able to see investments in data and analytics. These decisions are based off of big data. Big data is a catch-phrase, used to describe a massive volume of both structured and unstructured data that is too large to process using traditional database and software techniques. The volume of data is in most cases is too big, moves too fast or it exceed the processing capacity the company has. Despite these potential drawbacks, big data contains the potential to help companies by improving operations and making faster, more intelligent decisions. This can be broken into three key parts, knowledge, data, and information.

According to (Gartner Inc, 2013) Big Data is high-volume: the amount of data is extremely large, high-velocity: the data can be gathered quickly in real or next to real time and/or high-variety: the data can take many forms both structured and unstructured. This data then requires suitable processing to give added advantages to

Big data is the present most-liked theme of today 's technology. These research goes through all description of techniques and technologies of extracting of the data, storing of data, distribution of data, analyzing of data, managing of data with high velocity and from the structured data and helps in the handling of the extreme data. Big data has the presentation the capacity to improve predictions, saving money and enhancing the decision making process in the fields of the traffic control, weather forecasting, disaster prevention, fraud control, business transaction, education system, health and the national security.

With the growth of technology, there is a substantial growth of data by volume, variety and velocity satisfying the criteria of Big Data . Volume of data already exceeded 100 EB at the end of 1990s, reached 1.8 ZB at 2011, and we have already entered in the age of ZB. By 2020, it was forecasted that the volume of data will be 50 times bigger than the one at 2011 . Big data is used to describe the huge data sets (terabyte to Exabyte) and Big Data analytics are the techniques applied on them.

The paper ‘Event Correlation for Log Analysis in the Cloud’ proposes an approach for correlating different log based events so that these logs can be segregated [4]. It uses Infrastructure-as-a-service (IaaS) like OpenStack and through various correlation condition like Attribute-Based Correlation, Conjunctive Correlation, and Disjunctive Correlation and through this correlation criteria are set. Once policies for correlating events are defined, association rule mining can be applied to the log data. Based on the mining, classification and clustering of the events can be done which can later be used for detection of attacks and abnormalities in the system.

As Big Data grew and environments supporting Big Data become more robust, the data being stored by businesses evolved in complexity as well. All manner of nonstandard text (music, images, freeform text, videos) began being captured in the Big Data ecosystem. The changing needs of the data environment resulted in the creation of NoSQL databases. These databases (built on work done at Google and at Amazon) were optimized to store and retrieve data modeled using non-traditional non-tabular entities. And of course, Big Data cannot exist in a vacuum – it requires tools to process, analyze, and display the vast amounts of information in a manner comprehensible to mere humans. This has led to the rise of Big Data BI.

Monitoring and logging are important to any information security program. In general, monitoring ensures users are doing legal activities on company systems. To begin with, a risk assessment determines what computers and systems to log, and naturally, the information security team monitors the high-risk systems. Next, trained personnel configure systems to facilitate monitoring and logging to track security incidents with approved system utilities or auditing tools, in other words, scripts, log management software, and security incident event management (ISO, 2005). Also, management will pre-approve tools, and controls will safeguard operational systems during the analysis process. Consequently, monitored systems and security events generates an audit log entry, thereby producing a time-stamped reference trail. In the end, the monitoring and logging policy will aid in protecting electronic protected health information (EPHI) on information systems.

Understanding and applying the use of big data and data analysis creates advantages for businesses. As such, this paper discusses big data and analysis regarding differences between big data and small data, characterization according to 3 Vs, big data applications, data analysis, types of data structures, obtaining sources, tools and processes used for analysis, and advantages of data analysis.