Big Data Belongs In A Warehouse Not A Silo

The emergence of big data has provided different avenues for organizations to use data to improve different aspects of their respective operations. Be it customer service, research and development, or market position, Big Data has the potential to be a significant driving force in all these areas. However, there’s still a significant gap between the ability of Big Data to produce insightful analytical information based on real-time data and the ability of organizations to capture and utilize this readily available tool. This is, in part, due to the fact that the systems and processes necessary to fully maximize the usefulness of Big Data is currently lacking in most organizations. This lack of a conducive habitat for Big Data is further magnified in new organizations without any knowledge of Big Data. For organizations that have that have little to no knowledge of Big Data, there must be a thorough assessment of the benefits of big data and how they could improve the organizations overall place in the market. There also needs to be steps taken towards the design of frameworks that will enable the organization to better capture and utilize Big Data.

Using big data technologies to create an active archive use technology to create a large data archive information from active enterprise data warehouse. Make sure that the solutions for large Hadoop-based data provide an ideal platform for the construction of an active archive historical data from the data warehouse. This architecture, tools and methods that can be used to develop an active archive.

With the growth of technology, there is a substantial growth of data by volume, variety and velocity satisfying the criteria of Big Data . Volume of data already exceeded 100 EB at the end of 1990s, reached 1.8 ZB at 2011, and we have already entered in the age of ZB. By 2020, it was forecasted that the volume of data will be 50 times bigger than the one at 2011 . Big data is used to describe the huge data sets (terabyte to Exabyte) and Big Data analytics are the techniques applied on them.

Twitter added $47.5 million were to its coffers in 2013. This was not advertising revenue, but income generated by selling user data to businesses hungry to find new consumers. By researching target consumer buying habits, businesses can find more consumers and invariably increase profits. Because of this, big data is a highly sought after business asset; however, mining that data is a complex task.

As a result of the appearance of big data in our world, conventional data warehousing and data analysis methods no longer have the process power needed. What is Big Data you may ask and why is it such a big deal. NIST defines big data as anywhere “[…] data volume, acquisition velocity, or data representation limits the ability to perform effective analysis using traditional relational approaches […]” (Mell & Cooper, n.d.).

Additionally, social networking website Facebook, stores approximately 40 billion photos in total. (“Data, data everywhere”, 2010) Besides enormous data that generated from daily operational company transactions and social networks, the price drop of the data storage is also a strong factor triggering the fever of “Big Data”. For example, Google Drive - a cloud based data storage service – had a price drop of approximately 80% from March 2014. This price drop is considered a marketing approach to attract more computer users to adopt Google’s cloud service, which provides a more convenient and efficient way to access and store daily-used files. Although emerge of enormous data provides us opportunities to conduct further investigation and benchmarking, valuable information are not fully extracted and the potential power of using “Big Data” is undermined. In order to achieve thoroughly extraction of useful information from databases, many professionals in the academic field devoted into the study of data analysis and identified two of the most important drawbacks of traditional data analysis, which lacks of predictability and is less flexible in scalability.

Volume is often regarded as the primary attribute of big data. With that in mind, a large number of people define big data in terabytes—sometimes petabytes, but big data can also be quantified by counting records, transactions, tables, or files (Russom, 2011). Volume refers to the mass quantities of data that organizations are trying to harness to improve decision-making across the enterprise (Schroeck et al., 2012). The volumes of data have continued to increase at an unprecedented rate over the last couple of years. The sheer volume of data that is stored or available for storage today is exploding, it is expected that by the year 2020 40 zetabytes (ZB) of data will be stored (Zikopoulos et al. 2012) which

In recent years, there has been an increasing emphasis on big data, business analytics, and “smart” living and work environments. Though these conversations are predominantly practice driven, organizations are exploring how large-volume data can usefully be deployed to create and capture value for individuals, businesses, communities, and governments (McKinsey Global Institute, 2011). Big data refers to data volumes in the range of exabytes (1018) and beyond. Such volumes exceed the capacity of current on-line storage systems and processing systems. Data, information, and knowledge are being created and collected at a rate that is rapidly approaching the exabyte/year range. But, its creation and aggregation are accelerating and will

Presence of big data is a very common phenomenon now days, specially when talking about medium to large size corporation. Manyika et al., in their article (James Manyika, 2011) defined the term big data as “large pools of data that can be captured, communicated, aggregated, stored, and analyzed”. To clarify they suggested that big data refers to data, whose size makes it impossible to be processed by the typical software used for database management. Gartner (Gartner, 2012)defined big data in terms of its characteristics of high volume, high velocity and high variety. By volume, he referred to the size of the data, by velocity he referred to the speed at which the data is created and by variety he referred to the range of types of data.

Data has always been analyzed within companies and used to help benefit the future of businesses. However, the evolution of how the data stored, combined, analyzed and used to predict the pattern and tendencies of consumers has evolved as technology has seen numerous advancements throughout the past century. In the 1900s databases began as “computer hard disks” and in 1965, after many other discoveries including voice recognition, “the US Government plans the world’s first data center to store 742 million tax returns and 175 million sets of fingerprints on magnetic tape.” The evolution of data and how it evolved into forming large databases continues in 1991 when the internet began to pop up and “digital storage became more cost effective than paper. And with the constant increase of the data supplied digitally, Hadoop was created in 2005 and from that point forward there was “14.7 Exabytes of new information are produced this year" and this number is rapidly increasing with a lot of mobile devices the people in our society have today (Marr). The evolution of the internet and then the expansion of the number of mobile devices society has access to today led data to evolve and companies now need large central Database management systems in order to run an efficient and a successful business.

This is where NoSQL systems have been created to solve the data management challenges posed by Big Data. NoSQL is not a single system that can solve every single Big Data problem

This is where NoSQL systems have been created to solve the data management challenges posed by Big Data. NoSQL is not a single system that can solve every single Big Data problem

Big data is a popular term used to describe the improvement and availability of data in both structured and unstructured formats. Structure data is located in a fixed field within a record or file and it is present in the relational data bases and spreadsheets whereas an unstructured data file includes text and multimedia contents. The primary objective of this big data concept is to describe the extreme volume of data sets i.e. both structured and unstructured. It is further defined with three “V” dimensions namely Volume, Velocity and Variety, and two more “V” also added Value and Veracity. Volume refers to the amount of data, Velocity depends upon the speed of the data processing, Variety is described with the types of the

The modern RDBMS advancements are not capable of supporting unstructured information with ideal space necessity. The plan winds up plainly mind-boggling and is henceforth troublesome for designers. The requirement for unstructured information administration is so annoying with conventional RDBMS arrangements (Big data in financial services industry: Market trends, challenges, and prospects 2013 - 2018). Moreover, RDBMS turns out to be an exorbitant answer for creating light-footed web applications with direct information investigation necessities. NoSQL is developing as a proficient possibility in this situation, which connects the issues related with RDBMS innovation. The market development can credit to creative dispatches of NoSQL arrangements, and collective endeavors by NoSQL sellers and clients. The endeavors of organizations, to enhance their market offerings, are creating the request of NoSQL, as a back-end bolster (Big data in financial services industry: Market trends, challenges, and prospects 2013 - 2018). The emergence of agile software development is creating the demand for NoSQL (Big data in financial services industry: Market trends, challenges, and prospects 2013 - 2018). They offer users much more avenues to accept data in many different forms. NoSQL is adaptable as SQL but offers many more uses that can apply to many organizations.

The data is too large, moves too fast or does not meet the constraints of the database