The Challenges of Big Data and Extreme Workload

2964 WordsFeb 19, 201812 Pages
The Challenges of Big Data and Extreme Workload: Big Data has gained massive importance in IT and Business today. A report recently published state that use of big data by a retailer could increase its operating margin by more than 60 percent and it also states that US health care sector could make more than $300 billion profit with the use of big data. There are many other sectors that could profit largely by proper analysis and usage of big data. Although big data promises better margin’s, more revenue and improvised operations it also brings new challenges to the It infrastructure which is “extreme data management” .At the same time these companies should also need to look at workload automation and make sure it is robust enough to make to handle the needs that the big data is associated to as well as the needs of the business intelligence it there to serve. File transfers require to be scheduled for data to be moved to a central database or data-warehouse. This by far involves an Extract, Transform and Load (ETL) workflow as data is usually gathered from different types of database. Once the data is brought to a central database, to determine patterns in the data queries are scheduled from a variety of users using various applications. The frequency of the queries varies from business to business – it can be continuous, once a day or hourly. And of course, as data gets added to the database and moved to new databases, there is the routine task of database
Open Document