The IRMA Community
Newsletters
Research IRM
Click a keyword to search titles using our InfoSci-OnDemand powered search:
|
Big Data Management in the Context of Real-Time Data Warehousing
|
|
Author(s): M. Asif Naeem (Auckland University of Technology, New Zealand), Gillian Dobbie (The University of Auckland, New Zealand)and Gerald Weber (The University of Auckland, New Zealand)
Copyright: 2014
Pages: 27
Source title:
Big Data Management, Technologies, and Applications
Source Author(s)/Editor(s): Wen-Chen Hu (University of North Dakota, USA)and Naima Kaabouch (University of North Dakota, USA)
DOI: 10.4018/978-1-4666-4699-5.ch007
Purchase
|
Abstract
In order to make timely and effective decisions, businesses need the latest information from big data warehouse repositories. To keep these repositories up to date, real-time data integration is required. An important phase in real-time data integration is data transformation where a stream of updates, which is huge in volume and infinite, is joined with large disk-based master data. Stream processing is an important concept in Big Data, since large volumes of data are often best processed immediately. A well-known algorithm called Mesh Join (MESHJOIN) was proposed to process stream data with disk-based master data, which uses limited memory. MESHJOIN is a candidate for a resource-aware system setup. The problem that the authors consider in this chapter is that MESHJOIN is not very selective. In particular, the performance of the algorithm is always inversely proportional to the size of the master data table. As a consequence, the resource consumption is in some scenarios suboptimal. They present an algorithm called Cache Join (CACHEJOIN), which performs asymptotically at least as well as MESHJOIN but performs better in realistic scenarios, particularly if parts of the master data are used with different frequencies. In order to quantify the performance differences, the authors compare both algorithms with a synthetic dataset of a known skewed distribution as well as TPC-H and real-life datasets.
Related Content
|
Renjith V. Ravi, Mangesh M. Ghonge, P. Febina Beevi, Rafael Kunst.
© 2022.
24 pages.
|
|
Manimaran A., Chandramohan Dhasarathan, Arulkumar N., Naveen Kumar N..
© 2022.
20 pages.
|
|
Ram Singh, Rohit Bansal, Sachin Chauhan.
© 2022.
19 pages.
|
|
Subhodeep Mukherjee, Manish Mohan Baral, Venkataiah Chittipaka.
© 2022.
17 pages.
|
|
Vladimir Nikolaevich Kustov, Ekaterina Sergeevna Selanteva.
© 2022.
23 pages.
|
|
Krati Reja, Gaurav Choudhary, Shishir Kumar Shandilya, Durgesh M. Sharma, Ashish K. Sharma.
© 2022.
18 pages.
|
|
Nwosu Anthony Ugochukwu, S. B. Goyal.
© 2022.
23 pages.
|
|
|