WSEAS Transactions on Systems and Control
Print ISSN: 1991-8763, E-ISSN: 2224-2856
Volume 15, 2020
Towards Handling Incremental Load for Anomalies in Near Real Time Data Warehouse
Authors: , ,
Abstract: Refreshment anomalies occur in a data warehousing environment while performing Extract Transform and Load (ETL) to get the data for analysis from sources. There could be several reasons for the anomalies like not able to capture the delta on time, system time out, duplicate entries due to outer join operations and many more. Once anomalies are detected the compensation operation is executed to get the data that was missing into the data warehouse. In this work we would like to analyze scenario where it is necessary to perform incremental loads based on priority in an ongoing data warehouse maintenance work. The work proposes a novel approach to decide on when to perform ETL so that refreshment anomalies do not occur and to maintain integrity of data such that analytics queries always provide right information to the analyst. Two novelties have been discussed in this work one is to have a threshold before compensation of updates and two is while performing compensation updates prioritize the query with less freshness interval to have more time limits for the updates to be completed
Search Articles
Keywords: Data warehouse, Extract Transform Load, Incremental Load, Refreshment anomalies, query priority
Pages: 684-690
DOI: 10.37394/23203.2020.15.68