29525540-0af3-44aa-9dd9-c813dd8adfbd20210319075735449wseamdt@crossref.orgMDT DepositWSEAS TRANSACTIONS ON SYSTEMS AND CONTROL1991-876310.37394/23203http://wseas.org/wseas/cms.action?id=4073220202022020201510.37394/23203.2020.15http://wseas.org/wseas/cms.action?id=23195Muddasir N.MohammedDept. of IS&E,VVCE, Mysuru, Karnataka, IndiaK.RaghuveerDept. of IS&E,NIE, Mysuru, Karnataka,IndiaR.DayanandTechnical Director, IndiaRefreshment anomalies occur in a data warehousing environment while performing Extract Transform and Load (ETL) to get the data for analysis from sources. There could be several reasons for the anomalies like not able to capture the delta on time, system time out, duplicate entries due to outer join operations and many more. Once anomalies are detected the compensation operation is executed to get the data that was missing into the data warehouse. In this work we would like to analyze scenario where it is necessary to perform incremental loads based on priority in an ongoing data warehouse maintenance work. The work proposes a novel approach to decide on when to perform ETL so that refreshment anomalies do not occur and to maintain integrity of data such that analytics queries always provide right information to the analyst. Two novelties have been discussed in this work one is to have a threshold before compensation of updates and two is while performing compensation updates prioritize the query with less freshness interval to have more time limits for the updates to be completed.12720201272020684690https://www.wseas.org/multimedia/journals/control/2020/b365103-025.pdf10.37394/23203.2020.15.68https://www.wseas.org/multimedia/journals/control/2020/b365103-025.pdf10.1007/978-3-319-22729-0_17Stefan Dessloch, Weiping Qu, Vinanthi Basavaraj, Sahana Shankar, “Real-Time Snapshot Maintenance with Incremental ETL Pipelines in Data Warehouses,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 9263, pp. 28–39, 2015, doi: 10.1007/978-3-319-22729-0. W. Qu, “Incremental ETL Pipeline Scheduling for Near Real-Time Data Warehouses 1,” no. Btw, pp. 299–308, 2017.10.1109/aina.2010.54. Chen, W. Rahayu, and D. Taniar, “Towards near real-time data warehousing,” Proc. - Int. Conf. Adv. Inf. Netw. Appl. AINA, pp. 1150–1157, 2010, doi: 10.1109/AINA.2010.54.10.1145/223784.223848Y. Zhuge, H. García-Molina, J. Hammer, and J. Widom, “View maintenance in a warehousing environment,” in SIGMOD ’95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data, 1995, pp. 316–327, doi: 10.1023/A:1008698814840. 10.1007/978-1-4757-6132-0_2Y. Zhuge, H. Garcia-Molina, and J. L. Wiener, “Consistency Algorithms for Multi-Source Warehouse View Maintenance,” Distrib. Parallel Databases, vol. 6, no. 1, pp. 7–40, 1998, doi: 10.1023/A:1008698814840. 10.1145/253262.253355D. Agrawal, A. El Abbadi, A. Singh, and T. Yurek, “Efficient view maintenance at data warehouses,” pp. 417–427, 1997, doi: 10.1145/253260.253355. T. W. Ling and E. K. Sze, “Materialized view maintenance using version numbers,” Proc. - 6th Int. Conf. Database Syst. Adv. Appl. DASFAA 1999, pp. 263–270, 1999, doi: 10.1109/DASFAA.1999.765760.10.1109/dante.1999.844943A. Vavouras, S. Gatziu, and K. R. Dittrich, “Modeling and executing the data warehouse refreshment process,” vol. 2000, no. January, pp. 66–73, 2003, doi: 10.1109/dante.1999.844943. 10.1007/bfb0014149A. G. ; I. S. H. V. J. Mumick, “Data Integration Using Self-Maintainable Views,” in Advances in Database Technology —EDBT ’96, 1997, pp. 9–19, doi: 10.14220/9783666565441.9. 10.1109/cscwd.2005.194333Z. Shu, S. Li, Y. Zuo, X. Zhou, and Y. Tang, “Correction strategy for view maintenance anomaly after schema and data updating concurrently,” Proc. 9th Int. Conf. Comput. Support. Coop. Work Des., vol. 2, no. 031542, pp. 1046–1051, 2005, doi: 10.1109/cscwd.2005.194333. 10.1145/1866480.1866511A. Behrend and T. Jörg, “Optimized incremental ETL jobs for maintaining data warehouses,” ACM Int. Conf. Proceeding Ser., pp. 216–224, 2010, doi: 10.1145/1866480.1866511. 10.1007/978-3-642-14559-9_7T. Jörg and S. Dessloch, “Near real-time data warehousing using state-of-the-art ETL tools,” Lect. Notes Bus. Inf. Process., vol. 41 LNBI, pp. 100–117, 2010, doi: 10.1007/978-3-642-14559-9_7. 10.1145/1451940.1451956T. Jörg and S. Deßloch, “Towards generating ETL processes for incremental loading,” ACM Int. Conf. Proceeding Ser., vol. 299, pp. 101–110, 2008, doi: 10.1145/1451940.1451956. T. Jorg and S. Dessloch, “Formalizing ETL Jobs for Incremental Loading of Data Warehouses,” Datenbanksysteme Business, Technol. Web, pp. 327–346, 2009. 10.1109/iccasm.2010.5620479X. Zhang, L. Yang, and D. Wang, “Incremental view maintenance based on data source compensation in data warehouses,” ICCASM 2010 - 2010 Int. Conf. Comput. Appl. Syst. Model. Proc., vol. 2, no. Iccasm, pp. V2-287-V2-291, 2010, doi: 10.1109/ICCASM.2010.5620479. 10.1109/compsac.2011.44R. J. Santos, J. Bernardino, and M. Vieira, “24/7 real-time data warehousing: A tool for continuous actionable knowledge,” Proc. - Int. Comput. Softw. Appl. Conf., pp. 279–288, 2011, doi: 10.1109/COMPSAC.2011.44. 10.1007/978-981-15-4032-5_50N. Mohammed Muddasir and K. Raghuveer, “A Novel Approach to Handle Huge Data for Refreshment Anomalies in Near Real-Time ETL Applications,” Advances in Intelligent Systems and Computing, vol. 1154. pp. 545–554, 2020, doi: 10.1007/978-981-15-4032-5_50. H. Homayouni, S. Ghosh, and I. Ray, Data Warehouse Testing, 1st ed., vol. 112. Elsevier Inc., 2019. M. M. Susanne Englert, “Transaction Processing Performance Council (TPC) www.tpc.org info@tpc.org Legal Notice,” 2018.