Study of Methods to Achieve Near Real Time ETL

Real time analysis has become a norm in today's fast and competitive business environment. Availability of recent data is a requirement to perform real time analysis. The data warehouse is meant to store historical data for analysis and these are populated from time to time through a process called as extract transform and load (ETL). The ETL process was done during nights or weekends to make the transactional data available at the data warehouse for analysis. But the delay of a day or weekend is not acceptable for real time analysis hence a concept of near real time ETL is developed to perform real time analysis. We explore in this study methods of how near real time ETL could be achieved. We focus on three ways of how near real time ETL could be performed. First is the Meta data management which if done effectively will greatly reduce the development time. Second is the concept of change data capture when implemented with some intelligent would greatly help in making the correct data available for analysis at the warehouse. The third is the use of parallel techniques to partition the flow of ETL and reduce the time window with which the data are available at the warehouse.