Data Warehouse Modernization
Migrating Legacy Ab Initio ETL to Hydrograph on AWS Cloud
As part of its organizational strategy to move critical processes to Cloud, a Fortune 500 Financial Services company required a flexible, cost effective solution for migrating its Human Resources (HR) ETL process from the on-premise enterprise data management environment to Amazon Web Services (AWS) Cloud.
Client Challenges and Requirements
- The customer required a process for quickly and efficiently converting Ab Initio graphs to an ETL tool that is compatible with the AWS Cloud platform.
- The target ETL tool needed to provide an easy-to-use development environment that required minimal retraining for developers, while delivering robust capabilities for matching the performance of Ab Initio.
Bitwise partnered with the customer’s team to implement Hydrograph, an open source ETL development tool designed for the big data ecosystem that provides a GUI environment with Drag-and-Drop functionality that developers are familiar with.
The solution was developed to run on AWS EMR (Spark) with Redshift as the target Data Warehouse.
In order to convert the Ab Initio graphs to Hydrograph on Spark, Bitwise used its ETL Converter solution which helps to accurately scope the conversion and automate up to 70% of the overall process, including data validation.
Tools & Technologies We Used
AWS EMR (Spark)
First group within customer organization to successfully implement a fully Cloud ETL process using Hydrograph in Production
Jobs running in Production with 100% accuracy of data loads
Customer committed to leveraging Hydrograph for migration of entire HR ETL process to AWS Cloud