ETL Migration

Automated ETL Migration of DataStage to Azure Data Factory


A Fortune 500 multinational department store corporation needed to modernize its legacy ETL platform to a cloud-based solution using Azure as the preferred cloud service provider and Azure Data Factory (ADF) as the cloud-native ETL tool.

Client Challenges and Requirements

  • Upgrade the underlying technology stack to Azure cloud, converting 124 Parallel jobs and 22 Sequence jobs from DataStage to ADF within fixed time and cost deadlines.
  • Smooth migration of ETL jobs from DataStage to ADF, ensuring minimal disruption to client’s existing operations.
  • Migrate the ETLs in stages to minimize sudden operational impacts.
  • For a certain period, run both DataStage and ADF simultaneously to fix any abrupt issues and ensure a seamless transition.
  • Employee training to manage converted ADF pipelines.

Bitwise Solution

Bitwise used its migration services and proprietary automated ETL Conversion tool to expedite the timeline for migrating the DataStage ETLs to ADF while adhering to budgetary requirements.

  • Orchestrate the converted ADF pipelines using ADF native features. This was one of the significant highlights of the ETL migration project.
  • Develop generic functionality using Azure Functions for sending emails. This feature was crucial in achieving operational efficiency. Another generic functionality was created using Azure Functions for file validations. The objective was to streamline file validations and maintain stringent data quality checks.
  • Carry out performance optimization of long running jobs in ADF to enhance the efficiency of the system.
  • Conduct meticulous row-by-row and column-by-column data validation between DataStage and ADF output utilizing Bitwise’s Python based data validation accelerator to ensure the consistency and conformity of data.

Tools & Technologies We Used

Bitwise ETL Converter
IBM DataStage
SQL Server
Azure SQL DB
Azure Data Factory (ADF)
Azure Functions
Bitwise Data Validation Utility

Key Results

Improved Code Efficiency: Reduced 180+ UNIX commands/scripts to 15 Azure Functions

Optimized performance of long-running jobs

Accelerated Time-to-Production: Conversion completed in 12 weeks using automated ETL migration

Rigorous Data Validation: Up to 50% effort savings using Bitwise data validation accelerator

Download Case Study

    To get our latest updates subscribe to our Newsletter.

    Ready to start a conversation?