Automated Informatica ETL Migration to Azure Data Factory and Snowflake EDW

case-study-feature-img

A national used vehicle retailer needed to migrate its on-premise legacy Enterprise Data Warehouse built on Informatica and Teradata to the cloud in Azure Data Factory and Snowflake to leverage the advanced capabilities of the Azure ecosystem to enhance data integration, workflow orchestration and overall data management.

Client Challenges and Requirements

  • Migrate Informatica PowerCenter to ADF as part of program to modernize the Enterprise Data Warehouse (EDW)
  • Inventory to be migrated included 578 workflows, 3461 mappings, 801 Python, Java and VB Scripts (of which 611 needed to be re-engineered and 174 migrated as-is)
  • No Legacy System Knowledge / Documentation
  • Re-engineering of frameworks which were tightly coupled with Informatica to make them ADF compatible
  • Limited Azure awareness with stakeholders
  • Delays and changes to design decisions
  • Unavailability of adequate test data

Bitwise Solution

Bitwise provided automated ETL Migration services using its proprietary ETL Converter solution to accelerate project execution within defined cost constraints.

 

  • Conduct Assessment for detailed analysis of legacy system with Assessment Report to estimate efforts.
  • Since client had no documentation for the Informatica system, Bitwise used its Source ETL Analyzer tool throughout the project to reduce turnaround time and increase accuracy with analysis for source and target information, pattern findings, identifying writeback scenarios, linked service analysis, lineage and schedule information.
  • Used the Assessment reports to finalize high-level design / ingestion patterns leveraging cybersecurity best practices for handling sensitive / PII data and plan the Execution phase efficiently.
  • Developed reusable frameworks for replication, email notification, error and logging.
  • Integrated ADF with GitHub and Azure DevOps to build CI/CD pipeline to automate deployments from Dev to Prod.
  • Scaled elasticity with HA set up to meet customer’s RPO, RTO needs.
  • Provided standardized documentation through wiki while creating knowledge base and improved end-to-end ecosystem support and observability.
  • Introduced POD delivery structure to streamline governance during execution phase.

Tools & Technologies We Used

Bitwise Source ETL Analyzer
Bitwise ETL Converter
Informatica PowerCenter
Teradata
Azure Data Factory (ADF)
Snowflake
Python scripting
Bitwise Data Validation Utility

Key Results

Reduced 40-45% Assessment efforts using Bitwise Source ETL Analyzer for code analysis

Achieved up to 87% automation in converting Informatica mappings to ADF Dataflows, resulting in a 21% reduction in the overall effort for the migration

Achieved performance and cost optimization by re-engineering re-usable components, DF parallelism, Infra sizing, etc.

Delivered advanced Data Ingestion and replication capabilities with improved user experience, stability and performance

Download Case Study

    To get our latest updates subscribe to our Newsletter.

    Ready to start a conversation?