Data Migration

Data Migration from Teradata to Snowflake

A Direct Banking and Payment Services company needed to migrate their on-premise data warehouse residing on Teradata to cloud-based data warehouse Snowflake. The process involved migrating tables, DDL including historical and incremental data with support for cut over to on premise.

Client Challenges and Requirements

  • Unavailability of any tool for migrating Teradata DDL to Snowflake DDL with constraints
  • Time consuming manual process for migrating large number of tables and high volume of data
  • Manual validation of migrated data resulting in inaccuracy and inefficiency of the process
  • Challenges to migrate incremental data due to lack of CDC feature in Teradata
  • Migration of sensitive data with Security
  • No provision to handle pipeline failures and restartability

Bitwise Solution

Bitwise leveraged its data migration framework consisting of the following features:

Schema Mapping and Auto conversion: Mapped the schema of Teradata with Snowflake and auto convert the DDL

Data Validation: Developed a process/service to validate the historical as well as CDC data with provision of sample and full data after migration.

High Performance, Availability and Scalability: Implemented containerization to execute jobs for extraction, load and Validation

Auto Scheduling: Automatic scheduling of migration process for data extraction, load and validation

Tokenization: Implemented tokenization process to handle sensitive data

Monitoring, Logging and Reporting: Ability to monitor the data migration process to check the progress/status with user friendly dashboards and reports.

Tools & Technologies We Used

Teradata
Snowflake
Bitwise Cloud Data Migration Framework Using Java, Apache Spark
Containerization using Docker and Kubernetes

Key Results

Fast processing of data migration and validation

100% accuracy of migrated data with sample and full validation

High scalability and availability with reduced manual intervention

Download Case Study

Ready to start a conversation?