Data Migration

Data Migration from On-Premise to GCP Cloud

A major payment processing firm based out of the United States wished to build a cloud-based big data strategy that the enterprise, as well as external users, could leverage to build better analytics and make better business decisions. One of the key components and challenges in the build process was migration of on premise data to the cloud.

Client Challenges and Requirements

  • Migration of 15+ TB of uncompressed data between geographically separated locations
  • Maintaining on-premise cluster performance so as to have a balance between data migration load and current user workload
  • On-premise data in different file formats and compressions
  • Storage, data type and other compatibility changes between on premise and Cloud
  • Keeping migrated historical data in sync with on-premise cluster

Bitwise Solution

Schema conversion, data extraction, ingestion and data load through GCP Migration agent

Bitwise Cloud Data Migration Framework for Trigger based data load to ingest historical data once available in cloud storage bucket

In-built DQ, validation checks and Bitwise Cloud Data Migration Framework’s full and sample validation utilities to validate migrated historical data to ensure accuracy

Implemented throttling and elasticity for optimal load on source database and maximum extraction performance through Bitwise Cloud Data Migration Framework

CDC implementation through GCP Migration Agent to keep migrated historical data in sync with on premise cluster

Dedicated 1gbps network connection between on premise Hadoop cluster and Google cloud to avoid network congestion

Tools & Technologies We Used

Unix
Google BigQuery Data Transfer Service
Shell Script
Gcloud SDK
BigQuery
Bitwise Cloud Data Migration Framework

Key Results

Successful migration of historical data ensuring data integrity

Automated Data validation resulted in 30% reduction in migration time

Ease of scalability and enhancements

Download Case Study

    Ready to start a conversation?