
Closed
Posted
Paid on delivery
Duration : 6 Months Shift : US shift Job Summary: We are seeking a skilled Data Engineer with 5+ years of experience in designing, developing, and implementing data migration and archival solutions. The ideal candidate will have expertise in Azure cloud technologies, ETL/ELT development, data validation, and enterprise data migration projects involving ERP and HCM systems. This role will focus on building scalable data pipelines, ensuring data quality, and supporting large-scale migration and archival initiatives. Technologies: Azure Data Lake, Azure Data Bricks, Azure Data Factory, Azure DevOps, Gitlab, Github, & Snow Key Responsibilities: • Design and implement ETL processes to move data from various legacy systems to Azure data platforms. • Design and develop data orchestration workflows using Azure Data Factory (ADF). • Build data transformation and consolidation pipelines using Azure Databricks. • Use Azure Data Factory or Data Bricks orchestration tools to automate and schedule data pipelines/Job workflows • Design efficient data models (e.g., star or snowflake schema) for use in analytical applications and reporting systems. • Contribute to solution design discussions, code reviews, and implementation of migration best practices. • Continuously monitor the health of pipelines, jobs and infrastructure, ensuring they are running efficiently and securely • Address and resolve data issues, including failures in ETL pipelines, system performance problems, and data inconsistencies • Keep logs of pipeline performance, failures, and system updates to provide visibility for monitoring and troubleshooting • Work independently to deliver assigned migration workstreams while coordinating effectively with cross-functional teams. • Monitor migration execution, analyze results, and recommend improvements to increase reliability, scalability, and operational efficiency. • Ensure that all data engineering processes comply with industry security standards and best practices, including encryption, access controls, and data masking • Collaborate with legacy system owners, business stakeholders, application teams, and architects to understand migration requirements and define technical solutions. Qualifications: • Bachelor's degree in B.E in Computer Science, Data Management, or a related field. • Proficiency in coding skills such as Python, Pyspark & Azure SQL • Strong understanding of data management and automation. • Excellent analytical, troubleshooting and problem-solving skills. • Years of Experience – 5-8 years in a similar role.
Project ID: 40640365
48 proposals
Remote project
Active 1 day ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs
48 freelancers are bidding on average ₹52,372 INR for this job

Migration and archival across ERP and HCM sources is mostly a data-quality problem wearing a pipeline costume. The ADF work is tractable; proving the migrated data is correct is where projects overrun. How I approach it: - Reconciliation designed alongside the pipeline, not after. Row counts, control totals, and business-key matching between source and target for every load — so "the migration completed" and "the migration is correct" are two separate, evidenced statements. - Incremental and restartable pipelines. A six-month migration will fail mid-run at some point, and it must resume rather than restart. - Archival with retention and retrieval actually tested — an archive nobody has ever read back is not an archive. - Data quality rules explicit and logged, so rejected records are visible rather than silently dropped. Honest note on my side: my orchestration depth is Airflow, running 24/7 pipelines for industrial clients, plus strong Python, SQL and cloud. ADF I work with as a pipeline tool rather than having five years exclusively in it — the concepts transfer directly, but I would rather state that than have you assume otherwise. Questions: which source ERP and HCM systems, and roughly what data volume? US shift is workable for me from Germany for the overlap hours. Martin
₹40,000 INR in 10 days
6.0
6.0

With over 5+ years of experience, I've developed a deep understanding and proficiency in the necessary tools and techniques for effective data migration and archival solutions. I am adept at using Azure Data Lake, Data Factory, and Data Bricks among other tools you listed. Additionally, given my comprehensive knowledge of GITLab, GitHub and Snowflake, I am fluent in implementing version control and managing large-scale datasets. I have a knack for designing efficient data models which will be key to building analytical applications and reporting systems that'll meet your specific needs. My ability to troubleshoot, problem-solve, and resolve data inconsistencies will be invaluable as we collaborate to ensure seamless data migration and reduce system performance problems. Furthermore, as an Azure specialist, I meet all the essential qualifications listed for this project. My educational background in Computer Science is accompanied by 20+ years of relevant professional experience; underlining not only my technical skills but also my ability to adapt and embrace new technologies quickly. My extensive experience in Python, Pyspark, PHP-based development adds further value. Together with my focus on reliability and long-term scalability in project solutions, I believe I would be an excellent fit for your Azure ADB/ADF Migration Project. Let's connect soon to discuss your project!
₹45,000 INR in 15 days
5.7
5.7

Your incremental load strategy will determine whether you experience zero downtime or hours of production freeze during cut-over. If your source database sees heavy writes, you'll need change data capture or timestamp-based watermarking to avoid missing transactions between sync windows. Quick questions - what's your acceptable RPO during the final cut-over window? And are you planning Azure SQL MI, Synapse dedicated pools, or serverless for the target? Here's the architectural approach: - AZURE DATA FACTORY: Build parameterized copy pipelines with dynamic schema mapping, error handling to dead-letter queues, and Azure Monitor integration for real-time alerting on pipeline failures. - DATABRICKS + DELTA LAKE: Implement medallion architecture (bronze/silver/gold layers) using PySpark for schema validation, deduplication logic, and SCD Type 2 handling if dimension tables require history tracking. - TERRAFORM IaC: Provision ADF linked services, Databricks workspace, Key Vault secrets, and RBAC policies as versioned modules so you can spin up UAT and prod environments identically without manual clicks. I've migrated 4 enterprise databases into Azure with zero data loss and sub-hour cut-over windows using CDC patterns. Let's schedule a 15-minute call to align on your RPO requirements and finalize the incremental load design.
₹50,630 INR in 21 days
5.4
5.4

With over 18 years of experience in the industry, specializing in Azure, Cloud Computing, and Python, my team and I at CnELIndia can deliver the results you need for your Azure ADB/ADF Migration project. Our proficiency extends to the core technologies you've mentioned: Azure Data Lake, Azure Data Bricks, Azure Data Factory, as well as Azure DevOps. As a dedicated group, we focus largely on automation and data management - two key aspects in ensuring a smooth transition during migration. In terms of skills, our extensive knowledge of Python would come in handy when dealing with your data transformation and consolidation pipelines. Additionally, given that this is a 6-month job running on a US shift, our schedule flexibility (another legacy system adeptness) will be beneficial and allow for easy coordination with cross-functional team members. Let's make this an even more successful endeavor by leveraging my+
₹56,250 INR in 7 days
5.3
5.3

HI, KINDLY READ THROUGH MY PROPOSAL I will design, develop, and implement scalable Azure-based data migration and archival solutions for your ERP/HCM systems, delivering clean ETL/ELT pipelines, robust data validation, and production-ready orchestration with 5+ years of hands-on experience. MY APPROACH ✅ Phase 1: Analyze legacy systems, map data sources, and design efficient star/snowflake models plus migration strategy. ✅ Phase 2: Build and orchestrate ETL pipelines in Azure Data Factory, transform data in Databricks, and load into Azure Data Lake. ✅ Phase 3: Validate data quality, automate monitoring, resolve issues, and hand over fully documented, secure pipelines. DELIVERABLES - Production-ready ADF + Databricks pipelines - Data models, validation reports, and monitoring setup - Full documentation and knowledge transfer QUESTIONS 1. Can you share details on the source systems, data volumes, and current migration scope? 2. Preferred Azure environment setup and any existing ADF/Databricks assets? 3. Specific security, compliance, or archival retention requirements? Ready to start immediately. I can deliver the first workstream quickly after receiving access and requirements.
₹40,000 INR in 5 days
5.6
5.6

Hi, I’m Sahanaj. I’ve handled enterprise data migration and archival programs where Azure pipelines had to be scalable, auditable and—most importantly—accurate. My hands-on stack includes ADF, Azure Databricks, PySpark, Python, Azure Data Lake, Azure SQL, Azure DevOps, Git/GitHub/GitLab, plus ETL/ELT, data validation, reconciliation and performance tuning. I’ve worked across complex legacy-to-Azure migrations and understand the realities of ERP/HCM data. For this 6-month US-shift engagement, I can commit consistently. A realistic rate is ₹125,000/month for the expected capacity. I’m ready to start immediately and can take ownership of migration workstreams independently.
₹125,000 INR in 20 days
4.9
4.9

Hi, I checked your "Azure ADB/ADF Migration Project" project description, it looks like the focus is on delivering a clean, responsive website that works well across all devices. I prefer understanding the expected layout and user experience first, then building pages that closely match the design while keeping the code organized and easy to maintain. Feel free to share the design or current website, and I'll suggest the best implementation along with a realistic timeline. Final timeline and cost will be confirmed in chat after a complete understanding and documentation of the project expectations in detail.
₹37,500 INR in 16 days
4.2
4.2

Hi, Your Azure database migration requires more than simply copying tables—I’d build a controlled, repeatable migration pipeline that preserves relational integrity while minimizing production downtime. I’d use ADF for orchestration and incremental movement, with Databricks handling transformation, validation and post-load reconciliation. ?? Project Key Points: Source schema/dependency analysis, Azure Data Factory pipelines, incremental/CDC-style loading, Databricks cleansing and validation, row-level reconciliation, robust logging/alerting, Key Vault-secured credentials, performance benchmarking and production cutover planning. Execution Plan: I’ll first map the source schema, relationships, dependencies and data volumes, then design the target architecture and migration strategy. ADF pipelines will be parameterized and rerunnable, while Databricks notebooks will handle schema evolution, cleansing and automated source-to-target validation. With 5+ years in backend, cloud and data engineering, I specialize in Azure-based data platforms, ETL/ELT pipelines, Python automation, database migration and production reliability. Excited to build a migration process that is accurate, repeatable and production-safe. Best regards, Prateek
₹100,000 INR in 7 days
4.0
4.0

As a seasoned data engineer for over 5 years, I am well-versed in the technology stack your project demands. This includes my mastery of Azure Data Lake, Azure Data Factory, and Azure Databricks, which are key to successfully accomplishing your objective of migrating data within ERP and HCM systems. Having handled similar assignments in the past, I'm comfortable using Python and Pyspark for your ETL processes while maintaining top-notch data quality. Moreover, my extensive skills in building efficient data models such as star or snowflake schemas will contribute significantly to your analytical applications and reporting systems, ensuring you obtain actionable insights from the migrated data. I understand the importance of streamlining complex migration workflows and will efficiently manage all aspects of your large-scale migration and archival initiatives. My approach to this project emphasizes not just meeting the technical requirements but aligning them with industry security standards and best practices. Ensuring data consistency and system performance is a priority for me. That's why I have developed effective monitoring protocols that track pipeline performance, failures, and system updates which provide deep visibility for monitoring and troubleshooting. Executing a successful migration often requires effective cross-functional collaboration.
₹56,250 INR in 7 days
4.1
4.1

Hi, I can support your Azure ADB/ADF migration project as a Data Engineer for ETL/ELT pipelines, data migration, archival workflows, validation, monitoring, and performance optimization. The best solution is to first understand the legacy ERP/HCM data sources, target Azure architecture, data lake structure, migration rules, validation checks, security requirements, and scheduling needs. Then I’ll build scalable ADF orchestration workflows, Databricks/PySpark transformations, Azure SQL models, automated job monitoring, error handling, and reconciliation reports. I’m comfortable with Azure Data Factory, Azure Databricks, Azure Data Lake, PySpark, Python, Azure SQL, ETL/ELT migration, data validation, star/snowflake schema design, Git/GitHub/GitLab, Azure DevOps, pipeline troubleshooting, performance tuning, access control, encryption, masking, and production support. Deliverables will include: * ADF pipeline development * Databricks/PySpark transformations * Legacy-to-Azure migration workflows * Data validation and reconciliation * Scheduling and automation * Pipeline monitoring/logging * Error handling and troubleshooting * Azure SQL/data model support * Performance tuning * Git/Azure DevOps support * Migration documentation I’ll focus on clean, reliable, and scalable migration pipelines with strong data quality checks, secure handling, and smooth coordination during the 6-month US-shift engagement. Best regards Ankit
₹37,500 INR in 7 days
3.5
3.5

Hi, I specialize in turning business challenges into clean, reliable software — from automation and APIs to web applications, AI, and backend systems. What sets me apart: I focus on outcomes, not just deliverables. Every solution I build is designed to be maintainable, scalable, and aligned with your actual goals. Let's build something great together. Thanks, Deva
₹56,250 INR in 7 days
3.1
3.1

Hi, I’m interested in your Azure ADB/ADF Migration Project. I have 5+ years of experience in software development with hands-on experience in Azure, .NET, SQL Server, APIs, Azure Key Vault, and cloud-based solutions. I can help with end-to-end database migration using Azure Data Factory and Azure Databricks, including: • Source schema and dependency analysis • ADF pipelines for reliable data extraction and loading • Incremental/delta loads for minimal cut-over downtime • Databricks-based data transformation, cleansing, and validation • Row-level data reconciliation and post-load verification • Error handling, logging, monitoring, and rerunnable pipelines • Azure Key Vault integration for secure credentials • Performance tuning and optimization • ARM/Bicep or Terraform templates • Clear migration documentation and runbooks I focus on building reliable, scalable, secure, and maintainable Azure solutions while ensuring data accuracy and minimal production downtime. I’m available to discuss your current architecture and migration requirements. Best regards, Krishna Bhagwat
₹37,500 INR in 2 days
1.8
1.8

Having been in the IT industry for over 9 years, specifically in web and mobile development, I believe I can bring a unique and valuable perspective to your Azure ADB/ADF Migration Project. Although my experience has not focused exclusively on data migration, I have extensively worked with Python, PHP and MySQL. This experience aids me in creating efficient solutions that ensure data accuracy and offer scalability, making me highly compatible with this position. Moreover, my cross-functional expertise brings a wide range of benefits to the table. I can seamlessly coordinate with different teams and understand complex requirements. Given my exposure to various tools such as Gitlab, Github, Azure DevOps etc; I'm aware of what it takes to execute large-scale projects while adhering to schedule and quality standards. Additionally, my website development background provides me with significant HTML/HTML5/CSS experience that will be useful in understanding your system better as this is code you may have used in developing the current ERP and HCM system. All of these diverse skills combined would allow me to deliver flawlessly on responsibilities like designing ETL processes; monitoring their performance effectively; resolving any issues that might arise; and ensuring data integrity during the migration process. Overall, my technical acumen, broad-ranging exposure spanning across multiple technologies like Python and familiarity with GitLab and Github unproject
₹56,250 INR in 7 days
2.0
2.0

With a robust five to eight years of professional experience in the exacting world of Data Engineering, I am the precise fit for your Azure ADB/ADF Migration Project. I have deftly dealt with extensive data migration and archival initiatives involving ERP and HCM systems, as well as mastered the likes of Azure Data Lake, Azure Data Factory, Azure DevOps, Gitlab, Github, & Snow - all technologies which are key to your project.+ I bring a holistic approach to my work processes – right from designing and implementing efficient ETL processes to ensuring data quality, a "build it perfectly" mindset comes natural to me. Extensive experience in designing data models for analytical applications and reporting systems complements this approach as I can help design the most efficient schema for your specific purposes. It won't just be about producing great results on your project; I believe in iterative learning as well. To bring you the most effective solutions, I keep myself updated with industry standards and strive to implement best practices for migration throughout every task I undertake. I can work independently and yet positively thrive in cross-functional teams – testimonies of which you can find in my verifiable references. Considering all these qualities alongside a keen eye on compliance and a strong problem-solving disposition, hiring me will mean signing up for reliable support that will elevate your project from good to exceptional.
₹37,500 INR in 7 days
1.4
1.4

Hi, I can set up your Amazon affiliate WordPress site using the licensed ReHub theme and deliver a fully functional, mobile-friendly catalogue with automated product importing. I can handle: * ReHub installation and configuration, including comparison and deal modules * Amazon Associates API integration using your tracking ID * Content Egg, AAWP, or ASA2 setup if that is more suitable * Import and formatting of 1000+ products with titles, descriptions, prices, and images * Affiliate buttons with correct tracking tags and new-tab behavior * Product categories, search, filters, and WooCommerce-style catalogue pages * Mobile optimization and performance improvements * SEO-ready permalinks, meta fields, and structured data * Testing across desktop/mobile and verification of affiliate links * A clear admin guide covering product imports and ReHub settings I’ll make sure the imported products are structured consistently and that the affiliate workflow is easy to maintain rather than requiring manual updates for every product. I’m comfortable working directly on your existing WordPress installation and can troubleshoot API/import issues as part of the setup. Once the 1000+ products, ReHub layouts, affiliate links, and documentation are verified on your server, the project will be ready for handover. Ready to start with your WordPress access, ReHub license, Amazon Associates details, and product list.
₹56,250 INR in 7 days
0.3
0.3

Hi, I can handle your end-to-end Azure database migration into a new Azure landing zone using Azure Data Factory, Azure Databricks, Key Vault, and Azure SQL/Managed Instance or Synapse. I have experience with Azure data engineering, ETL/ELT pipelines, SQL optimization, and production database migrations, and I can start immediately. Best regards, Sushil
₹39,250 INR in 7 days
0.0
0.0

★★★★★★★★★★★★★★★ I can work full time for you. ★★★★★★★★★★★★★★★ I understand you need an ADF-led migration into a new Azure landing zone. I will map schemas, dependencies, incremental loads, and cutover controls. I use ADF, Databricks, Python, Scala, Key Vault, and Azure SQL. I will add logging, alerts, reruns, validation, and row-level QA. Terraform or Bicep will make the platform repeatable across environments. I will deliver runbooks, benchmarks, tuning, and deployment docs. Thank you for hiring me.
₹50,000 INR in 10 days
0.0
0.0

MIGRATION SUCCESS RELIES ON STRONG DATA PIPELINE DESIGN I'd love to help with your Azure ADB/ADF migration project. With over five years of experience in data engineering, I've successfully managed large-scale data migrations and implemented robust ETL processes in Azure environments. What specific challenges do you anticipate with the current legacy systems? Are there particular compliance standards you want to prioritize during the migration? The worst-case scenario of reaching out is a free consultation to discuss your needs further. Regards, Johan
₹37,500 INR in 7 days
0.0
0.0

Hello, I’m excited about the opportunity to assist with your Azure ADB/ADF Migration Project. Your need for a Data Engineer skilled in Azure cloud technologies and ETL/ELT development aligns perfectly with my capabilities. I understand the importance of designing and implementing efficient ETL processes to transition data from legacy systems to Azure platforms. My approach will focus on building robust data pipelines using Azure Data Factory and Databricks, ensuring data quality and seamless orchestration. Throughout the six-month duration, I will prioritize clear communication and collaboration with cross-functional teams, while continuously monitoring pipeline health and performance. My commitment to adhering to industry security standards will ensure that all processes are secure and compliant. For examples of my previous work, please refer to my portfolio: [insert portfolio link]. Let’s discuss how I can contribute to your project’s success. Best, Ryan ryang088
₹37,500 INR in 7 days
0.0
0.0

I have over five years of experience building ETL pipelines with Azure Data Factory and Databricks. I have designed star and snowflake schemas for analytical applications and reporting systems. My stack includes Python PySpark Azure SQL Data Lake and Snowflake for enterprise migrations. I have handled ERP and HCM data migrations with full validation security and access controls. I can monitor pipeline health troubleshoot failures and optimize performance independently. I am comfortable with Azure DevOps GitLab and GitHub for CI CD and version control workflows. I can work US shift hours and collaborate with cross functional teams on migration requirements.
₹56,250 INR in 7 days
0.0
0.0

Bengaluru, India
Member since Aug 11, 2026
₹12500-37500 INR
$3000-5000 USD
$30-250 USD
₹250000-500000 INR
$30-250 USD
₹1500-12500 INR
$110-150 USD
₹1500-12500 INR
$2-8 USD / hour
$250-750 USD
£250-750 GBP
₹750-1250 INR / hour
₹12500-37500 INR
₹37500-75000 INR
$10-30 USD
$8-15 USD / hour
₹1500-12500 INR
$30-90 SGD / hour
₹1500-12500 INR
₹37500-75000 INR