Senior Databricks Data Modeller

@vishal1145

Your Medallion architecture will fail performance SLAs if the Silver-to-Gold transformations aren't partitioned correctly - I've seen pipelines grind to a halt when Type 2 SCD logic scans full tables instead of using Z-ordering on surrogate keys. That bottleneck alone can push processing windows from 2 hours to 12. Before I map out the modelling approach, two quick questions. First - what's your current data volume at the Gold layer, and are you seeing any specific query patterns that timeout? Second - does your Unity Catalog setup enforce column-level masking, or are we working with table-level permissions only? This affects how I'll structure the dimensional hierarchies. Here's the architectural approach: - STAR & SNOWFLAKE SCHEMAS: Design fact tables with composite partition keys (date + region) and dimension tables with SCD Type 2 temporal columns, all documented in dbt-style YAML under version control so your India team can maintain it post-handoff. - PYSPARK + DELTA LAKE: Build merge operations using Delta's MERGE INTO with predicate pushdown, reducing full-table scans by 70% through partition pruning and Z-order indexing on high-cardinality join keys. - SCD TYPE 1 & 2: Implement CDC pipelines that flag row-level changes in Silver, then apply Type 1 overwrites for corrections and Type 2 inserts with effective-date ranges in Gold - I'll include idempotency checks so reruns don't duplicate history records. - UNITY CATALOG GOVERNANCE: Tag PII columns at schema registration, set up lineage tracking through notebook metadata, and configure dynamic views that auto-mask sensitive fields based on user groups. - PERFORMANCE BENCHMARKING: Deliver before/after Spark UI metrics showing shuffle reduction, partition skew elimination, and query runtime improvements - typically I achieve 3-5x throughput gains on large fact table joins. I've led four Databricks migrations in the past 18 months, including one for a fintech client processing 2TB daily through a three-layer Medallion setup. I don't take six-week sprints unless the requirements are locked and the infrastructure is provisioned. Let's schedule a 20-minute technical call to walk through your current Bronze schema and confirm there aren't any upstream data-quality issues that'll block the Silver transformations.

@uniqedeveloper

I’ve carefully reviewed your requirement and I totally understand—you need someone who can step directly into an active Databricks environment, strengthen the Medallion architecture, and deliver production-grade data models with optimized Delta Lake performance. Recently, I worked on a large-scale Databricks implementation involving Delta Lake, Unity Catalog, and PySpark ETL pipelines for structured and semi-structured datasets. I designed Star/Snowflake schemas, implemented SCD Type 1 & 2 pipelines, and optimized Silver-to-Gold transformations using partitioning, Z-ordering, and caching strategies, which significantly improved query throughput and processing time. I can contribute immediately across modelling, PySpark engineering, Delta optimization, and governance-aligned implementation while maintaining clean documentation and version control. A few quick questions: 1. Are pipelines currently orchestrated through Databricks Workflows, Airflow, or another tool? 2. Which business domains/data marts are highest priority for modelling? 3. Do you already have performance baselines to compare optimisation gains against? Let me know when you’re available to discuss this further I’d be happy to walk you through my approach or showcase examples relevant to this project. Looking forward to hearing from you! Best regards, Mulayam

@bhargavrp

Hello, I am a Big Data Engineer with 9 years of experience in PySpark, Python, SQL, Hadoop, AWS, and AWS Glue. I specialize in building scalable and production-grade data pipelines, ETL workflows, and optimizing big data processing solutions. I can help deliver efficient, reliable, and high-quality solutions based on your project requirements. Looking forward to working with you. Best Regards, Bhargav

@lengares

Dear Hiring Manager, I am a Senior Data Engineer with **6 years of hands-on Databricks experience**, and your project is a strong match for my core skill set. ## What I Bring - **Delta Lake & Unity Catalog**: Designed and managed production-grade Delta Lake environments with Unity Catalog governance, including fine-grained access controls and lineage tracking. - **Data Modelling**: Deep expertise in translating complex business requirements into optimised **Star and Snowflake schemas** for both OLAP and reporting workloads. - **Medallion Architecture**: Architected and refined **Bronze → Silver → Gold** pipelines handling large-scale structured and semi-structured datasets with high reliability. ## Key Deliverables I Will Provide ✅ Logical and physical data models — documented and version-controlled in Git ✅ PySpark notebooks and SQL scripts for Star/Snowflake table creation under Unity Catalog ✅ Proven SCD Type 1 & 2 routines integrated into Medallion layers ✅ Performance benchmark report with before/after throughput metrics ## Why Choose Me I have delivered similar Databricks data modelling projects end-to-end, from initial schema design to optimised production pipelines. I communicate proactively, deliver clean and well-documented code, and can onboard quickly with minimal hand-holding. I am happy to discuss the project scope in detail and share relevant work samples before we begin. Looking forward to working with you! Best regards, surekha lengare

@abhaykumar0007

Senior Data Engineer with experience in Python, PySpark, Advanced SQL, Data Warehousing, and Big Data technologies. Skilled in designing, building, and managing scalable ETL pipelines, data modeling, and warehouse architectures for large datasets. Hands-on experience with both batch and real-time streaming pipelines using Kafka. Worked extensively with DBT for data transformation and Airflow for workflow orchestration and scheduling. Strong expertise in developing reliable data pipelines, optimizing query performance, and handling structured and semi-structured data processing. Experienced in end-to-end data engineering workflows, pipeline automation, and delivering scalable data solutions for analytics and business intelligence needs.

@latifmd1

Hi, I am interested for this job and want to work with you. Please, visit my website www.vanecus.com. If you are interested then we discuss about rate. Thanks and regards Md. Abdul Latif Dhaka, Bangladesh.

@pushpendras339

Hi, This project aligns very closely with my recent experience building large-scale data platforms and production data pipelines at Uber and JPMorgan. Over the past several years, I’ve worked extensively on Spark-based batch and real-time systems, data lake architectures, large-scale analytics pipelines, schema design, and performance optimization for high-volume datasets. My experience includes building scalable ingestion and transformation systems using technologies such as Spark, Kafka, Hive, Flink, Hudi, and distributed cloud infrastructure. I’m particularly comfortable with: * Medallion-style data architectures * Star and Snowflake schema design * SCD Type 1 & Type 2 implementations * PySpark-based transformation pipelines * Delta Lake style storage patterns and optimization * Performance tuning for large structured and semi-structured datasets I also place strong emphasis on maintainability, documentation, and designing systems that downstream teams can reliably operate and extend. Given my current commitments, I’d be best aligned contributing as a senior data engineering lead/consultant with strong ownership around modelling, architecture, optimization, and implementation guidance. Happy to discuss the current implementation state, expected scale, and how I can help accelerate delivery. Best, Pushpendra