Apache Spark Jobs

We are seeking an experienced Data Engineer to build and maintain scalable, high-performance data pipelines and infrastructure for our next-generation data platform. The platform ingests and processes real-time and historical data from diverse industrial sources such as airport systems, sensors, cameras, and APIs. You will work closely with AI/ML engineers, data scientists, and DevOps to enable reliable analytics, forecasting, and anomaly detection use cases. Shape Major Skills : Spark, Flink, Iceberg Key Responsibilities · Design and implement real-time (Kafka, Spark/Flink) and batch (Airflow, Spark) pipelines for high-throughput data ingestion, processing, and transformation. · Develop data models and manage data lakes and warehouses (D...

I need a clear, scalable architecture that can detect and instantly alert on unusual spending patterns in our payment flow. The focus is narrow and well-defined: draw insights from live and historical transaction history, spot deviations from normal customer behavior, and trigger actionable alerts before settlement is completed. Scope of work • Map the entire transaction-processing path and identify points where real-time analytics can be inserted without adding noticeable latency. • Define the data lake or streaming layer that will store raw and enriched transaction history. • Recommend the analytics engine—rules, machine-learning models, or a hybrid—best suited for spotting spending anomalies while remaining extensible to other fraud signals later (for e...

We are seeking a highly skilled Azure Data Engineer to design, develop, and maintain robust data solutions on the Azure platform. This role requires strong technical expertise in Azure Data Engineering services and hands-on experience with Azure Kubernetes Service (AKS). Requirements: • Proven experience as an Azure Data Engineer with end-to-end data solutions • Strong proficiency in Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Databricks, and Azure Synapse Analytics • Hands-on experience with Azure Kubernetes Service (AKS) - this is essential • Solid experience in SQL programming and Python scripting • Experience with ETL processes, data modeling, and data warehousing concepts • Familiarity with version control systems (Git) and DevOps pra...