Suljettu

Create Production Ready ETL Pipeline using Python, Docker and Dagster

We need to create a dockerizable ETL Pipeline for our data using Python and Dagster. Our requirement / data flowchart is very clear.

The basic flow is read data from CSV, save to Kafka. Apply multiple transforms using functions (function code will be provided by us). Use intermediate kafka queues after each transform. Write final output to Postgres DB. Use Dagster for the data orchestration.

We just need to implement it in a robust manner. We need :-

1. Docker-compose to setup the Kafka, Postgres and other services.

2. Create a DAG for orchestration of the pipeline using Dagster (using placeholders for the transform functions which our team will add later).

Taidot: Python, Docker, Docker Compose, ETL, Apache Kafka

Näytä lisää: read csv save database using php, create app using python, how to create a dynamic website using python, python etl pipeline, how to create a print ready pdf file using adobe illustrator, how to create a website using python, how to create vm using python, how to create front end using python, how to create a tool using python, how to create a software using python, how to create dashboard using python, how to create a mobile app using python, how to create a social network site using python, how to create a website using python pdf, how to create windows application using python, create ppt using python, etl automation using python, how to create haar cascade xml for using with opencv python, how to create a registration form using python, create table in sql server using python

Tietoa työnantajasta:
( 0 arvostelua ) New Delhi, India

Projektin tunnus: #32203643

3 freelanceria on tarjonnut keskimäärin ₹22667 tähän työhön

f9programmer

Hello , I am a developer based out of Hyderabad , India . I have developed and hosted a few applications for my clients on PHP , Laravel , Nodejs , Vuejs , React and React Native . I have also helped my clients Lisää

₹35000 INR 7 päivässä
(3 arvostelua)
3.1
singhyamini1821

** Can start immediately once you hire me. Hello, I am an experienced mobile app developer with an experience of 5+ years. I’m pretty confident that I will be able to create your app as required. I am an expert in ma Lisää

₹20000 INR 5 päivässä
(0 arvostelua)
0.0
poiyutchain

Expertise in Python and Microservices: specialization: - Python, Django, Spark, ETL, Hadoop, BigData processing and analysis SQL and NoSQL - MySQL, PostgreSQL, MongoDB, Cassandra, Redis, etc - Docker - JMS, ActiveMQ Lisää

₹13000 INR 7 päivässä
(0 arvostelua)
0.0