Data ingestion using Kafka,Spark-Scala

Suoritettu Julkaistu 5 vuotta sitten Maksettu toimituksen yhteydessä
Suoritettu Maksettu toimituksen yhteydessä

I want a PoC to be done for the below requirement.

Source data files(in CSV format and SQL extract format) and a schema files (in JSON format) will be placed in GCS. Based on the schema definition rules given in the json file, data in CSV file should be loaded to Kafka and using Kafka streaming, data needs to be transformed to 3NF form and loaded to GCS/Google Big Query . The main objective is,When schema changes , dynamically the code should absorb the changes in kafka without modification in the code.

Skillsets: Big Data-Kafka,Spark-Scala,GCS

Big Data Sales Google Cloud Storage Scala Spark

Projektin tunnus: #17600107

Tietoa projektista

7 ehdotusta Etäprojekti Aktiivinen 5 vuotta sitten

Myönnetty käyttäjälle:

deytps86

Hello, I am working in Bigdata/Hadoop technologies for years and have experiences working in latest Spark,Kafka, Cassandra, Hive/HBase, ELK stacks. Can we talk in details? Thank you!

£220 GBP 3 päivässä
(15 Arvostelua)
4.4

7 freelanceria on tarjonnut keskimäärin £261 tähän työhön

amitkumar0327

Hi, I am Amit. I have experience in Spark, Scala, Kafka and GCP. I will be able to do the POC based on the requirement. I will be interested to know more about the Schema files that will be placed in GCS. The Schema fi Lisää

£150 GBP 3 päivässä
(2 arvostelua)
2.6
rumeshkrish

Hi, I have experience in spark streaming and Kafka streaming. We can discuss more and work on. I have knowledge on spark writing into big query. so you will use datapro to start the spark streaming. and this will read Lisää

£561 GBP 15 päivässä
(1 arvostelu)
2.8
dineshrajputit

hi, i can do it and work on spark Kafka pipeline..

£155 GBP 3 päivässä
(2 arvostelua)
1.9
surajosiinfotec4

Hello We are group of freelancers working extensively on Big Data Technologies. We have worked on Apache Kafka, Apache Spark, Scala Programming and GCS. We have worked on similar project: We are currently worki Lisää

£250 GBP 3 päivässä
(0 arvostelua)
0.0
Rakeshmeel5

We are almost 2.5 Yrs Experinced of BigData Industry and finished 10+ project exclusively in spark with scala/Java and Kafka

£266 GBP 6 päivässä
(0 arvostelua)
0.0