Spark Scla

Expertise in designing and deployment of Hadoop Cluster and different analytical tools including Pig, Hive, HBase, Sqoop, Kafka Spark with Cloudera distribution.

Working on a live 20 nodes Hadoop cluster running on CDH4.4.

Working with highly unstructured and semi structured data of 40 TB in size (120 TB with replication factor of 3)

Managing external tables in Hive for optimized performance.

Very good understanding of Partitions and Bucketing in Hive

Developed Spark scripts using Scala as per the requirement using Spark 1.5 framework.

Using Spark API’s over Cloudera Hadoop Yarn to perform analytics on data used for Hive stored at HDFS.

Developed Scala Scripts, UDFs using both Data frames/SQL and RDD in Spark for data aggregation, queries and writing data back onto HDFS.

Exploring Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark data frames, pair RDDs, double RDDs and Yarn.

Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.

Experience in deploying data from various sources into HDFS and facilitating report building on top of it as per the business requirement.

Performed transformations, cleaning, standardization and filtering of data using Spark Scala/Python and loaded the final required data to HDFS.

Load the data into Spark immutable RDDs and perform in-memory computation to generate quick and better response.

Analyzing how the data been processed by Informatica can be effectively processed using Spark and its API’s.

Taidot: Spark, Scala, Hadoop, Hive, SQL

Näytä lisää: different automation tools, name different institutes graphic interior designing lahore, list different automation tools, j2me designing different record stores application, compare different automation tools, expertise designing forms dreamweaver, powerpoint presentations comparison different automation tools, comparison different automation tools, comparison different automation tools power point presentation, poker analytical tools, different scale pictures including, Drawings of machines ,designing of all types of tools for automobile components, designing semi structured interview questions, different types of designing courses, analytical tools for data analysis, analytical tools for research, analytical tools and techniques, different spreadsheet tools, analytical tools, proxmox cluster different hardware

Tietoa työnantajasta:
( 0 arvostelua ) Queens, United States

Projektin tunnus: #30636255

6 freelanceria on tarjonnut keskimäärin $13/tunti tähän työhön


Hi, I am an experienced Data Engineer with a solid background in Spark. I have worked on many Big Data projects with Spark, Scala, Python, Cassandra, Snowflake, AWS ,... Let's have a call for more details about the pr Lisää

$8 USD / tunti
(2 arvostelua)

Hi, I have 6 years of experience and my entire work experience is on these technologies. I think we can discuss more about this

$15 USD / tunti
(1 arvostelu)

[login to view URL] are you today? I am professional software engineer with 7+ years experience in web field. Hive, Hadoop, Scala, Spark and SQL are my professional skills and have used it so many times for web dev. Please check Lisää

$40 USD / tunti
(0 arvostelua)

Having totally 10 years of IT experience which includes 6+ years in java and 3 years in big data. I have great experience in data engineering projects, hadoop, Hbase, Spark, Scala, Hive, Impala, Pig, and Big data proje Lisää

$5 USD / tunti
(0 arvostelua)

3 years of experience in Big Data Domain. Having experience in HDFS, Hive, Spark, Scala, Pyspark, Sqoop. Working in Capgemini.

$5 USD / tunti
(0 arvostelua)

As I am A meticulous and goal-driven Hadoop& Spark Developer with 2+ years of experience as an Hadoop Developer in Bigdata Solutions team. Adept at leveraging Hadoop and Spark for processing Large insights. Proven trac Lisää

$5 USD / tunti
(0 arvostelua)