Big Data Analyst

₹1250-2500 INR / hour

Closed

Posted about 1 month ago

₹1250-2500 INR / hour

I have a large, fully structured dataset sitting in HDFS and I need it transformed into clear, decision-ready insights. The goal is pure data analysis: design the workflow, build and tune the jobs, and leave me with a repeatable, well-documented process that does not require exporting data out of the cluster. Everything must run on Apache Hadoop (current 3.x stack on Cloudera CDP). If you feel a touch of Hive, Pig, or even straight MapReduce will speed things up, I am open to it, but Hadoop remains the core platform. SQL engines or Spark can be mentioned if they genuinely simplify a step, yet the final solution must stay centred on Hadoop. Deliverables: • Working Hadoop jobs that clean, aggregate, and store results back to HDFS • Clear, commented code in Git • A concise hand-off guide (read-me or screenshare) so my in-house team can rerun the workflow unaided Accuracy, performance tuning, and straightforward documentation are more important to me than flashy dashboards. When you reply, please reference comparable structured-data analysis you have completed on Hadoop and your estimated turnaround time.

Project ID: 40311000

About the project

8 proposals

Remote project

Active 1 mo ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

8 freelancers are bidding on average ₹1,703 INR/hour for this job

@bhartimalu

Hello, I can assist with Apache Hadoop data analysis including data processing, ETL, and generating insights using tools like Hive and Spark. Ready to start immediately. Regards, Bharti

₹1,875 INR in 40 days

5.0

(1 review)

2.2

@vishala81

Hello, This is exactly the kind of in-cluster analysis Hadoop was built for. I’ve worked on large structured datasets in HDFS (Cloudera/CDP environments) where the requirement was to clean, aggregate, and derive insights without moving data out of the cluster. The key is designing efficient jobs that minimize shuffles and I/O while remaining easy to rerun. I can build a repeatable workflow using native Hadoop components — typically Hive (for fast aggregation and SQL-like transformations) combined with MapReduce where custom logic or performance tuning is needed. If appropriate, I may leverage Spark on YARN for specific steps, but the solution will remain fully Hadoop-centric and store results back in HDFS as requested. Deliverables will include production-ready jobs, well-commented code in Git, and a clear hand-off guide so your team can execute the pipeline independently. I focus on correctness, resource efficiency, and operational simplicity rather than unnecessary complexity. I can start immediately. Turnaround will depend on dataset size and required transformations, but I work quickly once schema and objectives are clarified. Best regards, Vishal

₹1,250 INR in 40 days

5.0

(1 review)

1.7

@Yafoor1996

Title: Big Data Analyst | Hadoop & Hive Workflow Automation "I am a Data specialist focused on building repeatable, documented data pipelines. I recently completed a project involving the normalization and auditing of a 100+ record dataset, where I used Python and Regex to transform raw, 'dirty' data into structured, decision-ready formats. For your HDFS transformation, I can provide: Hive/SQL Optimization: I will design Hive scripts to clean and aggregate your data directly within the cluster, ensuring no data egress is required. Documented Workflow: I will provide a clear README and commented code in Git so your team can re-run the jobs independently. Performance Focus: I prioritize clean, efficient logic over flashy visuals to ensure your Hadoop 3.x stack runs at peak performance. I am comfortable working within the Cloudera CDP environment and can ensure all deliverables stay centered on Hadoop. I estimate a turnaround time of [ 3-5 days] once I review the specific data schema."

₹1,875 INR in 40 days

0.0

(0 reviews)

0.0

@khrystynad4

Hello, This is a great fit for my background in distributed data processing and structured data pipelines. I have hands-on experience working with Hadoop ecosystems, including HDFS, Hive, and MapReduce-based workflows, with a focus on building efficient, repeatable data processing jobs. For your project, I will: -Design a clean, end-to-end Hadoop workflow directly on HDFS (no external data movement) -Implement data cleaning, aggregation, and transformation using Hive/MapReduce (and Spark only if it clearly improves performance) -Optimize jobs for performance and resource efficiency on CDP -Store processed outputs back into HDFS in structured, query-ready formats -Deliver well-documented, reusable code in Git -Provide a concise handover guide so your team can run everything independently I’ve previously worked on structured datasets involving batch processing, aggregation pipelines, and performance tuning in distributed environments, so I understand the importance of accuracy, scalability, and maintainability. Estimated turnaround: 3–5 days, depending on dataset size and complexity. I can start immediately and keep the solution simple, robust, and production-ready. Best regards.

₹1,875 INR in 40 days