
In Progress
Posted
Paid on delivery
PROJECT OVERVIEW We need a small, working proof of concept that collects retail sales transactions over MQTT, stores them locally in SQLite, loads them into Hadoop (HDFS + Hive) and produces basic sales reports. Everything must run on one machine with Docker so we can demo it and build on it later. DATA FLOW 1. A sales simulator (Python) publishes transactions to an MQTT broker (Mosquitto) 2. A collector subscribes and saves every message in SQLite (it keeps working if Hadoop is down) 3. A loader moves new SQLite rows to HDFS on a schedule and on demand 4. Hive tables on HDFS and SQL queries produce the sales reports DELIVERABLES 1. [login to view URL]: Mosquitto, single-node Hadoop 3.x (HDFS + YARN), Hive. One command starts it all. 2. Sales simulator - 5 stores, 200 products, configurable rate (default 20 messages/sec) - Topic: sales/{store_id}/transactions - JSON fields: transaction_id, store_id, product_id, category, quantity, unit_price, payment_method, timestamp (UTC, ISO 8601) 3. MQTT to SQLite collector - QoS 1, reconnects automatically - Validates each message. Bad messages go to a "rejected" table instead of being dropped silently. - A repeated transaction_id is ignored (primary key) 4. SQLite to HDFS loader - Uploads only rows not yet uploaded, and marks them as uploaded only after HDFS confirms the write - Partitioned by date, e.g. /data/sales/dt=YYYY-MM-DD/ - Safe to re-run: never creates duplicates in HDFS 5. Hive tables and five reports, each exported to CSV: - Daily revenue per store - Top 10 products by revenue and by quantity - Hourly sales trend - Revenue by category - Revenue by payment method 6. README covering setup, running each component, configuration and how to extend it 7. A 5-10 minute screen recording of the full pipeline working ACCEPTANCE CRITERIA - On a fresh machine, "docker compose up" plus the README steps runs the full pipeline - The simulator sends 50,000 transactions and the Hive row count matches exactly - Hadoop is stopped for 10 minutes while the simulator runs. After restart, no data is lost and nothing is duplicated. - Running the loader twice in a row creates no duplicates - All five reports run, and their totals match a spot-check against SQLite - Clean code, delivered in a Git repository TECH STACK Python 3.10+ (paho-mqtt, sqlite3), Eclipse Mosquitto, Apache Hadoop 3.x, Apache Hive, Docker Compose. Spark SQL instead of Hive is acceptable if you explain why. TIMELINE (5 DAYS) Day 1: Docker environment running, simulator publishing to MQTT Day 2: Collector writing to SQLite with validation and duplicate handling Day 3: Loader to HDFS with date partitions and safe re-runs Day 4: Hive tables and all five reports Day 5: Failure testing, README, recording, handover MILESTONE PAYMENTS 30% - MQTT and SQLite working, shown in a live demo (end of Day 2) 40% - Data in HDFS, reports running (end of Day 4) 30% - Final delivery, after all acceptance criteria pass OUT OF SCOPE Production or multi-node cluster, cloud deployment, web dashboard. MQTT TLS and authentication are welcome but not required. HOW TO BID - Start your bid with the word "PIPELINE" so we know you read this - Describe one Hadoop/Hive or MQTT project you have done (a link or a few lines) - Confirm your fixed price and that you can deliver in 5 days - Suggest any change you would make to this design Bids missing these items will not be considered.
Project ID: 40766232
109 proposals
Remote project
Active 5 days ago
Set your budget and timeframe
Get paid for your work
Outline your proposal
It's free to sign up and bid on jobs

PIPELINE :) Hello **"MQTT-TO-HADOOP DATA PIPELINE"** - What can break this demo is losing transactions during Hadoop downtime or creating duplicate rows when the loader runs again. I'll build a Docker Compose setup with Mosquitto, a Python sales simulator and MQTT collector, SQLite validation and deduplication, plus a retry-safe HDFS loader and Hive reporting. I'll verify recovery after downtime, 50,000-message counts, partitioning, repeat loads, and all five CSV reports against SQLite. You'll receive clean code, setup documentation, and a recorded end-to-end demo. My Python, Hadoop, SQLite, Docker, and data-processing skills align with this workflow. One improvement I'd make is using stable transaction IDs and idempotent date-partitioned writes, with upload status updated only after confirmed HDFS writes. I'll also include a free pipeline integrity checklist. Best regards Van Hieu
$300 USD in 3 days
0.0
0.0
109 freelancers are bidding on average $170 USD for this job

Hi There, Hope you are well. I am interested in developing the proof of concept for the retail sales data pipeline project. My experience in Python, MQTT, SQLite, Docker, and Hadoop makes me a suitable candidate. I have previously worked on a similar project integrating MQTT with SQLite for real-time data processing. Thanks in advance for the opportunity to working with you. Best Regards, Serazum
$55 USD in 1 day
7.6
7.6

PIPELINE Hello I checked your project and the main point is a Docker-based Python pipeline that safely carries MQTT sales data through SQLite, HDFS, Hive, and CSV reports. I have experience with Python data processing, SQLite workflows, validation, and large dataset handling. I can build the simulator, Mosquitto collector, rejected-message handling, and idempotent HDFS loader using paho-mqtt, sqlite3, Docker Compose, and Hive. I don't have a public Hadoop/MQTT sample link to share, but I can demonstrate the complete pipeline through the required recording. I would use transaction_id as the deduplication key, SQLite upload states for recovery, and temporary HDFS files followed by atomic renaming. Hive date partitions and repeatable queries will keep the five reports consistent with SQLite, including the Hadoop outage test. I can deliver this as a fixed-price project within your requested schedule, with the repository, README, tests, and handover included. I’d suggest adding a small checkpoint or manifest table for each HDFS partition, making reruns safer even after a partial write. Could we clarify the preferred Hadoop and Hive Docker images, along with the expected CSV layout, before implementation? Best regards
$999 USD in 5 days
7.6
7.6

PIPELINE. (Putting it per your request) I have previous MQTT experience, including publishing messages and receiving responses, and I can build on that experience for the messaging component of this project. I would implement the pipeline in stages: a Python simulator publishing JSON transactions to Mosquitto, a reliable MQTT collector storing validated records in SQLite, and an idempotent loader transferring new records to date-partitioned HDFS storage. Hive would then provide the SQL queries for all five CSV sales reports. One design detail I would pay particular attention to is recovery after Hadoop downtime. SQLite should remain the durable local queue, and records should only be marked as uploaded after a successful HDFS write. I would also account for retries and partial failures to prevent duplicate transactions when the loader runs repeatedly. For the proof of concept, I would keep the components simple and independently testable, with Docker Compose for the environment and a README documenting setup, configuration, recovery, and verification. I confirm the fixed price and five-day delivery commitment after assessing the Hadoop/Hive container setup and the expected demo environment.
$146 USD in 5 days
7.0
7.0

PIPELINE. Hi there, I understand you need a Docker-based pipeline that streams retail transactions through MQTT, stores them reliably in SQLite, loads them into HDFS, and generates five validated Hive sales reports. My approach would cover five stages. First, I’d configure Docker Compose with Mosquitto, Hadoop, and Hive, alongside a Python simulator generating configurable transaction volumes. Next, I’d build an MQTT collector with QoS 1, automatic reconnection, message validation, rejected-message logging, and transaction-ID deduplication. Then, I’d develop an incremental SQLite-to-HDFS loader with date partitioning, upload tracking, and safe retries. I’d configure Hive tables and all five CSV reports, then test 50,000 transactions, Hadoop downtime recovery, duplicate prevention, and report accuracy against SQLite. Finally, I’d deliver documented code in Git, setup instructions, and a 5-10-minute demonstration recording. Could you confirm whether the HDFS loader should run at a fixed interval, on demand, or both, as specified in your requirements? I’d recommend keeping SQLite as the durable ingestion buffer and using idempotent batch loading with recovery checks to prevent data loss and duplication. Please initiate a chat so we can discuss the technical requirements and next steps. Warm Regards, Aneesa.
$85 USD in 1 day
7.1
7.1

PIPELINE Hi There! As a Top 1% verified freelancer with 11+ years of experience and 465+ completed projects, I specialise in building containerised big data pipelines, real-time MQTT streaming, and robust ETL architectures using Python, SQLite, Docker, Hadoop, and Hive. I recently developed an IoT telemetry pipeline integrating Mosquitto brokers with Python collectors and HDFS/Hive storage, ensuring zero data loss during network outages through SQLite buffering. I confirm my fixed price and 5-day delivery capability. As a design enhancement, I suggest incorporating Spark SQL alongside Hive for report generation, as it handles date-partitioned CSV exports and aggregations significantly faster on a single Docker node. What is your preferred fixed price for this proof of concept?
$140 USD in 7 days
6.6
6.6

PIPELINE — Hello, i’ve done very similar recently with production data pipelines, Python processing, and reliable automated data delivery. I’d build your Docker Compose setup with Mosquitto, Python/paho-mqtt, SQLite, Hadoop HDFS, and Hive. My experience includes aggregating 23 live data sources with validation, idempotent processing, and automated delivery. I suggest using SQLite upload-status tracking and deterministic HDFS filenames so retries survive Hadoop downtime without duplicate rows. Can we use date-partitioned JSONL files as the HDFS staging format before Hive queries? I’ll deliver the simulator and collector first, then the safe HDFS loader and five CSV reports. Finally, I’ll test 50,000 transactions, Hadoop downtime, and repeated loads, then provide the README, Git repository, and demo recording within five days. Best, Dev S.
$250 USD in 4 days
6.6
6.6

Hello, I'm Denis, a full-stack developer experienced in building data pipelines that reliably move transactions from message brokers into data warehouses, handling both success and failure cases. You need a working proof of concept that captures retail sales from MQTT, buffers them safely in SQLite, loads them into Hadoop on schedule, and produces five standard reports—all running on one machine in Docker while surviving Hadoop downtime without losing or duplicating data. I would build a Docker Compose stack with Mosquitto, single-node Hadoop, and Hive. Then create a simulator for realistic transactions, an MQTT collector with validation and duplicate detection using transaction_id as the primary key, and a loader that tracks uploaded rows for safe re-runs. The critical part is the loader: I would mark rows as uploaded only after HDFS confirms the write succeeds, then query SQLite for unmarked rows on each run. This ensures Hadoop downtime causes no data loss or duplication. I would validate every MQTT message and send invalid ones to a rejected table so nothing disappears silently. I'll use a tested Docker image to minimize Hadoop setup complexity and document any quirks in the README. I have built MQTT-to-database collectors and batch loaders for IoT sensor data. I can deliver this in 5 days for $2,500 fixed price and suggest adding a CSV output from SQLite as a sanity check for report totals. I can start immediately. Let's discuss the details. Thanks, Denis.
$30 USD in 2 days
6.0
6.0

PIPELINE hi thanks for the detailed specification i’d implement the pipeline with python, mosquitto, sqlite and a dockerized single-node hadoop environment. the collector would validate and persist messages independently of hadoop availability, while the loader would use reliable upload tracking and idempotent writes to prevent data loss or duplication. i’d then configure hive tables, generate all five csv reports and test the complete 50,000-transaction flow, including hadoop downtime and repeated loader runs. one improvement i’d recommend is using deterministic transaction batches and explicit hdfs file manifests so recovery and duplicate verification are easier to demonstrate. two quick questions: - must hive run as a separate service, or would you consider spark sql if it simplifies the single-machine docker setup? - do you already have a preferred base image or docker environment for hadoop and hive, or should i select compatible versions? Houssame
$60 USD in 3 days
6.5
6.5

PIPELINE. Hi there, I have reviewed your brief and understand you need a complete, locally runnable data pipeline that reliably moves retail transactions from MQTT into SQLite, then HDFS and Hive, with accurate sales reports and no data loss or duplication even during Hadoop downtime. I can help you build this as a modular, Docker-based proof of concept with reproducible end-to-end testing. My approach will be to configure Docker Compose with Mosquitto, single-node Hadoop, and Hive, then develop the Python sales simulator and MQTT collector with QoS 1, message validation, rejection logging, automatic reconnection, and duplicate protection. I will implement an idempotent SQLite-to-HDFS loader with date partitioning and upload-state tracking, followed by Hive tables and all five requested CSV reports. I will also test the 50,000-transaction requirement, Hadoop downtime recovery, repeated loader runs, and report totals against SQLite. As final deliverables, you will receive the complete Git repository, Docker configuration, clean source code, README, verified reports, and a 5-10-minute demonstration recording. Could you confirm whether Hadoop and Hive must both be included, or would you consider Spark SQL if it simplifies the five-day delivery? I can work toward your five-day milestone schedule, with delivery validated against the agreed acceptance criteria. Regards, Imran A.
$80 USD in 1 day
6.2
6.2

PIPELINE Hi, I am a Python data engineer with 8 years of rich experience in software development, with a background in streaming data pipelines and Docker setups. I am familiar with Python, paho-mqtt, Mosquitto, SQLite, Hadoop, HDFS, Hive, Spark and Docker Compose. One change I'd make is in the loader. Each batch would be written under a fixed name built from its first and last row IDs, uploaded to a temp path and then renamed in HDFS, which is atomic. Rows are only marked as uploaded after that rename succeeds, so a crash or a second run just overwrites the same file and can never duplicate data. I'd also store batches as Parquet instead of CSV so Hive reports run faster, and register each new date partition right after upload so reports always see fresh data. The collector would use SQLite in WAL mode with a persistent MQTT session, so nothing is lost while Hadoop is down. I'm an individual freelancer and can work on any time zone you want. Please contact me with the best time for you to have a quick chat. Looking forward to discussing more details. Thanks. Emile.
$250 USD in 7 days
5.9
5.9

PIPELINE — I will build a complete MQTT-to-Hadoop data pipeline for you using Docker Compose, incorporating Python 3.10+, Mosquitto, SQLite, HDFS, and Hive. My approach: Day 1 – Simulator and Docker setup; Day 2 – QoS 1 collector and validation; Day 3 – Secure HDFS loader; Day 4 – Five CSV reports; Day 5 – Testing for 50,000 transactions, Hadoop downtime, duplicate prevention, and data reconciliation. I will also provide a README, a Git repository, and a 5–10 minute demo video. 1. Is there a preferred Docker image available for Hadoop/Hive? 2. Is a daily live demo and milestone review feasible over the five-day period? Suggestions: Implement date-based partitioning in HDFS and transaction-ID-based idempotency; mark SQLite records as 'uploaded' only after confirming the upload. I will focus on delivering clean, testable code and verifying all acceptance criteria. Please clarify which five CSV reports are required, and confirm the pricing and deliverables based on the scope.
$150 USD in 7 days
5.6
5.6

Hi there, PIPELINE I am excited to submit my proposal for your MQTT to Hadoop Sales Data Pipeline project. I understand you need a proof of concept that demonstrates a seamless data flow from an MQTT broker to Hadoop, with SQLite as an intermediary and Hive for reporting. With extensive experience in data integration and processing, I am well-equipped to deliver this solution. In my previous project, I developed a similar data pipeline using Hive to process and analyze IoT sensor data collected via MQTT. This involved setting up a robust MQTT broker and ensuring data integrity through careful validation and duplicate handling. For your project, I will ensure a reliable setup where the sales simulator efficiently communicates with the MQTT broker, and the collector meticulously stores validated transactions in SQLite. The loader will be carefully implemented to manage data uploads to HDFS without duplication, even after interruptions. The Hive tables will be optimized for generating insightful reports that meet your specified criteria. My approach emphasizes clean, maintainable code, backed by comprehensive documentation and a screen recording to guide you through the system. I am confident in my ability to meet the acceptance criteria and deliver a robust solution you can build upon. I look forward to the opportunity to work on this project and provide a seamless data pipeline solution. Best Regards,
$30 USD in 5 days
5.3
5.3

PIPELINE — I can build a Docker-based retail sales data pipeline using Python, MQTT, SQLite, Hadoop HDFS, Hive, and Docker Compose. I would start by validating the single-node Hadoop and Hive environment, then implement the configurable sales simulator and a QoS 1 collector with automatic reconnection, message validation, rejected-message tracking, and transaction ID deduplication. Next, I would build a retry-safe loader that writes date-partitioned data to HDFS and marks SQLite records as uploaded only after verifying successful writes. I would configure the Hive tables and export all five requested CSV reports, then test the 50,000-transaction count, a 10-minute Hadoop outage, loader reruns, and report totals against SQLite. I propose a fixed price of $1,000 and can target delivery within 5 days, provided the container compatibility and environment are confirmed on Day 1. My relevant background includes Python, SQLite, Docker, and data-processing development; I cannot provide a verified, directly comparable Hadoop/Hive or MQTT project link, so I will not invent one. One design improvement would be to use deterministic batch filenames and HDFS manifests, allowing retries to verify existing batches before writing again. Would you be able to confirm the expected demo operating system and whether Spark SQL is acceptable as a fallback if the Hive image proves incompatible?
$140 USD in 4 days
4.5
4.5

Spark for transforms, MQTT broker feeding into HDFS via a consumer that batches transactions before landing them, then a SQLite layer for quick queries on top. I can run it all in Docker so it is easy to hand off. About 4 days for a working proof of concept, available to start today. These numbers are based on the description as written and may shift once we go through the full data flow together. Want me to send a quick scope doc so we can get moving?
$150 USD in 6 days
3.9
3.9

**PIPELINE** Hello, I can build this MQTT-to-Hadoop pipeline using a resilient architecture that keeps the system running even if Hadoop is down, while ensuring no duplicates in HDFS. I’ve set up a similar system for a retail client—used Mosquitto and Hive to process 100K transactions per day with zero data loss during outages. I’d approach this by running the collector as a separate service with SQLite as a buffer, then using a scheduled loader to move data to HDFS with date partitioning. The biggest improvement will come from handling failures gracefully—like ignoring duplicates and validating messages before writing to HDFS. Proper setup will reduce your risk of lost data and ensure the reports are accurate. I’ll also include a failure test script to confirm the system works even if Hadoop is stopped. Once the project is complete, I’ll walk you through the setup and provide a checklist for future maintenance. If this approach works, I’d be happy to discuss the details further. Thanks, Lazar.
$30 USD in 1 day
3.7
3.7

Nice to talk you , After reading in detail the requirements of your project and concluding that they match my areas of knowledge and skills, I would like to introduce myself. My name is Anthony Muñoz and I am the lead engineer for DS Pro IT agency. I have worked for over 10 years in Backend and software development and have successfully done multiple jobs. It will be a pleasure to work together to make your project a reality. Please feel free to contact me. I´m looking forward to working with you. I really appreciate your time and remain attentive to any request or question. Greetings
$171 USD in 7 days
3.8
3.8

PIPELINE Hi, I can help build your MQTT-to-Hadoop sales data pipeline using Python, Mosquitto, SQLite, Docker Compose, HDFS, and Hive. My focus would be reliable data ingestion, duplicate prevention, recovery after failures, and reproducible deployment. Proposed implementation: • Day 1: Docker environment and configurable MQTT sales simulator. • Day 2: QoS 1 collector with SQLite persistence, validation, and duplicate handling. • Day 3: Incremental HDFS loader with date partitions and restart-safe processing. • Day 4: Hive tables, SQL queries, and CSV sales reports. • Day 5: Run 50,000-record validation, Hadoop outage recovery tests, documentation, and demo recording. One architectural improvement I'd recommend is deterministic batch identification with atomic HDFS publication to prevent duplicates if the loader crashes between uploading data and updating SQLite. Deliverables include complete Python source, Docker Compose configuration, Hive SQL, five report categories, README, and a working demonstration. Fixed price: $100 Timeline: 5 days, subject to confirming the environment and scope. Could you confirm your target OS, available RAM, and whether Hive is mandatory or Spark SQL is acceptable? Best regards Ankit
$100 USD in 5 days
3.9
3.9

IF YOU ARE NOT HAPPY YOU DON'T PAY! PIPELINE The hardest of your acceptance criteria is the loader. If HDFS confirms the write but the 'uploaded' mark in SQLite fails, a rerun appends the same rows again. I'd write each batch to a deterministic file name like dt=YYYY-MM-DD/batch_<first_id>_<last_id>. json via a temp file and rename, so a rerun overwrites the file and can't duplicate it. Being honest: I haven't delivered a Hadoop/Hive or MQTT project I can point you to. My Python and Docker work is in custom back-ends. Fixed price is $250 and 5 days works for me. One design change: I'd run Hive with a local metastore and skip HiveServer2 load on YARN, so it fits on one laptop. Is the demo machine 16GB RAM or more? Send me a message and I'll explain exactly how I'd approach your project.
$250 USD in 3 days
3.5
3.5

Hey, PIPELINE. I'm Ruslan, and I specialize in utilizing Python and Data Analysis for complex projects, which is why I am confident in delivering the MQTT to Hadoop Sales Data Pipeline you need. I have a deep proficiency in the technologies required, including paho-mqtt, sqlite3, Hadoop, Hive and Spark SQL. In fact, I recently completed a similar project for a large retail company where I built a real-time data pipeline that employed MQTT and Apache Kafka for collecting and processing sales transactions from multiple stores. Additionally, over my years of practice in the field, I have mastered resilience and accuracy in data management. I can build the SQLite collector to automatically handle duplicate transactions ensuring no data is lost or duplicated during any failure or system disruptions. This same guarantee extends to creating CSV reports that match against SQLite's totals accurately. Finally, I promise clean code delivery in a Git repository with detailed guidelines on setup and usage (the README you seek) as aesthetic code greatly simplifies extension. Let me help you turn this seed of an idea into an innovative reality!
$70 USD in 3 days
3.8
3.8

Hi, yes, I can build the MQTT sales pipeline into SQLite, HDFS and Hive with the five reports, done in 5 days. The loader is the risky part: a row is marked uploaded only after the HDFS write, and the file name comes from the transaction ids so a rerun replaces it. REST API to database sync on my profile is the same incremental re-run pattern. - Within hours of the award you can watch the simulator publish and the collector store good rows and reject bad ones. - Then date partitions, Hive, and the five reports checked against SQLite. - Last is the Hadoop-down test, a second loader run, the README and the screen recording. Milestone 1 is 30 USD, broker plus simulator plus collector, the same day. Milestone 2 is 58 USD, loader, reports, failure tests, README and recording, within 5 days. The price is 88 USD. I can start the moment you award this, on the simulator or on the Mosquitto compose, whichever you want first.
$88 USD in 5 days
3.4
3.4

Frisco, United States
Payment method verified
Member since Sep 29, 2026
$15-25 USD / hour
$10-11 USD
$15-25 USD / hour
₹12500-37500 INR
$1500-3000 USD
₹37500-75000 INR
$250-750 USD
₹12500-37500 INR
₹750-1250 INR / hour
₹750-1250 INR / hour
₹12500-37500 INR
₹1500-12500 INR
₹12500-37500 INR
$15-25 USD / hour
€12-18 EUR / hour
$25-50 USD / hour
₹600-1500 INR
$250-750 USD
₹100-400 INR / hour
₹100-400 INR / hour