Expertise Required: Python-based Web Scraper Development

@Microlent

Hello, I am Dr. Rajesh Rolen, PhD in Computer Science & Engineering, with experience of over 20+ years in PHP, React.js, Web Scraping, Python, Django As a preferred freelancer in the top 1%, I have done 400+ projects here on freelancer.com, I have 4.9 ratings out of 5 on average, which showcases my quality of work and timely delivery. Key Highlights: - Free Hosting Support on the Cloud or any desired platform. - Free 3 months of post-delivery support to ensure that our client doesn’t face any challenges after the launch of the project. - Free Dedicated tester on projects to ensure quality delivery, so clients don’t need to act as a tester. - 10+ Years experience UI/UX team to ensure intuitive UI. Portfolio: https://www.freelancer.com/u/Microlent Please open the chat and send me a message, so we can have a more detailed discussion about the project to give you the project timeline and cost. Thank you for considering my services. I look forward to engaging in a productive conversation and understanding how I can be of assistance in bringing your project to life. Regards Rajesh Rolen

@rajat07me

SCRAPER ENGINE A multi-tenant scraper platform like this usually fails at queue stability, dedup logic, and anti-bot handling long before the UI is finished. I’d approach it backend-first: define tenant-safe data models, queue architecture, retry/failure handling, and scraping pipelines first, then connect the dashboard, Sheets sync, exports, and webhook delivery on top of that. I’ve worked on Python-based scraping and automation systems involving dynamic sites, browser automation, background workers, API syncing, and admin-controlled processing flows. For this type of build, Django or FastAPI with PostgreSQL, Celery/Redis, and Playwright-based scraping is a solid foundation, with proxy rotation and per-source throttling designed around the target sites rather than bolted on later. To size the milestones properly, I’d like to confirm a few points: - Do you already have the target job portals/directories list, or is source discovery part of scope? - Should tenant isolation stay at shared-db with strict user_id separation, or do you want a stronger tenant model? - Is the £250 figure a placeholder for the first phase only, or for the full application build?

@starttechno

SCRAPER ENGINE — I’d start with the two Celery pipelines, a PostgreSQL deduplication key built from user_id plus source record identifiers, and a Playwright/Scrapy split for dynamic versus static portals. Proxy rotation would use a provider abstraction with isolated browser contexts, rate limits, retries, and challenge outcomes logged without blocking workers; Redis and Celery Beat would manage scheduling and progress events. I’ve built an AI outreach system covering Google Maps business discovery, contact scraping, Gemini personalization, and paced SMTP delivery with rotation and unsubscribe handling. In 7 days, I’ll deliver the tenant-isolated auth/RBAC foundation, both scraping engines, contact normalization, Excel export, Google Sheets OAuth, webhooks, admin scheduling, logs, and the React dashboard in staged milestones. Should Google Sheets OAuth be connected per user in the first MVP, or can it follow the core scraper release? Muhammad Saad

@DeveloperStation

SCRAPER ENGINE Hi, For the anti-bot side I lean on rotating residential proxies fronting stealth headless Playwright, with Selenium only where a site needs it, and Celery workers split into separate queues so scraping jobs never starve the enrichment and export tasks. Celery Beat handles the admin-controlled intervals, Redis holds the queue, and Postgres carries a hash index on user_id plus record key for the dedup lookups you described. One question on Module 2: for the LinkedIn/executive enrichment, do you already have API keys and budget for a provider, or do we design the interface provider-agnostic so you can swap later? We build multi-tenant Python SaaS with background AI workflows and Redis-backed queues. BD Automation: internal MangoCoders platform, evidence-backed workflows Since there is no history between us, we can start with a milestone on the first working scraping engine so you only release on delivered output. Which module do you want built first? Adil

@ranjika08

SCRAPER ENGINE The hard part here is not scraping alone; it is keeping two independent engines reliable under scheduling, retries, tenant separation, and changing site structures. I would break this into core layers first: source adapters, queue workers, normalized lead/contact schema, and dedup/enrichment flow, then wire that into the user/admin dashboard and Google Sheets sync. We handle custom Python builds where Playwright/Selenium scraping, background jobs, structured exports, and API integrations all need to work together cleanly. For a system like this, I’d likely recommend Django for the admin-heavy side plus Celery/Redis and PostgreSQL indexing for duplicate prevention and job tracking, with anti-bot measures managed per source profile. A couple of details would help lock the delivery plan: - How many initial sources are expected for module 1 and module 2 at launch? - For OTP verification, do you prefer email OTP, SMS OTP, or both? - Will enrichment APIs be provided by you, or should that connector layer be built as a generic plug-in interface first?

@shrec

SCRAPER ENGINE Hi, I reviewed the architecture and the scope is clear. This is best approached as a modular scraping platform, with Django or FastAPI handling the API layer, PostgreSQL for tenant-isolated data, and Celery + Redis for scheduled background jobs. For scraping, I’d use Scrapy for high-volume extraction and Playwright where dynamic rendering is required, with per-source selectors, pagination, rate limits, retry handling, and proxy rotation. The workers can be separated by scraping task so larger jobs do not block the dashboard or other tenants. I can also structure the contact discovery, deduplication, Google Sheets OAuth sync, Excel exports, webhooks, RBAC, logging, and progress tracking as independent components. I can’t provide 2–3 portfolio examples without details from you, so I won’t invent them. For the core application, I’d estimate 4–6 weeks, with milestones covering the platform foundation, scraping engines, enrichment/contact processing, dashboard, and integrations. A fixed price can be finalized after reviewing the source sites and exact requirements. Best, Ivane

@Herculesz

Hi, This project requires a robust Python backend with scalable scraping and task orchestration capabilities, which aligns well with my experience building automation pipelines and multi-tenant systems using FastAPI, PostgreSQL, and Celery. The main engineering challenge is reliably managing anti-bot countermeasures and background task orchestration to ensure continuous data extraction without service interruptions. I usually structure scraping systems with clear separation between ingestion, enrichment, and user-facing layers, enabling easier debugging and scaling. My work on AI-Driven Marketing Suite Development involved similar task queue management and API integration challenges. I approach reliability by implementing monitoring and alerting for task failures, along with fallback strategies for proxy rotation and CAPTCHA handling. Deduplication and data sync with external APIs like Google Sheets are also critical. I typically design systems for long-term production use, focusing on maintainability and operational transparency. Thanks, Hercules

@Joya1996

SCRAPER ENGINE What you’re outlining is really a data pipeline product with a web app around it, so the risky areas are tenant-safe processing, bot resistance, and data quality rather than just page extraction. My first step would be to map the scraping lifecycle end to end — collect, normalize, validate, deduplicate, enrich, sync/export — so the system stays maintainable as new sources are added. I’ve been involved in builds combining Python scraping stacks, browser automation for dynamic targets, queue-based processing, and dashboard workflows where admins need visibility into logs, failures, and scheduled tasks. Your spec suits a Django/FastAPI backend with Celery workers, Redis, PostgreSQL, and a React-based frontend, with source-specific throttling and headless browser strategies defined early. A few scope points to clarify before fixing milestones: - Do you need CAPTCHA solving integrated from day one, or only a proxy/stealth-ready architecture initially? - Should Google Sheets syncing support one sheet template per tenant or custom field mapping by user? - Are failure alerts only for admins, or should end users also receive job-status emails?

@naminghn

SCRAPER ENGINE To create a robust multi-tenant web application for automated scraping and lead management, I recommend utilizing Django with Scrapy for scalable background processing. My approach includes integrating Celery for task management and implementing effective anti-bot measures such as rotating proxies and CAPTCHA bypass techniques. This architecture ensures reliability while adhering to the specified requirements for contact discovery and data enrichment. Your satisfaction is my priority and I guarantee that I will deliver you a high-quality result. Regards, Ali

@AnumFatima210

Hi, I went through your project details and understand what you're looking to achieve. I'd be interested in working with you on this. Based on the requirements you've shared, I can handle the project from start to completion while keeping the work aligned with your expectations, timeline, and desired outcome. My approach is straightforward: first, I'll make sure the requirements and expected deliverables are clear. Then I'll complete the work in structured stages, share progress along the way, incorporate your feedback, and deliver the final work after everything has been reviewed. Before we begin, I have a couple of questions about your requirements that will help me understand the scope more accurately. Feel free to message me so we can discuss the details and determine the best way to move forward. Best regards, Anum

@toriquldev123

Hello There! I’m Md Toriqul Islam, an experienced full-stack developer with 10+ years of experience specializing in Python, Django, React, scraping, automation, and scalable data platforms. I understand you need a multi-tenant scraping and lead-generation platform with two independent engines, contact enrichment, deduplication, background processing, Google Sheets sync, RBAC, scheduling, and real-time monitoring. I can build a modular architecture using Celery/Redis, PostgreSQL, Scrapy/Playwright, and React/Next.js. I am skilled in Python, Django, FastAPI, Scrapy, Playwright, Celery, Redis, PostgreSQL, React, APIs, and Docker. I have some questions: 1. Which sources should be prioritized for the first scraping engine? 2. Do you already have proxy providers and enrichment APIs? Looking forward to hearing from you. Best regards, Md Toriqul Islam

@devendrathakur12

SCRAPER ENGINE Hi, I’m a Senior Full Stack Python Developer with 20+ years in data pipelines and web platforms. I have gone through your specific requirement for multi tenant scraping platform I built something like this for data driven clients, with 20+ years across Python backends. I would use Celery with Redis over synchronous scraping because 200 records per source needs workers to fail independently without blocking user requests. I will build Django APIs with PostgreSQL tenant isolation and Celery Beat for scheduled scraping jobs. For crawling, I will use Scrapy for normal pages and Playwright only where JavaScript rendering is needed, keeping browser workers separate. And I will keep proxy rotation and failure handling inside workers, at least that is where I would start. Relevant Python, scraping and data pipeline samples I can send. Which sources already have selectors or scraping rules defined? How many concurrent users and sources do you expect at launch? Which enrichment providers and CRM webhook formats are already chosen? Free for a quick call this week? Or answer those three and I’ll map out the first version. Dev Singh

@denisSolution

Hello, I'm Denis, a full-stack developer with experience building data extraction systems and multi-tenant applications. I've worked extensively with Python scraping pipelines, background task processing, and dashboard systems like yours. I understand your core needs: reliable scraping with two engines, high-volume data extraction, automatic contact enrichment, and clean deduplication for Google Sheets sync or export. The anti-bot layer and fallback logic are critical for data quality. I've built similar systems using Scrapy with Playwright for dynamic content, Celery workers for background processing, and PostgreSQL with proper indexing for fast deduplication. I'm experienced with rotating proxies, CAPTCHA detection, and stealth browser configuration. My approach: start with a requirements review to lock down scraping targets and enrichment priorities. Build the backend with Django or FastAPI, set up Celery with Redis for task queuing, and implement multi-tenant database structure with proper isolation. Each scraping engine would be modular with its own selector logic and rate limiting. Contact discovery runs as post-processing, followed by a React dashboard with real-time monitoring and Google Sheets sync. Key technical consideration: deduplication performance at scale. I'd use PostgreSQL indexes on hash columns combined with user_id for fast, efficient lookups without impacting write speed. One realistic risk: scraping targets may change structure or strengthen anti-

@PINESCRIPTERR

SCRAPER ENGINE Hi, I can build this with Django, PostgreSQL, Celery/Redis, Playwright, and Next.js. Each tenant will have isolated data, RBAC, queued jobs, deduplication, progress logs, exports, webhooks, and per-user Google Sheets OAuth. I’ll use rate limits, retries, API-first collection, and compliant browser automation. I would not bypass CAPTCHAs or access controls; blocked sources should use approved APIs or licensed providers. Relevant examples can be shared privately. Which target websites and directories are required for the first milestone? Regards, Houssame

@technologiesit

As someone who has been deeply immersed in the world of software development for the past 8+ years, I've worked on over 200 projects which has fortified my expertise in full stack development, making me more than qualified for your web scraping project. I've been extensively involved with Python and its frameworks, and have quite a reputation when it comes to building scalable applications with Django or FastAPI. In addition, my proficiency extends to the tools and technologies needed for a successful scraper development including but not limited to Scrapy, Playwright, Selenium, and BeautifulSoup. To address the meat of the project; I have had substantial experience not only with job-site scraping but also general data and lead generation as well as handling automated contact discovery and formatting operations - all which aligns perfectly with what you're seeking. Furthermore, providing multi-tenancy solutions is something I'm well-versed in via role-based access control (RBAC) ensuring a safe environment where users' data remain logically separate.

@krupnovdenis

Hello! We can provide an external team to build this scraping and lead management platform. 1. Are you open to working with an external contractor or team for this project? 2. Which module should be delivered first? — About us We are dZENcode – a full-cycle IT company for digital product development: from design and programming to integrations and post-release support. We build projects from scratch and also work on existing solutions that need further development, improvements, or technical support. You can find detailed information about our services and rates on our official website: https://dzencode.com. Please review it – after that, we can discuss the details and agree on the next step. ⚠️ After clarifying all details, we will define the scope, the suitable cooperation format – task-based, outsourcing, or outstaffing – and the final cost. Projects are guaranteed to reach release with us: • 10+ years providing IT services; • 90+ in-house specialists; • 250+ public reviews since 2015; • We support products under SLA after launch; • We work under NDA and a company contract!

@BarryCrouch

SCRAPER ENGINE — ✋ Hi, the main challenge here is not the individual scrapers but keeping two scraping engines, tenant isolation, enrichment, deduplication, Google Sheets sync and Celery workloads reliable under one application. I’d separate source-specific crawling from the shared processing layer so Scrapy/Playwright workers can handle collection while normalization, contact discovery, deduplication and exports stay reusable. Celery queues should also be split by workload so browser-heavy jobs cannot block lighter enrichment or sync tasks. I’d first establish the tenant/data model and one complete scrape-to-storage pipeline, then add queue management, enrichment, Sheets OAuth and the second engine before expanding monitoring and admin controls. Do you already have the initial target websites and expected extraction schemas defined for both scraping engines, or will source discovery and selector design also be part of the build?

@Tanmoy236

SCRAPER ENGINE Warm greetings! I'm an expert in Python web scraping and scalable data platforms, with over 9 years of experience building complex scraping, lead generation, and automation systems. Here's how I can help: Build both scraping engines with Scrapy, Playwright, Celery, Redis & PostgreSQL Develop multi-tenant Django/FastAPI + React dashboards with RBAC Implement proxy rotation, stealth browsing, rate limits & resilient crawling Add contact discovery, enrichment, validation, deduplication & Google Sheets OAuth Deliver XLSX exports, webhooks, monitoring, logs & failure alerts I can share 2–3 relevant examples and provide a fixed-price milestone plan. What launch timeline and preferred Django/FastAPI stack do you have in mind?

@sasaprogramer21

SCRAPER ENGINE Hi, Aleksandar here, from Serbia, to help you. A multi-tenant scraping platform like this only works long term if deduplication and user isolation are baked into the database schema from day one, not retrofitted once real data starts flowing. I noticed you want hash-based deduplication scoped to user_id plus unique record identifiers - that usually means the schema needs a composite index designed around that check from the first migration, so I'd build that layer before touching the scraping engines themselves. I'd build this with Django and Celery/Redis for background task processing, Playwright with rotating proxies for the anti-bot layer, PostgreSQL with indexed deduplication, and a React dashboard with RBAC and OTP authentication. Given the budget listed here is quite small relative to the full scope described, would you like the first milestone to focus on one scraping engine plus the core dashboard, or the multi-tenant auth and data architecture foundation first? Looking forward to working with you.