Monthly Web Scraper for EU Institutional Site

€250-750 EUR

Suoritettu

Julkaistu 29 päivää sitten

€250-750 EUR

Maksettu toimituksen yhteydessä

Build a monthly web scraper for a European institutional website (projects, good practices, events) I need a scraper that extracts textual content from all pages, PDFs and other documents available in a specific website. These would cover: • Approved projects / project pages (project description and key fields) • Good practices (good practice pages + text fields) • Events (event pages + date/location/description) Requirements: • Python preferred (Scrapy + Requests/BS4; Playwright if needed) • Output as CSV (or database + exports) • Run automatically once per month • Track changes: new/updated/deleted pages since last run • Provide clean code, documentation, and logs • Respect robots/ToS and implement rate limiting Deliverables: • Source code repo • Deployed scheduled job (GitHub Actions / Cloud Run / AWS Lambda / VPS cron) • Example output files • Setup instructions and maintenance notes I need a well-structured Python scraper that harvests the textual content of three sections on an institutional European site: approved project pages (project description + key fields), good practice pages, and event pages (including date, location, description). The crawler must detect and label anything new, updated, or removed since its previous run so that the dataset always reflects the current state of the site. Stack & execution – Please build it with Scrapy and plain Requests; fall back on headless techniques only when unavoidable. – The job will run automatically every month through GitHub Actions, so the repo should contain a workflow file that installs dependencies, executes the crawl, and pushes the fresh export to a dedicated branch or release asset. Data handling – One tidy CSV is the only required export, but architect the pipeline so the data objects could just as easily be dumped in other formats later. – Versioned outputs should indicate the crawl date and summarise counts of added / changed / deleted records in a log. Operational rules – Honour [login to view URL] and any Terms of Service; throttle politely with an adjustable rate-limit. – Emit clear logs, HTTP error handling, and retry logic. – Include a README covering setup, environment variables, and how to tweak schedules or selectors when the site changes. Deliverables • Git repository with fully commented source code, Scrapy settings, and the GitHub Actions workflow • Example CSV produced from initial crawls • Change-tracking mechanism (diff JSON or similar) demonstrated in that first run • README / maintenance notes explaining deployment, updating selectors, and extending output formats

Software Architecture

Web Scraping

Projektin tunnus (ID): 40311852

Tietoa projektista

31 ehdotukset

Etäprojekti

Aktiivinen 29 päivää sitten

Haluatko ansaita rahaa?

Sähköpostiosoite

Freelancerin tarjouskilpailun edut

Aseta budjettisi ja aikataulu

Saa maksu työstäsi

Kuvaile ehdotustasi

Rekisteröinti ja töihin tarjoaminen on ilmaista

Myönnetty käyttäjälle:

@farhanyoussef18

Youssef, Full-Time Freelancer with Python Programming expertise in web scraping, automation, and data extraction, skilled with Scrapy, Playwright, and BeautifulSoup. I can build your monthly web scraper for the European institutional website, focusing on approved project pages, good practice pages, and event pages. My solution will precisely extract textual content, efficiently handle dynamic elements, and implement robust change tracking to detect new, updated, or deleted content. I will set up the automated monthly runs via GitHub Actions, ensuring adherence to [login to view URL] and polite rate limiting. I have extensive experience building and deploying similar sophisticated scraping and automation solutions.

€500 EUR 1 päivässä

5,0

(161 arvostelut)

7,3

31 freelancerit tarjoavat keskimäärin €457 EUR tätä projektia

@ZohaibRoy

⭐⭐⭐⭐⭐ Build a Monthly Web Scraper for European Institutional Website ❇️ Hi My Friend, I hope you are doing well. I just reviewed your project requirements and see you are looking for a web scraper for an institutional website. You have no need to look any further; Zohaib is here to help you! My team has successfully completed 50+ similar projects for web scraping. I will create a Python scraper using Scrapy and Requests to gather data on approved projects, good practices, and events. The scraper will run automatically every month, ensuring you always have the latest information. ➡️ Why Me? I can easily build your web scraper as I have 5 years of experience in web scraping and automation, specializing in Python, Scrapy, and data extraction. My expertise includes data handling, error management, and scheduling automated tasks. Besides, I have a strong grip on GitHub Actions and cloud services to ensure smooth deployment. ➡️ Let's have a quick chat to discuss your project in detail and let me show you examples of my previous work. Looking forward to discussing this with you in chat. ➡️ Skills & Experience: ✅ Python Programming ✅ Web Scraping ✅ Scrapy Framework ✅ Data Extraction ✅ Automation ✅ API Integration ✅ CSV Handling ✅ Error Handling ✅ GitHub Actions ✅ Documentation ✅ Rate Limiting ✅ Change Tracking Waiting for your response! Best Regards, Zohaib

€350 EUR 2 päivässä

4,9

(160 arvostelut)

8,1

@diektech

Hi, This is Elias from Miami. I checked your project description and understand you're looking to build a monthly web scraper for a European institutional website to gather information on projects, good practices, and events. I have experience developing similar web scraping solutions that efficiently extract and manage data. For this project, I recommend using Python with Scrapy for the scraping framework, paired with BeautifulSoup for parsing HTML. This setup is robust and allows for easy data extraction, while also enabling scalability as the website evolves. Utilizing AWS for deployment will ensure reliability and allow for automated monthly runs, with data being stored securely and managed effectively. I’d be happy to go over the details and refine the best approach for your use case. Q1 – What specific data points are you looking to extract from the website? Q2 – Are there any particular performance benchmarks or limits you want the scraper to adhere to? Q3 – Do you have a preferred format for the extracted data (e.g., CSV, JSON, etc.)? Looking forward to hearing from you.

€500 EUR 11 päivässä

4,9

(7 arvostelut)

7,3

@OscarMozo

Hi, I can help you with this. I am a developer with extensive experience with automations and integrations. I've helped clients with similar projects. Let me know your interest, Sincerely, Nicolas

€500 EUR 7 päivässä

5,0

(5 arvostelut)

5,3

@Hubready

Hello!, I am a US-based senior software engineer with extensive experience in Python, data processing, and web scraping. I’ve read your project description thoroughly and understand your need for a monthly web scraper for your European institutional site focused on projects, good practices, and events. With around 15 years of experience in building scalable software solutions, I’m confident I can deliver a robust and efficient scraper tailored to your requirements. To ensure I’m aligned with your vision, could you please clarify the following questions to help me better understand the project? 1. What specific data points do you need to scrape from the site? 2. Are there any particular formats or databases you prefer for the extracted data? 3. Do you have any existing infrastructure on AWS that the scraper should integrate with? My approach involves designing a multi-phase plan: first, I’ll create a prototype to validate the scraping methodology; second, I’ll implement and test the scraper; and finally, I’ll set up automation for regular data extraction. I’m dedicated to delivering a solution that not only meets your expectations but also scales with your needs. Let’s chat to discuss further! Best, James Zappi

€500 EUR 5 päivässä

5,0

(3 arvostelut)

3,8

@DngPhgNam

⭐⭐⭐⭐⭐ ✅Hi there, hope you are doing well! I have built similar web scrapers for institutional and public sites where data was extracted monthly, tracked, and organized in easily consumable formats such as CSV. From my experience, the key to successfully completing this project is implementing a robust change-tracking mechanism while respecting site usage policies. Approach: ⭕ Use Scrapy with Requests/BS4 as primary tools, resorting to headless browsing only if necessary. ⭕ Architect the scraper to extract and structure data for approved projects, good practices, and events. ⭕ Implement versioned CSV export including date stamps and change summaries. ⭕ Automate monthly runs using GitHub Actions with clear logging, error handling, and polite rate limiting. ⭕ Provide comprehensive documentation, setup instructions, and maintainable codebase. ❓ ❓ Could you please share the target website URL to understand structure and access restrictions? ❓ Are there any specific fields or formats for extracted data you'd like to prioritize? ❓ Do you have preferences on where the output CSVs should be stored or pushed? I am confident in delivering a clean, maintainable scraper that meets your requirements efficiently. Looking forward to collaborating with you. Best regards, Nam

€550 EUR 5 päivässä

5,0

(1 arvostelu)

3,8

@SkillCrafts

I have extensive experience building robust data pipelines for EU portals, including ESIF and Horizon databases, where navigating nested structures and inconsistent regional formats is a common challenge. I understand that institutional sites require high precision to capture every 'good practice' and event entry without duplicates during recurring monthly runs. My focus is on delivering high-fidelity, structured data that mirrors the site’s complexity while remaining perfectly cleaned and ready for your internal database or analysis tools for long-term project tracking. For this recurring task, I will deploy a Python-based architecture using Scrapy for performance or Playwright if the site relies on dynamic JavaScript. I’ll implement custom middleware to manage session headers, ensuring the scraper remains undetected by institutional firewalls. Data will be processed through a validation pipeline to handle schema changes automatically, and I will containerize the solution using Docker to ensure it runs consistently in the cloud or via a scheduled GitHub Action, providing you with a fully hands-off automation. Do the project entries contain PDF attachments that need to be parsed, or should we focus on the on-page metadata? I am also curious if you need the monthly data to be cumulative or a fresh snapshot of the site’s current state. I am available for a brief call or chat to review the target URL and provide a granular breakdown of the extraction logic so we can begin immediately.

€602 EUR 21 päivässä