I would like to engage a coder skilled in scraping to assist with managing our scraper currently hosted on Amazon. It has a php control panel, using Python 3 for the scraper and MySQL for the database.
The scraper works but we require some functionality.
There are currently 2 scrapers - one for New Zealand and one for Australia. We scrape jobs on a large job board based on particular parameters being met. On the NZ site, we scrape ALL jobs in all categories meeting our criteria. On the AU site, we only scrape from the TRADE category.
We'd like the following enhancements:
1. Add the ability to scrape from all categories on the Australian site - that is, scrape all jobs on the site. There is a control panel that was originally setup to allow us to add multiple categories like this but it has subsequently been hard coded to only scrape from All Trades. (It's setup like this on the NZ site so it should be a simple alteration to the code).
2. The scraper is set to run continuously with a definable period to wait before running again. I'd like to ensure that when running through subsequent scrapes, jobs that have already been checked are skipped from re-checking to make the scraping faster.
3. The control panel allows to define various settings - I don't think the depth of scrape - i.e. "Past Days" is working as it's only going back 7 days. - [login to view URL]
4. I'd like a counter on the scraper that shows the total number of jobs available in the category and the number of jobs scraped/remaining.
5. I'd like error reporting integrated so we can determine if the proxies are failing.
6. We 'd like your recommendation on the best proxies we should be using.
If you are skilled in these languages and scraping, we want you to fix the following issues and offer ongoing services as required. You must be able to work fast, accurately and for a fair price.
38 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön
Hello. How are you today? I have good experience with python scraper & python3 db management etc. Please message me for more detail discussion. I read your project description and understand it.
Hello! I am a python developer. I looked at your project and it seems interesting. I have all necessary skills required for this project. Ping me to discuss in detail.