Web crawling amazoncom työt

Project Title: Web Crawler Target Websites: - Search engines Data Gathering: - The web crawler should gather both text and image data Data Frequency: - The web crawler should gather data on a daily basis Ideal Skills and Experience: - Proficiency in web scraping and data extraction - Experience with crawling and parsing HTML and CSS - Knowledge of image scraping and handling - Familiarity with data storage and management systems - Strong programming skills in languages such as Python or Java - Understanding of web protocols and APIs - Attention to detail and ability to handle large volumes of data efficiently

I am looking for a Google Search Console expert who can help me resolve crawling issues on my website and increase its visibility. Specific issues with my website in Google Search Console include crawling issues. My main goal in resolving these issues is to increase my website's visibility. I want to rank for everything related to my business. Ideal skills and experience for this job include: - Expertise in Google Search Console and its features - Knowledge of website crawling and indexing - Experience in improving website visibility and rankings - Ability to conduct keyword research and optimize website content for targeted keywords.

I am looking f... "value": 123, "Final Score": 300 } } ] } 3. Output should contains (1) failed indicator names and failed reason (2) remapping indicator_name and value fields provide (1) (2)Python code (3)A simple test case( and test command) Skills and Experience: - Strong experience in web scraping and data crawling - Proficiency in Python or another suitable programming language for web scraping - Familiarity with the World Bank's data structure and API Data Format: - The crawled data should be in JSON format. Data Cleaning and Structuring: - The client requires the data to be cleaned and structured according to specific data attributes. Please provide examples of similar projects you have complete...

...configurable. Data Formatting and Storage: The scraped data needs to be formatted and stored in a MongoDB. It should be updated every week with the latest information. What to Deliver: A python (3.11) project that can be run locally and/or on Google Cloud. I want to be able to configure multiple scraping/crawling sessions that all in paralel get this info. I should be able to plan the times it is running and the city/area it should crawl. Ideal Skills and Experience: - Proficiency in web scraping and data extraction techniques - Experience with Python and working with GCP. - Experience working with Google Business API or similar platforms - Strong knowledge of MongoDB and data storage - Familiarity with data formatting and updating processes - Attention to detail and...

I need JUST the website URLS for all the Gift Shops in the U.S. Should be around 20-30k I think. This should be an easy scrape. Only one piece of information to gather. It could be done by crawling a google search or by using google maps/earth. Just set it and come back a few days later when it's done. To show you read this, please bid exactly $19. That is what I would like to pay. If the way you know how to do this is too time consuming to bid $19, please don't bid. There is someone out there who knows an easy, fast way to do it, and they will bid $19. Also, please tell me when you expect the list will be ready.

Webcrawling Project - ES I am looking for a skilled webcrawler who can extract text data from a specific website and deliver it in CSV format. Specific Requirements: - Webcrawling expertise: Experience in crawling specific websites and extracting data efficiently. - Text data extraction: Proficiency in extracting text data from webpages, ensuring accuracy and completeness. - Data formatting: Ability to deliver the extracted data in CSV format for easy analysis and manipulation. Ideal Skills and Experience: - Proficient in webcrawling tools and technologies. - Knowledge of HTML and CSS to navigate and extract data from webpages. - Strong attention to detail to ensure accurate and complete extraction of text data. - Experience in formatting data in CSV format. If you have the nece...

Robust human target detection and acquisition/locking without losing track under occlusions for outdoor operational scenarios. Visuals alarm generation upon anomaly detection (i.e human jumping/ crawling/ running). I am looking for a skilled developer who can create a robust human target detection and acquisition system with high accuracy (above 90%). The system should be able to track multiple targets simultaneously, which is a crucial requirement. Additionally, the system should be able to function in both indoor and outdoor environments. Ideal Skills and Experience: - Strong background in computer vision and object detection algorithms - Proficiency in programming languages such as Python or C++ - Experience with deep learning frameworks like TensorFlow or PyTorch - Familiarity...

I am...looking for someone to help me with the settings on my e-commerce website. I am currently facing crawling issues and I have no understanding of the settings and its role in SEO. Requirements: - Experience in working with settings on e-commerce websites - Strong understanding of SEO and its impact on website crawling and indexing Tasks: - Review and update the settings to resolve crawling issues - Provide explanations and guidance on the role of settings in SEO Ideal Skills and Experience: - Extensive experience in working with settings on e-commerce websites - Deep understanding of SEO and its impact on website crawling and indexing Please provide examples of previous projects related to settings and SEO.

...630 keywords - Web frameworks 125 - programming 200 - cms system 20 - cloud platforms 60 - databases 85 - developers 100 - online stores 40 Job Description: The main task in this project is to optimize our website for search engines (SEO) to improve organic search results. The task includes - but is not limited to - the following key elements: Keyword research: in addition to existing keyword lists, Identify more relevant keywords related to our products, services and target audience. Analysis of "long tail" keywords is also encouraged. On-page SEO: Revise our website's meta titles, descriptions, headings, image alt text and content to optimize for keywords. Technical SEO: Review the website code for errors and potential for improvement in terms of load times...

I am looking for a freelancer who can help me crawl a specific website for personal use. The main purpose of ...me crawl a specific website for personal use. The main purpose of the website crawl is data extraction. I have a specific website in mind that I would like to crawl. Key requirements for this project include: - Proficiency in web scraping and data extraction techniques - Experience with crawling websites and extracting data efficiently - Knowledge of programming languages such as Python or PHP for automating the crawl process - Familiarity with tools and libraries such as Beautiful Soup or Scrapy for data extraction If you have previous experience in website crawling and data extraction, and are able to efficiently extract data from a specific website, ple...

...on website CMS, if our developer can access the back-end code / file. 3. SSL Certificate - You will have to buy SSL certificate and need to provide us with the SSL and Hosting/Domain Details so that we can enable it 4. User Experience (UX) - can be increased Up to some extent 5. 404 Error Pages - If developer can access the back-end, that will be additional and cost you additionally 6. Core Web Vitals - can be fixed Up to some extent On-Page SEO 1. Duplicate Content - You will have to provide us with the Text and we will update or if we will need to write the text content, that will be considered additional work and you will have to pay for that 2. H1, H2 Tags - if CMS will allow, we can do it 3. Meta Titles & Descriptions - can be done from our side 4. Keyword Optimiza...

I am looking for a freelancer who can help me with a web crawling project. I will provide a list of websites that I want to crawl, and I do not need the data to be cleaned and structured as I will handle that myself. The intended use of the crawled data is for other purposes. Ideal Skills and Experience: - Proficiency in web crawling and data extraction - Knowledge of programming languages such as Python or Java - Familiarity with web scraping tools and libraries - Attention to detail and accuracy in data collection - Ability to work independently and meet project deadlines

...on website CMS, if our developer can access the back-end code / file. 3. SSL Certificate - You will have to buy SSL certificate and need to provide us with the SSL and Hosting/Domain Details so that we can enable it 4. User Experience (UX) - can be increased Up to some extent 5. 404 Error Pages - If developer can access the back-end, that will be additional and cost you additionally 6. Core Web Vitals - can be fixed Up to some extent On-Page SEO 1. Duplicate Content - You will have to provide us with the Text and we will update or if we will need to write the text content, that will be considered additional work and you will have to pay for that 2. H1, H2 Tags - if CMS will allow, we can do it 3. Meta Titles & Descriptions - can be done from our side 4. Keyword Optimiza...

Data Scraping Tool to Extract Contact Information from Apollo Skills and Experience: - Proficiency in data scraping and web crawling techniques - Experience with Apollo platform and familiarity with its data structure - Strong understanding of data quality and cleaning processes - Expertise in handling CSV files and manipulating data within the format - Attention to detail and ability to accurately extract contact information Project Description: I am looking for a skilled freelancer who can develop a data scraping tool to extract contact information from the Apollo platform. The tool should be able to scrape the required data efficiently and accurately. Specific Requirements: - Scraping Contact Information: The tool should be designed to scrape contact information from th...

...Little Helpers" you will notice that the website home page is indexed, but the content is not displayed with the message" No information is available for this page. Learn why", which points to an issue with Robots.TXT. However all testing using tools provided by google as well as external tools shows that the is not blocking the site from being crawled and there should be no issue with crawling the site? I have attached Screenshots to highlight the problem. The problem seemed to have started around the beginning of July 2023. I had not made any changes to the website when the problem started happening. Info that may help : 1) Site is built in Shopify 2) Site has been operating for many years 3) The domain was changed from . to .com in June 2022 Please can you ...

Desarrollo de un multisitio web educativo que integre herramientas de ● Uso de atributos ALT en las imágenes del sitio web. ● Gestión de protocolos Lazy Load para Ifrems, imágenes y videos. ● Optimizador de velocidad Web y CDN. ● Uso de protocolos de Crawling para optimización de indexación del sitio web. Incluye migración de información de periodos anteriores Tiempo de desarrollo 2 meses Solo requiero propuesta de Ecuador por el tema de facturación de servicios profesionales.

Web crawling for conference I am looking for a freelancer who can perform web crawling for a conference website. The specific information that needs to be extracted is the schedule and agenda of the conference. Ideal Skills and Experience: - Proficiency in web crawling and data extraction - Experience with extracting data from conference websites - Familiarity with Excel and the ability to deliver the extracted data in an Excel spreadsheet format The web crawling task needs to be performed once, and the data should be delivered in an Excel spreadsheet. Attention to detail and accuracy are crucial for this project.

I am looking for a freelancer to create and deliver an Excel Spreadsheet of product names and their respective prices, volumes (if possible) and before discount prices (if possile) from a beauty e-commerce website. The needs are for a one-time task, meaning I do not need the web crawling to be done on an ongoing basis. If you feel that you possess the skills necessary to complete this task, please reach out and we can discuss the specifics in more detail.

Ich suche für ein B2B Daten Projekt einen langfristigen Technik-Partner bzw. einen Full Stack Developer. Für eine erfolgreiche Zusammenarbeit sind folgende Kenntnisse in den Programmiersprachen erforderlich: Laravel, PHP, PHP Crawling/Scraping, JavaScript, MySQL, API Entwicklung - Idealerweise bringst du auch Kenntnisse in Python mit. Auch GitHUB sollte für dich kein Problem sein. Unser langfristiges Ziel ist es, eine Vertrauensbasis aufzubauen und einen zuverlässigen Partner für Aufgaben wie Fehlerbehebungen, Design-Anpassungen, UI/UX-Weiterentwicklung, Datenbankanpassungen sowie Datenbankoptimierung zu haben. Zudem soll die Plattform kontinuierlich weiterentwickelt und verbessert werden. Falls du Interesse an dieser Herausforderung hast und die e...

...the website's performance on search engines. Project Scope: SEO Audit: Conduct a thorough audit of our website to identify strengths, weaknesses, and opportunities for improvement. Analyze current SEO performance, including website structure, content quality, backlink profile, and technical aspects. Identify areas for potential penalties, duplicate content, and issues affecting search engine crawling and indexing. Keyword Research: Perform in-depth keyword research specific to the insurance industry in the US. Identify high-volume and relevant keywords related to insurance services, policies, and coverage. Analyze keyword competitiveness and prioritize target keywords based on search intent and potential traffic. On-Page Optimization: Optimize existing website content fo...

对类似boss直聘，猎聘，58同城的网站app的数据监控和采集，需要爬取会员的手机号码（在目标网站上是公开的信息）、城市、时间、招聘主体、岗位类型等信息，并导出表格。最好留有采集历史和导出历史等，价格可谈。 For the data monitoring and collection of website apps like boss direct Pin, Liepin and 58 cities, it is necessary to crawl the member's mobile phone number (which is open information on the target website), city, time, recruitment subject, post type and other information, and export the form. It is best to leave collection history and export history, etc., the price can be talked about.

...identify and SOLVE any potential issues that may have arisen during the migration process. Key requirements for the project include: - Conducting a thorough audit of the website to identify any technical SEO issues, including 301 redirects - Checking for any broken links or missing pages - Ensuring that the website's sitemap, and htaccess files are properly configured for optimal crawling and indexing by search engines - Ensuring that search console is properly set up for the 3 shops Ideal skills and experience for this job include: - Strong knowledge of WordPress and its SEO capabilities - Experience conducting SEO audits and implementing on-page optimization strategies - Familiarity with tools such as Google Analytics 4, Google Search Console and Semrush (all of them...

If you have previous experience with developing a script for a web scraper on I would like to hear from you. Please state in your reply if you have worked with before. I will only consider offer from developer that has previous experience developing against the Apify api and platform. I need a basic web scraper that can scrape all pages on a WordPress website. It needs to find an element with a specific class or id and get the text content from that element. If that element is not found the grab the content of the body As it will be used as part of an automated process it must entirely be operated using the CLI. Additional requirements: - Scrape the text content of pdf files. - Report progress to a webhook at intervals - Post the content in batches to a webhook - Must

I will optimize your website for SEO, resulting in a higher ranking on search engines once crawling is complete. Within 1-2 weeks of project completion, you will observe your website being indexed on the first page. Our primary and targeted keyword will be "HereToKnow". There are a total of 6 pages and 60 post pages that have been published. The charge per page will be 150 Indian Rupees. Therefore, the total number of pages is 66, resulting in a total amount of 9,900 Indian Rupees. To calculate this, we multiply 66 by 150, which equals 9,900. Our goal is to finish this project before the deadline.

I am looking for a skilled freelancer who can develop a data crawler and scraper for my project. The primary goal is to extract ...video live streaming data both I need to fetch from the other websites. Websites to scrape data from: - [Input field for client to manually input websites] - News websites - Social media platforms Real-time or periodic updates: - Real-time - Periodically - I'm not sure, please advise Frequency of data updates: - Hourly - Daily - Weekly Ideal skills and experience: - Proficiency in web scraping and data crawling - Strong knowledge of different scraping techniques and tools - Experience in extracting real text feed and video data - Familiarity with various websites and social media platforms - Ability to handle real-time data updates effic...

...crawler will start from a set of brand and boutique websites (200 more or less). This initial list must be extended through the crawling activity at points 1 and 2 (from each brand website new boutiques will be discovered and from each boutique website new brands will be discovered) in an automatic circular loop, till all the sector websites will be crawled. Key Requirements: - Ability to extract data from many different websites with different structures (we believe this will require ML techniques) - No specific crawling techniques required due to anti-crawling measures - Information should be updated on a monthly basis Ideal Skills and Experience: - Proficiency in web scraping and data extraction - Familiarity with fashion websites and their structure - K...

Project Title: Crawler stops running on Linux server Description: - I am looking for a skilled developer who can help...Although there is no specific error message provided, the crawler has been stopping unexpectedly on the Linux server. - I have access to the server logs, which may provide useful information for debugging and resolving the issue. - The ideal candidate will have experience in troubleshooting and debugging Python programs, particularly on Linux servers. - Additionally, familiarity with web scraping and crawling techniques would be beneficial for understanding the code and identifying potential issues. - The goal of this project is to identify and fix the issue causing the crawler to stop running on the Linux server, ensuring that it runs smoothly and continuo...

1. Crawler for Real Estate Listings: Objective: To create a web crawler that fetches real estate listings from specific websites related to Mallorca (such as Engel, First Mallorca, Idealista, Foto Casa) and downloads 20 new listings daily. Features: Web Crawling: Crawl specified websites and extract real estate listing information. Filtering and Parsing: Parse the listings to extract key details like price, location, number of bedrooms, etc. Daily Updates: Automate the crawling to run daily and retrieve 20 new listings. 2. WordPress Plugin: Objective: To integrate the information obtained by the crawler into the back office of a WordPress site with clean design. Features: Display Listings: List properties in a clean and structured way. Drop-Down Menu and Filte...

Crawling data from Remote website using php-webdriver I am looking for a skilled developer to help me crawl data from a remote website using php-webdriver. The specific data I need to crawl is the text content. Also remote website uses Jquery ajax to avoid from direct html parser. Skills and experience required: - Proficiency in php-webdriver - Strong understanding of web scraping and data crawling techniques Data storage: The crawled data should be saved into a database. Crawling frequency: The data should be crawled and updated on a weekly basis. Please include examples of previous projects or relevant experience in your proposal.

...Feeding Journey: Record of breastfeeding, formula feeding, or introducing solids. Baby's reactions to new foods. Weaning process and transitioning. Chapter 7: Developing Personality: Notable personality traits emerging over time. First laughs, cries, and expressions. Interactions with siblings or pets. Chapter 8: Physical Achievements: Rolling over, sitting up, crawling, and standing. Baby's reaction to different stimuli (toys, music, etc.). First attempts at grabbing, holding, and manipulating objects. Chapter 9: Communication Milestones: Babbling, making sounds, and first words. Attempts to mimic sounds or gestures. How the baby responds to different voices and languages. Chapter 10: First Adventures: Baby&#...

...Ongoing SEO maintenance needed: Yes, both on-page and off-page SEO - Desired timeframe for seeing results: More than 3 months I am looking for an experienced SEO specialist who can help optimize my website for better search engine rankings. The ideal candidate will have expertise in technical SEO, including optimizing website structure, improving page load speed, and ensuring proper indexing and crawling by search engines. In addition to technical SEO, I also require ongoing SEO maintenance, both on-page and off-page. This includes keyword research, content optimization, link building, and monitoring and adjusting strategies to improve search engine rankings over time. I understand that SEO is a long-term process and it takes time to see significant results. Therefore, I have ...

I am looking for a skilled freelancer to create images for an ad. I also need a logo/ watermark for the pictures Type of images needed: - Photographs The topic is party, it will be for a pub crawling. company and need this to be done today. Requirements: - Ability to create both original and stock images - Experience in graphic design and photography - Proficiency in working with JPEG file format Ideal Skills and Experience: - Strong photography skills - Creativity in designing eye-catching images - Proficiency in graphic design software - Familiarity with stock image websites If you have the necessary skills and experience to create high-quality images for an ad, please submit your proposal.

...platforms. Social Media Marketing - Leverage popular social media platforms to engage with the target audience and generate leads. This includes creating and sharing relevant content and responding to comments and queries. Technical SEO: Improve website speed and optimize it for mobile viewing. Identify and fix crawl errors, broken links, and any issues that could impact search engine crawling and indexing. Implement structured data () to enhance search engine understanding of the site's content. Local SEO: Optimize the website for local searches, considering the nature of the medical tourism business. Manage and optimize Google My Business listing. Content Strategy: Content Marketing – Implement a strategy for marketing content developed for the s...

I will provide an excel file with links of the product detail pages of Link Format: I need a program which gets the content text and pictures of these pages into an MS access database. Every data row should include for every div class="prems--cms-container" - The item number "c01-108-b023eb" of the above sample link - ID - text - pictures Details of which div I want to get to be defined Prefered programing language: MS Access or VB.Net Please give me your price per hour and time estimate in hours.

Looking for an SEO Specialist, who will do onpage and off page optimization of my website to grow organic traffic to my website Keyword Research and Analysis: Conduct comprehensive keyword research to identify relevant, high-traffic, and low-competition keywords for our industry. On-Page SEO Optimization: Implement on-page SEO best practi...Research and Analysis: Conduct comprehensive keyword research to identify relevant, high-traffic, and low-competition keywords for our industry. On-Page SEO Optimization: Implement on-page SEO best practices, including meta tags, headings, URLs, and internal linking, to optimize website pages for targeted keywords. Technical SEO: Audit and identify technical issues that may impact search engine crawling and indexing. Off-Page SEO and Link ...

I am looking for a freelancer who can help me identify and resolve indexing issues on Google. Main Issue: Indexing Issue...who can help me identify and resolve indexing issues on Google. Main Issue: Indexing Issues I am unsure of the reasons why indexing is not working, so I need someone with expertise in troubleshooting and resolving these types of issues. Timeline: Not necessary in a rush. Skills and Experience: - Experience in identifying and resolving indexing issues on Google - Knowledge of Google's crawling and indexing algorithms - Familiarity with website optimization techniques - Strong problem-solving skills - Excellent communication and collaboration abilities If you have the skills and experience necessary to solve this indexing issue, please submit...

I want to ask an expert to review the Google Search Console and help with fixes to enable sooner crawling by Google. Project using live time traking monitoring.

Need help in crawling content from a blogspot which contains links, images and plain text. It is legal and no issues.

I am looking for a talented freelancer to create 2D illust...mind, so the illustrations should be in line with this concept. The site searches for ads for used cars on other sites, allowing the user to gather various offers on the platform. The illustration should clearly show the idea of crawlers gathering car ads The links below show similar generic images, which can be adapted for the automotive market. The image will be insert in this template

I am looking for a freelancer who can develop an Search engine (Apache Nutch 1.0) Crawler system with an integrated AI backend. The project requires the following functionalities: - Optimized crawling and indexing - Advanced crawling and indexing with custom plugins - Crawling, indexing, and AI-driven data analysis - single page search engine The data analysis should be integrated with an existing database system. Additionally, the ideal candidate should have experience in working with Neural Networks as the AI algorithm.

I am looking for a soccer auto-betting software that has the following features: - Live match updates and statistics to stay updated on the latest information - Automated betting based on pre-set strategies for efficient and seamless betting - Integration with popular bookmakers for easy and convenient betting I already h...experience for this project include: - Proficiency in developing betting software with live match updates and statistics integration - Knowledge of automated betting strategies and experience in implementing them - Familiarity with popular bookmakers and their API integration for a seamless betting experience - Ability to create a user-friendly interface for easy usage of the software -Familiar with crawling technology, familiar with WINFORM, C# 4.8, WEBSOCKET,...

...research tools and analyzing search trends. Technical SEO: Understanding website architecture, crawling, and indexing mechanisms. Skills in identifying and fixing technical issues like broken links, crawl errors, page speed optimization, and implementing structured data (schema markup). Content Creation: The ability to produce high-quality, engaging, and valuable content that aligns with user intent and incorporates relevant keywords. Content can include articles, blog posts, infographics, videos, and more. Link Building: Knowledge of strategies to acquire high-quality backlinks from authoritative and relevant websites to improve a website's authority and trustworthiness. Analytics: Proficiency in using web analytics tools like Google Analytics to measure and analyze ...

$10-30 USD Have you ever heard of TradingView? Do you know what PineScript is? Do you have experience with Javascript? Do you know about website crawling techniques and extracting hidden/protected data? Are you a creative thinker who can "think outside the box" when solving a problem? If you answered "yes" to all the questions above, you might be suitable for the job! I have recently purchased access to a protected/private trading script on TradingView (written in PineScript) and I need your help to extract the source code. I have no idea if the task has been done in the past or not, but your job will be to find a way to do it. Only apply if you are confident you can think of at least one way to achieve it. One common approach might be going through the Chr...

6 Month old Headless CMS website with over 150K pages & growing fast. Similar to a (cars for sale) listing-style site with a blog. Current team consist of a head developer, front end designer, back end SEO structure. Attached are snapshots of the sites indexing issues (see attached), Needed is someone that can jump...issues (see attached), Needed is someone that can jump into the mix & fix the issues, (not an advisory role, but an actual do the work role). We require a proficient individual who can integrate with our team to address indexing issues. The candidate must be capable of troubleshooting existing problems and establishing systems for future content indexing. A comprehensive understanding of swiftly crawling and indexing sizable content sites is essential.

...and web crawling to extract data from a website to extract data for 3 bed 2 bath townhomes which has no basement. 1. Extract for sold properties 2. Extract for listed properties . Here are the details of the project: Website: To be provided Data Refresh: - The data needs to be refreshed once daily. Data Storage: - I need the data to be stored permanently in Grafana And have a time series graph Skills and Experience: - Strong expertise in data scraping and web crawling. - Proficiency in programming languages such as Python. - Experience working with APIs and web scraping frameworks. - Familiarity with data storage and database management systems. If you have the skills and experience for this project, please submit your proposal along with e...

Hi Freelancers! Our website is terrible on loading. We have installed the Seraphinite Accelerator plugin but still it is not very fast loading. In our opinion there are a couple of unused plugins which do generate quite some server load. Our configur...mem: 512 MB PHP max message: 128 MB PHP time limit: 240 PHP max vars: 4000 SUHOSIN geïnstalleerd: – MySQL versie: 5.7.41 Max. Upload: 128 MB Current website: 10.8GB Also we experience a lot database connection error. Currently we host with Strato but not on a Pre-installed WP server. We might need to migrate to another server when the current server is not sufficient. Currently we have the crawling by search engines not enabled due to the slow loading issues. We need a WP expert on the server site. Thanks!

...visibility and rankings. Specific Requirements: - Optimize the URL structure of the website to make it more search engine friendly - Implement best practices for URL optimization, including using relevant keywords in URLs - Ensure proper website indexing and crawling by search engines Ideal Skills and Experience: - Proven experience in onsite SEO optimization - In-depth knowledge of URL structure and optimization techniques - Familiarity with meta tags and keywords for SEO purposes - Strong understanding of search engine crawling and indexing processes Restrictions on Keywords: - There are no restrictions on the keywords used for onsite SEO optimization Additional Services: - There are no additional services required to boost the SEO performance of the website If you...

I am looking for a freelancer who can help me prevent crawling of all URL's on GitLab in order to improve website performance. The ideal candidate should have experience in the following: - Knowledge of GitLab and web crawling prevention techniques - Understanding of website performance optimization - Familiarity with authentication methods, specifically username/password - Ability to assess and implement security measures, such as IP whitelisting The project may also involve defining a specific user agent, although I am not sure if this is necessary. Therefore, the freelancer should be open to discussing and determining the need for a specific user agent. If you possess the skills and experience mentioned above, please submit your proposal. Thank you.

We are crawling a cloudflare protected website. We need to make interactions with the website buttons, therefore we have a python code that runs chrome over Selenium. Our bot needs to check the website every 1 minutes, therefore we use 100 http proxies and rotate and shuffle these proxies when giving them to the bots. However, every 1 or 2 weeks, all of our proxies get blocked by the cloudflare systems. Is there any solutions around this problem?

...experienced C# developer to update our existing web crawling tool Ver2. The specific updates needed for the tool are bug fixes. ============== I need to crawl data from my tool is working fine before but they have updated the site so the data structure was changed. That is why I need someone modify to the correct field. No need to add more functions. JUST NEED TO map the correct field for a tool to get the right information. Product ID for testing: 3256804659474341 3256804340980991 Install MongoDB before running the tool. ============== I prefer working with a freelancer for this project. Previous experience with web crawling tools is required. Skills and experience