Modify my web crawler to not have race condition
$10-30 USD
Maksettu toimituksen yhteydessä
I would like my web crawler script be fixed so that when saving the file into chunks of specified number of lines to not have a race condition.
The race condition occurs when I run multiple instances of the crawler from the shell.
*For example, I need to have the output from the crawler, [login to view URL], [login to view URL], ... files have only 10 lines per file even when accessed simultaneously from other instances of the same script. It now has a problem of each outputted file having over 10 lines per file, and this is not desired.
The crawler needs to be able to make sure the outputted chunks of files are specified number of lines even when running multiple instances and accessing the same file. The actual specified number of the lines per file is 50,000 lines per file.
The variable in [login to view URL]
$lines_per_file determines how many lines per line
I believe the needed knowledge is: race condition, concurrency, forking etc.
Projektin tunnus: #10696959
Tietoa projektista
Myönnetty käyttäjälle:
I can update the code so you can run multiple instances and it will still have correct number of line in the output file. Thanks. Roman
2 freelanceria on tarjonnut keskimäärin $178 tähän työhön
Hi sir, I am scraping expert, I have did too many similar projects, please check my feedback then you will know. Can you tell me more details? then I will provide demo data for you. Thanks, Kimi
