Käynnissä

Webcrawler to save link relations between websites

i need a coder for an webcrawler that saves the link relations between urls / domains

the crawler need the following features:

- performant for billion saved link relations!!!

- crawl filter: just crawl specific top level domains / websites within an specific country (based on ip country db)

- follow redirect links and saves them too

- respect [url removed, login to view], meta robots and rel attributes

- multiple threads & crawlers (on different machines)

- one mysql database for all crawlers (OR: if you want to use an other database -> api to get the data)

- linux compatible

its ok for me if you use opensource libaries and codes as long as the crawler is fast enough and it doesnt generate too much cpu load

Taidot: .NET, C-ohjelmointi, Java, Linux

Näytä lisää: top java websites, top codes, robots websites, api programm, realtime mysql database replication, websites use java, webcrawler, respect, fast crawler, filter amp, mysql save data, mysql attributes, attributes amp, java crawl, mysql filter amp, api link, mysql link, websites generate, mysql java api, java link, link urls, link database, generate codes, redirect link

About the Employer:
( 3 reviews ) Verden, Germany

Projektin tunnus: #435091

Myönnetty käyttäjälle:

rmvarasu

We do have experienced professionals for java and .net and hence we will deliver quality work. Please refer pmp for more details.

400 $ USD 10 päivässä
(5 arvostelua)
4.4

17 freelanceria on tarjonnut keskimäärin 449 $ tähän työhön

rajtuhin1

_____________________________________ SERIOUS bid here, let us discuss details given by you, and start ASAP. Please see my excellent 46 reviews given by my clients all over the world. I have 8+ years experience. Thanks Lisää

400 $ USD 10 päivässä
(151 arvostelua)
7.3
SigmaVisual

We can help in your project, please check PMB to see our related experience.

400 $ USD 5 päivässä
(36 arvostelua)
6.2
interpb

Hey pls view pmb

350 $ USD 15 päivässä
(54 arvostelua)
6.1
aruhat

Hi, Thanks for giving an opportunity to place a BID.Further kindly go through PM. Regards, Bhavik

450 $ USD 7 päivässä
(6 arvostelua)
5.9
andreiandrei

Hi,please check PM.

600 $ USD 15 päivässä
(3 arvostelua)
5.0
avishekcode

Hello! We have done this kind of Web Crawling development before. Also we are very keen on customer satisfaction. So you are getting good experienced developers at a fantastic bargain !!! Please check PMB. Lisää

400 $ USD 15 päivässä
(4 arvostelua)
4.2
huupq

Can be done well for you, Thanks and regards!

400 $ USD 15 päivässä
(4 arvostelua)
4.4
logicalxpression

hi, i am interested, we can discuss further details.

550 $ USD 30 päivässä
(2 arvostelua)
3.9
phpXpertbd

Please check PM. Thanks

450 $ USD 12 päivässä
(2 arvostelua)
2.8
FrozenEye

I already have a framework that does what you need. But I have questions before accepting this task. -"link relations": what does this mean ? -just crawl specific top level domains / websites within an specific Lisää

500 $ USD 20 päivässä
(2 arvostelua)
2.4
silv3rm00n

can do this for you.thanks

750 $ USD 20 päivässä
(2 arvostelua)
2.0
icybrain

Hello, please allow me to do the job.

280 $ USD 20 päivässä
(2 arvostelua)
1.6
vianvi001

Dear friend pls kindly view pmb for more details

400 $ USD 12 päivässä
(0 arvostelua)
0.0
matrixn

Please check your PMB! Thanks.

700 $ USD 20 päivässä
(0 arvostelua)
0.0
esantanu

I am an experienced Java professional having a total of 6+ years of working experience with various projects and technologies. I hold a Bachelor’s degree in Electrical Engineering. I have worked in Product Developmen Lisää

300 $ USD 15 päivässä
(0 arvostelua)
0.0
movvawork

I have used the open source library used developed buy CMU. It was slow but I tweaked it a bit before. I could continue that work again.

300 $ USD 5 päivässä
(0 arvostelua)
0.0