Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL

We are a startup focussing on blockchain products.

One of our project details are as below for which we are looking for a team of freelancers with various skill set who can together build the product.

Phase 1 (Duration: 7 days):

IT solution to collect, load and process data from various websites.

Data will be available :

1. Standard APIs of those websites (json format)

2. Websites crawls (by Automation to collect tabular data)

Need to build batch automated processes to collect these data and load in Postgresql/MS-SQL on defined interval of time. This would require setting of PostgresqlMS-SQL server on Ubuntu latest version and further processing to consolidate collected data in desired tables' format. Which requires processing via stored procedures.

Parameters, like, frequency of batch run and hyperlink of the API/Website to crawl should be read from a database table. Solution should be generalised, such that, similar pattern websites are read similarly.


1. Standard APIs (samples):

[login to view URL]

[login to view URL]

[login to view URL]

2. Websites Data Crawl (samples):

[login to view URL]

[login to view URL]

[login to view URL]

Once data is collected directly from these websites, it needs to be processed and collated in target summary tables based upon few business logics.

Phase 2 (duration: 4 weeks) & 3 (duration: 3 weeks):

To be discussed once approaching finish of phase 1.

Skills needed for phase 1:

JSON processing, API processing, Automation to crawl websites and collect data, Postgresql/MS-SQL scripting, database knowledge etc.

We want this project to be worked on PostgreSQL (open source) rather MS-SQL.

Taidot: API, Data Analytics, JSON, PostgreSQL, verkkosivujen suunnittelu

Näytä lisää: data collection websites, collect api data cgi script, data processing websites, automated data extraction, automated web login data extraction, data processing feeds automation, perl automated data mine, google kind api data, build web base app data entry form, javascript extract data website load time, capturing data 2008 websites, find truck load websites, data extraction websites, data entry websites home, automated drop shipping websites, convert data md5 hash mssql, automated data mining information websites, build custom weather api, automated data polling websites, automated forex api

Tietoa työnantajasta:
( 0 arvostelua ) India

Projektin tunnus: #17387188

12 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


This is Anshuman from Atventus India Private Limited. We expertise in Blockchain and BitCoin development. We have lots of experience in setting up direct BitCoin blockchain transactions with all the wallet. features. Lisää

₹150000 INR 20 päivässä
(17 arvostelua)

Dear ukrblockchains, I have experience with these skills. I do have similar experience doing something similar to yours "Build automated data load from websites (automation) & APIs to Postgresql/MS-SQL" therefore I Lisää

₹52500 INR 24 päivässä
(12 arvostelua)

Hi I have gone through your project and I fulfil your all requirements.I have a expertise in MS-SQL and Web development & App development. Have a look on some similar project which you needed- WordPress: www. Lisää

₹62500 INR 20 päivässä
(6 arvostelua)
₹62500 INR 20 päivässä
(21 arvostelua)
₹62500 INR 20 päivässä
(2 arvostelua)

Dear Sir/Madam, Hope you are doing great, We thoroughly went through your requirement and assure you that we can deliver the best version of your it that’s too on time and at competitive price. Let’s discuss w Lisää

₹62500 INR 20 päivässä
(5 arvostelua)

Looking for Mobile App Solutions, we've got you covered. Contact us today, our developers are eager to solve your problems.. We specialize in Hybrid and Native Mobile Apps. Our prices are affordable. Contact us today.

₹62500 INR 20 päivässä
(0 arvostelua)

I can do your job.

₹66666 INR 20 päivässä
(0 arvostelua)

Hello! This appears to be similar to work I did to build data marts hence loading data and writing stored procedures is not an issue

₹62500 INR 20 päivässä
(0 arvostelua)

Hello There, I have more than 6 years experience manual testing and automation testing both. Linkedin : [login to view URL] My skype: [login to view URL] My Website : http://perfectq Lisää

₹62500 INR 20 päivässä
(0 arvostelua)

Hi, I have work experience related to Web Crawling, Data Extraction. Currently Working as Data Science. Have knowledge of Postgress, Distributed System.

₹62500 INR 24 päivässä
(0 arvostelua)

I am a junior data scientist and a software engineer. I can do this within a week. .I am working on such a project currently. So it won't take me long to produce the results.

₹44444 INR 20 päivässä
(0 arvostelua)