I am needing a custom script written for BrownRecluse.
I am looking to have some data mining done for my own
site, pdmautoparts.com. I will use it essentially to create a product data feed. Now a majority of the data is generated dynamically and I would like to be able to pull all this info down. For example, I need all the year, make, model, category, price, and etc info to be pulled down and then be exported to a CSV file. The data to be retrieved can be found by starting the drill down process here: [url removed, login to view] .
As you can see the data is pulled dynamically but I only need to the actual text to be crawled, i.e. 2005 BMW 325i Sedan (Engine:M54; Chassis:E46;) then all the part categories, etc for each vehicle make.
This can be done as I have seen it before. If you are unfamiliar with BrownRecluse, it is a spider that is 100% programmable. I have attached the installer for the program and there is also a small synopsis of how it can be coded/written, SEE BELOW.
The language was designed for Internet spidering programs. This means, you do not need to worry about making functions to access the internet, pull web pages or files etc. The language already include everything you need. For example, if you wish to pull a web page and display its source code in the output window, you only need 4 statements. One to create a URL object, one to assign the actual URL to pull, one to get the data, and finally, one to display the data. Here's how to do it...
Link = New(URL);
[url removed, login to view] = '[url removed, login to view]';
[url removed, login to view];
Output([url removed, login to view]);
So you see, it is very easy. You do not need to know how to program sockets and protocols.
if [url removed, login to view] then
output([url removed, login to view])
Output('There was an error.');
There is no need for the conventional endif closing statement. In short, white spaces (spaces, tabs, returns) are ignored by the compiler, so you can spread your code over as many lines as you wish.
SBL uses both, Basic like and C++ like assignments. For example...
x++; is the same as x=x+1;
x+=y; is the same as x=x+y; "
For an experienced programmer, this should ideally take only a few hours.
Please post any questions, if there are any.
Additional Info (Added 4/6/2009 at 9:11 EST)...
*** Important ***
I will need the crawler/script to retrieve Year, Make, Model, Category, Subcategory, and Product and then have it exported to a CSV file separating into its own columns.
If you cannot work with BrownRecluse, I am not opposed to having a custom crawler/scraper made as long as we can stay close to our budget.
You can also download the app here:
[url removed, login to view]