C code to index large text library and find similar

I need a mini-app (Compiled C on Linux) that groups similar sentences together.

I have 100,000 sentences (say in a PostgresSQL DB, Unicode text). It must perform VERY fast - by indexing each root-word to a 16bit integer (which would reduce its memory foot print), then re-creating a new data structure with sentence delimeters and sentence length. Group into buckets of similar sentence length.

Then iterate through doing word-by-word comparisons (16bit comparisons).

Two algos are acceptable:-

1. Simple - Take a source sentence and iterate through XORing word by word (irrespective of word order or word frequency). If there are more than x words outstanding - then it is NOT a similar sentence. X in this case would be 25% of the number of total words.

We leave such large gap so that we don't need to worry about word roots.

From the smaller data set - we then proceed to do a classic levenstechn comparison - but with an upper bound of x deviation - meaning after it detects more than say 10% deviation - it exists that comparison. Here it is a character by character comparison.

The app should communicate with a folder of .gz files that contain the text and it could use a text boundary to distinguish each sentence.

The output would need to be a new text file that sorts every sentence into groups of similarity - separated by a text boundary.

I need something in 36 hours. A mediocre algorithm is fine.

Taidot: C-ohjelmointi, C# -ohjelmointi, C++ -ohjelmointi, Linux, Python

Näytä lisää: document search tool, docfetcher download, file indexing software open source, c++ 2d graphics library, docfetcher alternative, faiss tutorial, docfetcher review, docfetcher portable, code compare excel files find similar items, sorting large text file, vba code extract email text field, script numbering large text file, large text file viewer, parse large text files java, nutch index large, php code send emails text file, text library, easy code mafia online text game, css code product description text oscommerce, large text flash website

Tietoa työnantajasta:
( 15 arvostelua ) Ultimo, Australia

Projektin tunnus: #17551738

12 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


You can trust my expertise, I can finish in time, thanks a lot! I am very proficient in c and c++. I have 16 years c++ developing experience now, and have worked for more than 7 years. My work is online game developin Lisää

$300 USD 1 päivässä
(140 arvostelua)

Hello, I have read your proposal and it is very interesting for me, because I have very strong skills in Algorithm and Programming in C/C++/C#. This project can be divided into 3 main parts. 1 - Input data from Postg Lisää

$500 USD 3 päivässä
(23 arvostelua)

I'm very interested in your project. I have a lot of experience with c and c++ programming and algorithms on string. I've checked the project in detail and I can finish your project. I'm a full stack developer and I Lisää

$600 USD 10 päivässä
(52 arvostelua)

Hello, I'm c developer with 6+ years of experience and mathematician with a number of publications. Also I'm participant and problem writer of many algorithm competitions (Topcoder, ACM ICPC, etc). Just 2 weeks Lisää

$300 USD 1 päivässä
(23 arvostelua)

Hello, I have more than 6 years experience writing software with Python. I can make a very fast, maintainable script for this in Cython if you are interested? Consider that: 1 - The main slowdown is from cache Lisää

$250 USD 0 päivässä
(5 arvostelua)

We can make a small team for delivering the assignment Can break the work for read, algorithm and output process I hope you will need documentation also Also as tomorrow is Sunday please share the date which is ex Lisää

$833 USD 3 päivässä
(25 arvostelua)

Hi im free so i can do this type of jobs in quick manner as you have 36 hours for the job lets dont waste the time and get it started

$555 USD 10 päivässä
(7 arvostelua)

Hello, I am an experienced algorithm designer and would really like to work on your project. I appreciate how detailed your project description is and have understood every aspect of it. Award me the project and I w Lisää

$250 USD 3 päivässä
(11 arvostelua)

Hi, I have 4 years of experience in C/C++ development in Linux environment. Looking forward for your response to discuss further. Regards, Akram

$250 USD 1 päivässä
(5 arvostelua)

Hi, Hope you doing well sir i read your message in given below i make sure you that i can help you to build mini-app (Compiled C on Linux) that groups similar sentences together. as better as per your given requir Lisää

$555 USD 10 päivässä
(1 arvostelu)

Dear Prospect Hiring Manager. Thank you for giving me a chance to bid on your project. i am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned I have Lisää

$555 USD 10 päivässä
(0 arvostelua)

I have expertise in C/C++ My plan to solve this thing: 1) You give me example of dataset 2) I do rapid prototyping in python and show you approximate result of algorithm execution and timing. 3) If you like i Lisää

$333 USD 2 päivässä
(0 arvostelua)