Suljettu

Optimize Python LDIF Parsing Code

I need to reduce the time demanded to parse 12 ldif files (9 GBytes each file) by 5X, current approach process one file at a time and save the results in different csv files.

Output can be csv files (one file per parsed category) or data loaded in a PostgreSQL database (second option preferred).

Software should be in python and properly commented, but some easier approach can be discussed.

Multi-processing or Multi-threading can be used, current version uses one processor at a time.

I´ve attached the SW, ldif test files will be provided.

Solution must work over windows 7 and 10

Taidot: Python, tietojärjestelmäarkkitehtuuri, Linux, PostgreSQL

Näytä lisää: code project access database, php code pull listings database, php parsing xml feed mysql database, ldif parser python, python ldap example, ldif to json, python parse ldap results, python ldap3, python ldap dump, ldif to csv, ldif parser java, example servlet code midlet code access mysql database, python example parsing complex xml file, code use sqlite3 database thread environment, simple vb6 code restore sql database, php code logging without database, python wordsearch puzzle code, java parsing text file query database, aspx code google map database, php code webpage mysql database

Tietoa työnantajasta:
( 0 arvostelua ) Spain

Projektin tunnus: #21454012

8 freelanceria on tarjonnut keskimäärin 205$ tähän työhön

captek

Greetings from Capanicus! I would certainly help you with this project. If possible, kindly provide us Project Requirements Document so that we can review the functional flow of the web -app and on that basis, we will Lisää

$240 USD 7 päivässä
(6 arvostelua)
6.3
topexpert713

Hi, Nice to meet you! I am very interesting your project and I am confident of I can help your job. I am confident of this project as I'm a professional Python expert with over 7 years of experience. Seems to be an in Lisää

$140 USD 7 päivässä
(26 arvostelua)
5.3
Arahan00

Hi, I have worked with multithreading for parallel csv file processing in Python. I would like to work on your project. Let me know if you want to discuss further. Regards, Monir

$250 USD 14 päivässä
(11 arvostelua)
4.7
Kane96419

Dear Thanks for your posting. My name is Ze S. I have read your proposal and understood what you want to do. I have been working in the team for developing cloud solution since 2012 and have 10+ years development ex Lisää

$140 USD 7 päivässä
(14 arvostelua)
4.3
Valuesolutions

Hello, i have read the details provided..please contact me to discuss more on the project deadline and some other few things

$150 USD 5 päivässä
(11 arvostelua)
4.9
tracygearth

Dear Sir/Madam, I have very good experience in Python parsing and RDBMS with gigabytes of data in legacy system and multi processing programming writing, anuj

$200 USD 5 päivässä
(5 arvostelua)
2.7
sabranny

Hello, I've seen the code and I think i can improve the results (maybe even more than the 5x). Contact me if you're interested. Kind regards.

$370 USD 20 päivässä
(0 arvostelua)
0.0
Bikashconan

Using pyspark can easily speed up the process 5x+, I have used this to reduce a workload from 2 days to 6 hrs.

$150 USD 4 päivässä
(0 arvostelua)
0.0