Python Large File parser using Pandas

We need to parse a large file (20GB-25Gb) what is ';' separated and has from 2-??? columns per row

Message_type is in first column the rest is the message itself nad we need this to be in high performance

We need to generate a new fil for every message_type (24 types) and of that 20Gb of data roughly 3Gb are data with a tag we are interested in.

The logic for the tags needs to be dynamic (if we add a file load_whattoload that needs to be imported dynamically, code for that that is somewhat wokring is provided, but can be changed if needed) We don't wan to have 24 IF IFEL IFEL ELSE statements

Since we are using pandas we the file needs to parsed with pandas, unless there is a really good reason why not to.

Wokring simple code with IF IFEL is provided, with a few tags completed and a small and medium dataset sample aswell

We need this to be done ASAP and needs to be done within 12h after acceptance.

IF you know SQL that is big plus since if you are a dedicated coder, you can keep converting the current SQL based parser functions to Python.

Taidot: Python

Näytä lisää: parse large log file, perl script parse large file, large file parse unix, parse large csv file information need, python binary file parser, php parse large xml file mysql, linux python multi thread process large file, python analyze large file, php parse large xml file, parse large json file java, python text file parser, parse large json file python, php parse large json file, python find string in large file, parse large json file android, parse large json file javascript, python replace string in large file, parse large xml file, python shuffle large file, python split large file into smaller files

Tietoa työnantajasta:
( 4 arvostelua ) Sundsvall, Sweden

Projektin tunnus: #31522138

33 freelanceria on tarjonnut keskimäärin €509 tähän työhön

(174 arvostelua)
(64 arvostelua)

Dear Client Thank you for your project. I've just checked your job description carefully. I'm senior developer with 9+ years of Python,Django and Flask. By using Python, I developed AI engine, BOT, Web Scraping Tools, Lisää

€700 EUR 15 päivässä
(3 arvostelua)

Hello, I've complete MS Software Engineering. I've 5-Year experience in Core Python development. My Core Skills are Machine Learning and AI - Deep Learning - Tensorflow - Pytorch - OpenCV - Deep Learning - Data Analys Lisää

€500 EUR 5 päivässä
(16 arvostelua)

PYTHON developer I have read your job description and I am pretty sure that I can complete every bit of your requirements. Further details and cost can be discussed in chat

€250 EUR 3 päivässä
(36 arvostelua)

⭐⭐⭐⭐⭐ Python Expert ⭐⭐⭐⭐⭐ Dear sir! I am very familiar with data processing using python language. I've checked just your attached project and I am very interested in your project. When do you need the project finished Lisää

€250 EUR 1 päivässä
(24 arvostelua)

Hi, I am interested in this project and I have extensive experience in similar data processing projects. I have 15 years experience and my main area of expertise includes Data Engineering (Small or Big Data level), Da Lisää

€500 EUR 1 päivässä
(5 arvostelua)

Hello there! I will use threads to parse large files with pandas, I am also a pro in SQL. Can we talk I will submit it within 12 hours? Thank you!

€550 EUR 1 päivässä
(21 arvostelua)

Dear client, I hope you are doing well As a professional developer, I have many years experiences of python and SQL I will complete the project within your time -12 hours prfelctly until now I have developed many pytho Lisää

€500 EUR 1 päivässä
(9 arvostelua)

Hello Python is my main skill. I have rich experience in Python Script Development. Let's discuss the project in detail. Thanks Dusko

€500 EUR 7 päivässä
(1 arvostelu)

Hello? How are you? Thanks for posting this job. I am a Senior Python developer with 7+ years. I have been doing this kind of python projects many times. Please let me know. Best Regards

€250 EUR 3 päivässä
(6 arvostelua)

Hi, Dear Employer, I am Al.A. I am a Ph.D. backgrounder professional Research Writer. In my University life, I worked for much research. Every semester needs to research writing to pass the semester. In that course, I Lisää

€300 EUR 4 päivässä
(9 arvostelua)

Hello! I'm a pandas expert! All my scrapping project has a pandas feature on it. I can help you finish this task , with the best performance. Contact me. Thanks.

€500 EUR 7 päivässä
(7 arvostelua)

Hi there! You want to build an Tour Industry application that will incorporate a fleet management system, travel booking, accommodation booking, generate itinerary for tour packages with a gateway payment system.I am c Lisää

€500 EUR 7 päivässä
(1 arvostelu)

Hi there. No problem. I will do my best for you to provide High Quality, Fast Speed, Full Time. I'm a senior Node and Python developer with rich experience of python and MySQL. While reading your requirement, I am sure Lisää

€500 EUR 7 päivässä
(2 arvostelua)

Hi, I have 6+ years of experience in application development. I can parse the doc as you need with Pandas but 12 hr time is very short. At least should be 3 days.

€600 EUR 3 päivässä
(1 arvostelu)

Nice to meet you. I would like to work on python project such as desktop, flask, django. Already experienced in many modules such as selenium, numpy, flaks, django, tensoflow, pyqt, tkinter etc. So can help you in anyw Lisää

€500 EUR 1 päivässä
(2 arvostelua)

Dear Client! I have read your project description. I'm glad to say I can help you. I have a rich experiences for projects like yours. If you hire me, then I will do my best and finish it in time. We can discuss in more Lisää

€500 EUR 7 päivässä
(3 arvostelua)

Hey there ! Good day. I am Pankaj Ranga I would like to work with your project and I can surely provide the details you have specified on the project description. I can confidently say that I am knowledgeable enough to Lisää

€500 EUR 7 päivässä
(1 arvostelu)

Hello, I have been did the same project before and so my capabilities and experiences is the best thing for this project. Let's discuss the detail. Regard.

€500 EUR 7 päivässä
(1 arvostelu)