Log collector shell script

The assignment is to create a log collector shell script that reads csv-formatted log events from stdin and publishes them in json format to a remote server via HTTP requests.

The functional requirements are as follows:

-The script is launched by a daemon process which constantly writes event records to stdin of the script.

-The script buffers records until either the buffer size exceeds x records or x seconds has elapsed since first record in the buffer.

-When the buffer is flushed, the records are sent to remote server via HTTP POST request (using curl)

-Input is in csv format (i.e value1,value2,value3 ...). Some fields are transformed using lookup maps and then mapped to a json record (i.e {"field1": "transformedValue1", "field2": "value2", "field3": "transformedValue3"...}). Payload of the POST request is a json array of the records.

-If record mapping fails (i.e field value not found in lookup map when transforming a value), the record is ignored.

-If flushing fails due to network error, the script should keep retrying until it succeeds.

-If flushing fails due to request error (HTTP status other than 200), the script should write error reason to stderr and exit.

-The script should not continue reading from stdin until the buffer is flushed succesfully

-The script accepts remote server address and an authentication token (passed as a header with the POST requests) via command line arguments. Other configurations are defined as constants in the beginning of the script.

Taidot: Shell Script

Näytä lisää: shell script remove line beginning, shell script delete lines beginning file, shell script monitor oracle application server log, aws linux logs, linux log analyzer, ecs logs to cloudwatch, ecs log collector, logs from ecs task, ecr logs, ecs logs, ecs log location, unix shell script alert log file, linux shell script log parsing, shell script delete log, ldap log rotation using shell script, shell script add line beginning file, shell script scan log files, use shell script create csv file log file, shell script remove lines beginning, shell script update write log file results

Tietoa työnantajasta:
( 1 arvostelu ) Tallinn, Estonia

Projektin tunnus: #17867449

Myönnetty käyttäjälle:


I am master in Bash, and have worked with everything you mentioned in your description - csv, json, POST-ing, etc.

€60 EUR 1 päivässä
(155 Arvostelua)

8 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


Hello, Not sure if I understood your requirement fully. The base requirement is clear like you need to collect data from a csv and push into a remote http server. That can be easily done - by the way do you have endpo Lisää

€111 EUR 4 päivässä
(14 arvostelua)

hi Can i develope it in php-cgi? It is simple - I can use #!/usr/bin/php or something similar in the start of script and chmod 7 to file - and it will work as shell script - it will be simple for me about some ideas Lisää

€170 EUR 2 päivässä
(1 arvostelu)

Hello! I can help you to develop a log collector shell script according to your requirements. As for me I'm a professional system admin with 10+ years of experience. Check my reviews. Recently I've successfully c Lisää

€155 EUR 3 päivässä
(2 arvostelua)

Hello, sir Your job is exciting me. Let me do this job for you now just to prove my expertise in this field. Hope I could awake you interest in me. I am always online, work hard and very sensitive to details. Thank Lisää

€155 EUR 3 päivässä
(0 arvostelua)

Hello I am working in scripting technologies for 20 years. I worked in various shells c, korn, bash in various Unix system. Can you please share the URL and sample log file to parse and output format etc? Thanks!

€220 EUR 3 päivässä
(0 arvostelua)

I can do it .r

€155 EUR 3 päivässä
(0 arvostelua)

Hi I am a senior C++ developer who has been programming on Linux systems for more than 8 years. I have been dealing with shell and python scripts during the same period of time because of the nature of my job. When we Lisää

€133 EUR 2 päivässä
(0 arvostelua)