I have a Kinesis Firehose which dumps JSON data into S3 bucket. My requirement is to read each record of JSON Data and do validation checks
1) Data Type Check
2) Format Check
3) Date format check
4) Required Fields Check.
5) Valid format check
Please mind that the out of kinesis firehose dumps data in a single line with no comma separators i have added the firehose output for your reference.
Once all these checks have been done the files should be sent to success S3 bucket and in case of any json not complying it should go to error bucket along with a log file mentioning the errors in the json. I want the code to written in python which should seamlessly work on aws environment. i am uploading the Input File format and the json schema which shows how the json data should look like based on which the validation rules can be done.
People who can make the code work in aws environment are welcome to bid on this project. any questions please message me.