I got 3 tables as attached. All automatic metrics (BLEU, NIST, TER, METEOR, EBLEU, RIBES) are from 0 to 100 only NIST is from 0 to 15 and TER - the lower the better.
In this tables rows are test samples. I asked 10 people to do some task. They had to do transcription from text that was read for them. The same was done using automatic computer system. In excel I have 3 sheets, Sheet 1 is comparison between original text and work done by humans, Sheet 2 has comparison between original text and automatic system work. Sheet 3 has comparison between human work and automatic tool - the better scores the more similar.
NER is evaluation done by humans and REDUCTION is how much humans made text shorter. Best results are obtained when NER and REDUCTION are higher. But is not that easy because sometimes when NER is lower REDUCTION may be higher so those must be somehow averaged.
What I need is some analysis of how the automatic work correlates with the one done by humans, I need information if automatic work correlates with work done by humans, some information which metric is most reliable - sometimes for example BLEU decreases when EBLEU rises.
For all of those I would need significance tests.
Most importantly I need comparison of each metric to (NER and REDUCTION) so to human work, I need knowledge which metric mostly correlates with human judgements and if some will correlate I would need significance and confidence for this measurements. Also I would need what levels of those metrics would be equivalent to NER and REDUCTION values.
I would also need a very detailed description how analysis was conducted and a very detailed explanation of obtained analysis results.
Finally I would need some visualization for example using some diagrams - the results can be normalized for easier understanding.
In some time I will add a 1-3 columns with other metrics and I will need to add comparison. So the methods you choose should be easy to upgrade results when I got new data.
19 freelanceria on tarjonnut keskimäärin 378 $ tähän työhön
Hello, am sure i can help you in completing your analysis on statistical data using [url removed, login to view] message me so as to discuss further on the [url removed, login to view]
Hi. I have good experience with data analytics and statistical testings.I am sure we can work together for your project. Feel free to contact me when you wish to discuss further. Thanks.