Scientists are becoming increasingly concerned that a lack of reproducibility in research could lead to flaws that hinder scientific productivity and erode public trust in science, among other things. A group of researchers now claims that constructing a prediction market, in which Artificially Intelligent (AI) agents make predictions on hypothetical replication studies, might lead to an explainable, scalable method of estimating confidence in published scholarly work.
Experiments and research that are replicated, which is an important step in the scientific process, assist establish confidence in the results and showing whether they can be generalised across contexts. Scientists increasingly lack the resources for effective replication efforts as studies have become more sophisticated, costly, and time-consuming – what they now refer to as the replication problem.
As scientists, we want to do work, and we want to know that our work is good. Our approach to helping address the replication crisis is to use AI to help predict whether a finding would replicate if repeated and why.
− Sarah Rajtmajer, Assistant Professor, Information Sciences and Technology, Penn State
Based on trading patterns and elements of the papers and claims that drove the bots’ behaviour, a bot-based approach scales and offers some explainability of its findings. Bots are trained to recognise key features in academic research papers, such as authors and institutions, statistics, and linguistic cues, downstream mentions, and similar studies in the literature, and then assess the confidence that the study is robust enough to replicate in future studies, according to the team’s approach. The bot then bids on its level of confidence, just like a person betting on the outcomes of a sporting event. The outcomes of the AI-powered bots are compared to human predictions.
While human-based prediction markets are well-known and have been successfully applied in a variety of industries, prediction markets are innovative when it comes to analysing research data. Prediction markets are a skill that humans have mastered. They are, however, using bots for our market, which is unique.
The system offered confidence scores for about 68 of the 192 publications that were subsequently duplicated, or ground truth replication studies, according to the researchers, who presented their findings at a recent meeting of the Association for the Advancement of Artificial Intelligence. The accuracy on the set of papers was roughly 90%.
Because humans are better at predicting research reproducibility than bots, the researchers propose that combining the two approaches — humans and bots working together — could give the best of both worlds: a system with more accuracy while still being scalable.
They train the bots in front of human traders on a regular basis, then deploy them offline when we need a speedy result or when replication efforts at scale are required. They can also construct bot markets that take advantage of such intangible human wisdom.
As reported by OpenGov Asia, according to experts at the University of California, Irvine, producing smarter, more accurate systems necessitates a mixed human-machine approach. They describe a new mathematical model that can increase performance by integrating human and computational predictions and confidence scores in a paper published this month in Proceedings of the National Academy of Sciences.
To put the framework to the test, researchers ran an image classification experiment in which human volunteers and computer algorithms competed to accurately identify altered images of animals and everyday objects like chairs, bottles, bicycles, and trucks. The human participants assessed their level of confidence in each image’s correctness as low, medium, or high, while the machine classifier generated a continuous score. Across photos, the results revealed significant variations in confidence between people and AI computers.
