How to Increase Replicability in Quantitative Psychology?

Editorial

In recent years, psychology research has been facing serious issues about whether some of its research techniques threat the validity of its findings. Indeed, replicability and reproducibility of research findings in psychology and behavioral research have been increasingly questioned [1]. Serious concerns regarding the strategy of claiming conclusive findings based only on p-values have led to what has been called a replication crisis [2]. As reported by Nature [3], the project known as the “Reproducibility Project: Psychology” (the largest replication study) “casts doubt on many published positive results”. As stated in the article, “According to the replicators' qualitative assessments, as previously reported by Nature, only 39 of the 100 replication attempts were successful. (There were 100 completed replication attempts on the 98 papers, as in two cases replication efforts were duplicated by separate teams.)”.

From this perspective, as indicated by the Open Science Collaboration, scientific claims should gain credence by the replicability of their supporting evidence [4]. Scientists collect data, analyze them and make conclusions and claims about specific research topics, therefore, in order to avoid bias and boost the clinical impact, maximizing accuracy and minimizing replicability issues are strongly required.

Over the past few years, there has been growing interest within the scientific community in the use of analytical methods which allow to address the mentioned limitations. In general terms, one reason for lack of replicability is due to the presentation of models developed using all the available participants of the experiment. It is known that using such strategy for model building leads to over-optimistic results. A more accurate estimate of true accuracy that replicates well may be derived using cross-validation or out-of-sample testing. Cross-validation is a statistical method of evaluating and comparing learning algorithms by dividing data into two segments, one used to learn or train a model and the other used to validate the model [5]. In machine learning (ML), an application of artificial intelligence concerned with the development of algorithms able to automatically extract information from the data, the 10- fold cross-validation procedure usually permits maximum replicability. Classification models, built using this technique, result in realistic predictions that usually confirm their performance in replication experiments.

The application of ML allows predictions or decisions at the individual level and such have high translational potential in clinical practice [6]. This is particularly relevant in psychological sciences, where amongst others, the goal is to understand and predict human behavior, make accurate predictions of treatment response, generate a distinction between patients and controls relying on traditional clinical assessments.