SciWalker Studio powers new metric to assess life sciences papers
The need for alternative evaluation methods that can effectively assess the completeness, reliability and relevance of scientific papers is a pressing topic within the research community. While scientists commonly employ Impact Factor and H-index as the standard evaluative tools, recent research has revealed limitations to these metrics, including the lack of assessment at journal or author level and incomplete reflection of the quality and impact of articles.
A novel metric known as MAATrica presents a fresh solution, according to a paper in the European Journal of Medicinal Chemistry.
An alternative method of analyzing and categorizing text
MAATrica offers a new way to analyze and categorize text, focusing on research documents in life sciences, particularly in medicinal and nutraceutical chemistry.
The authors explain that MAATrica utilizes Dimensions’ SciWalker Studio (SWS) as a semantic search engine. Semantic tools make the process of extracting information more streamlined, enabling the analysis of extensive collections of papers, identification of key patterns or criteria and validation of experimental design robustness. These tools can efficiently process unstructured or semi-structured data, thereby extracting pertinent information with minimal manual intervention.
A complementary addition
MAATrica, as an extension of the SWS platform, is not designed to replace traditional metrics such as the H-Index. Instead, it aims to complement the metrics available to researchers, authors, and reviewers, thereby enhancing the quality of literature assessments.
A 95 percent accuracy when compared with manual results
The paper’s authors assessed MAATrica by collecting a dataset of ∼90,000 research articles from the SciWalker (SW) database. SW and SWS share the same annotation engine and have very similar user interfaces. The key difference lies in SWS, which is a customized software version of SW. SWS enables users to install the software locally on their computers and annotate documents that have not yet been published or included in general literature databases.
To assess the effectiveness of the customized SWS the researchers performed a manual evaluation (ME) on a test set comprising 100 randomly selected published scientific papers. Following the manual evaluation, the same set of scientific articles underwent automated assessment using the customized SWS. They computed the MAATrica value for each article, considering both the outcomes from ME and the performance of the automated SWS assessments.
The results revealed a 95% accuracy when comparing SW MAATrica values with ME results, Furthermore, it matched or surpassed human evaluators, thereby confirming the reliability of the developed ontologies.
‘A significant step forward’
The authors say the development of MAATrica represents ‘a significant step forward’ in the evaluation of research papers within medicinal and nutraceutical chemistry. Unlike traditional metrics that may rely heavily on citation counts or journal impact factors, MAATrica focuses on the content of the papers themselves.
MAATrica also aids in evaluating the methodological robustness of a study assessing the breadth of experimentations exposed. Integrating MAATrica into the SWS platform provides a user-friendly interface that allows for the rapid assessment of papers, empowering researchers to make informed decisions about the pertinence of scientific findings to their work.
Where to try it: Researchers and institutions are invited to engage with MAATrica through the SciWalker platform here.