Introduction

The rapid expansion of academic publishing challenges the established systems of quality assurance for the world’s academic literature. The past two decades have seen major changes across the academic publishing industry in consequence of the internet revolution, which have also transformed the economics of publishing. The Open Access movement has brought about a surge in journal titles of widely varying quality and ethical reliability. Authors and readers are tested to find the most reliable and quality assured journals in a very complex information ecosystem.

Trust in the academic publication process is particularly important in the fields of Medicine and Healthcare. Publication malpractice at the article and journal level can have serious impacts upon human health and welfare on an individual and on a global scale. That trust is partially underwritten by the conscientious quality assurance work of professional teams in the global bibliometric cataloguing systems, for which MEDLINE and Scopus provide transparent and public explanations of their evaluation strategies. Web of Science runs a similar system but historically with anonymised and internally employed specialists.

The MEDLINE and Scopus advisory board take on a public responsibility for the oversight of quality assurance in the academic healthcare literature, which in turn requires credible technical systems for accession and re-evaluation of that content:

• MEDLINE is a global publicly funded open access portal to large numbers of articles in the Medicine and Healthcare subject fields.
• Scopus is a large commercially supported citation system behind a paywall which provides similarly assured content for authors, publishers, Universities, Corporations and Governments, along with academic books, book series and curated conference proceedings.

Both MEDLINE and Scopus use independent expert evaluation processes to select journals for inclusion in their systems. Both publish lists of such journals for review and research. Scopus has a formal long standing arrangement with MEDLINE to receive,  process and republish MEDLINE content alongside its much broader corpus of journals and articles from other disciplines.

An introduction to the MEDLINE journal collection

MEDLINE continuously reviews suggested healthcare journals and their content for incorporation and listing in the MEDLINE database. It listed some 5200 such journals in some 40 languages as of July 2022.  MEDLINE is also the primary component of PubMed, which is maintained by the US National Centre for Biotechnology Information (NCBI) on behalf of the US National Library of Medicine (NLM). Some MEDLINE citations in PubMed link to the free full-text of the relevant article in PubMed Central.

Journals are proposed for acceptance into MEDLINE by applicants, who may be publishers or editors. The recommendation to accept or reject a journal into the MEDLINE collection is then made by members of the US MEDLINE Literature Selection Technical Review Committee (LSTRC), which in turn is affiliated to the National Institutes of Health (NIH). The LSTRC includes researchers, physicians  and medical librarians, who review the journal content for scientific and editorial quality.  The precise criteria for acceptance are not clear on the public websites.

Content Classification in MEDLINE

Once accepted for MEDLINE, the accepted journals are processed at the article level (1). Articles are enriched with metadata, listed and made searchable. MEDLINE uses the Medical Subject Headings (MeSH) vocabulary (2, 3) to classify journal content at the article level. This thesaurus is a controlled and hierarchical vocabulary which is produced by the NLM. It is used for indexing and searching of biomedical and health-related information.

MeSH includes the article subject headings which appear in MEDLINE/PubMed, the NLM Catalogue, and in other NLM databases. These were originally used in the Index Medicus, which contained 4,400 headings in 1960. They have been used and updated over more than 60 years, such that the 2010 index contained more than 25,000 subject headings. These are organised into an eleven-level hierarchy, with more than 80 subheadings.

It is important to note that MeSH terms are assigned primarily to individual articles, and not journals. The question “How are journals classified in MEDLINE?” yielded no returns on a Google Search for this paper.

An introduction to the Scopus healthcare journal collection

In comparison to MEDLINE, Scopus comprises a large multidisciplinary database of more than 26,000 active journals across all STEAM, Arts and Humanities Journals, of which some 8000 address Medicine and Health Sciences related subjects (4).

Scopus originally agreed an acquisition policy MEDLINE-sourced journals in 2004 for the original Scopus core collection, without further direct evaluation through the Scopus Title Evaluation Platform (STEP). However, All SCOPUS content is regularly scanned by the “SCOPUS Radar” system which seeks out statistically aberrant performance data in a range of Key Performance Indicators.

MEDLINE content is regularly supplied in updated feeds to the Scopus data systems, where its recommendations are processed with full institutional affiliation data and cited references for all articles. The MEDLINE article records are thus merged into Scopus as one item with their indexed data (MEDLINE keywords, PubMed ID) combined.

Scopus also contracted in 2008 with the NLM to cover OLDMEDLINE content that was published between 1949 and 1965. Additionally, Scopus seeks to maintain a collegiate contractual relationship with all of the publishers of MEDLINE sourced content, and some journals are sourced directly from publishers as well as from MEDLINE.

Publishers rarely decline permission for the republishing in Scopus of MEDLINE derived  journal information, and when they do so for whatever reason their wishes are respected. Efforts continue to harmonise contractual agreements with all parties to optimise the consistency of the MEDLINE and Scopus data holdings.

Some journals with Medline listings are also submitted directly to STEP for evaluation. This provides those journals with the additional value of a formal evaluation and feedback on options for further journal improvement.

The Allocation of Journal Categories to MEDLINE Sourced Journals.

MEDLINE sourced content is not provided with a Journal classification. Submitted and MEDLINE sourced Journals are therefore further classified on accession to SCOPUS by an in house coding team of taxonomy officers, including former librarians, using the proprietary Elsevier All Sciences Journal Classification (AJSC).

Journals are thus allocated the most relevant and  logical subject heading in the AJSC in the relevant Medicine, Healthcare and related subject fields, where they are used within Scopus and by commercial partners such as the SCIMAGO Journal and Country Ranking system.

Decision Making on SCOPUS Titles

As in MEDLINE, the primary unit of currency for evaluation purposes in SCOPUS is the Journal, from which the large Scopus holdings of individual articles are abstracted and enriched with metadata.  All newly proposed journals to SCOPUS are processed by the STEP administrative team for assessment and validation by external and stipendiary members (Subject Chairs) of the Scopus Content Selection and Advisory Board (CSAB)  through the proprietary Scopus Title Evaluation Platform, STEP (5).

There is currently no formal reciprocal system through which MEDLINE takes Scopus listed “MEDLINE-naive” journals for parallel evaluation.  Therefore, while there is a general alignment in the content between the two systems, there are also significant differences in the processing pipelines, in that Scopus adds additional proprietary data, and Scopus enriches the journal data set with the All Sciences Journal Classification (ASJC) process.

During a recent review of policies on journal holdings, we recognised the opportunity provided by the MEDLINE-to-Scopus journal processing sequence to examine directly a cohort of MEDLINE sourced journals for the alignment of the MEDLINE acceptance process with our own processes.

METHODS

The Scopus databases for the journal holdings in the relevant subject fields in the MEDLINE data feed and the Scopus Core Collection were searched and matched. 114 journals were found in the MEDLINE collection which had not yet been incorporated in the Scopus Core Collection were each reviewed according to standard Scopus title selection criteria  by two subject matter experts (DAR and ML).

We therefore undertook a preliminary comparison of the MEDLINE and Scopus journal holdings, matched by subject classification, in July 2022, using data that was downloaded into a spreadsheet.

We were seeking any mismatch in holdings between the two systems and for explanations as to why any such mismatch might exist, if discovered.

The list of “mismatches” was presented as an Excel sheet to each of the CSAB Subject Chairs (DAR and ML), who reviewed the key information, the website, the presentation and representative content of each journal, as is consistent with the process within STEP, according to the general criteria in the STEP process, as set out in Table 1.

Table 1: The established criteria for evaluating a journal through the SCOPUS STEP Process

The key information included the Source/Journal ID which is used in Scopus; the Source title; the Print and E- ISSN numbers: the Publisher imprint and the Publisher name; and web links to the journal web home page; a link to the journal profile page on Scopus.com, and in cases where there was a STEP review history of the journal: STEP ID, status, final decision, date of decision, date of embargo expires and notes.

Based upon our adjudications, there were three Decision options:

• To accept the Accept for full Scopus coverage without further consideration
• To Reject the Journal for Scopus coverage, with reasons, or
• To undertake a more detailed review through STEP.

A number of journals with weak metrics data or suboptimal web sites were recommended for review through the STEP process to provide the opportunity for constructive feedback on possible improvements to long term sustainability and performance.

To close the audit and feedback loop, the data set was revisited in  January 2025 to assess the impact of the recommendation.

Results

From the ~5200 MEDLINE sourced journals that we identified, we generated a list of 114 MEDLINE listed journals from 96 different publishers which had not been previously selected or listed for Scopus coverage.

Following a review of the journal data and their websites for each of the 114 journals from MEDLINE which had not previously been listed in Scopus by the two Scopus Subject Chairs, the following decisions were made:

75 journals were accepted for Scopus coverage without further review,

34 journals were referred for further assessment through the STEP system.

Three journals had ceased publication.

Two journals required further information from the publisher to permit a decision.

On review in January 2025, of the cohort of 39 journals for which further assessment had been deemed advisable, all journals were accepted for SCOPUS except for eight journals which had seemingly ceased publication between 2022 and 2025.

Separately, 17 Medline listed journals which had previously been declined for accession following separate application to SCOPUS through the STEP advisory process between 2010 and 2018 were found to have improved sufficiently to justify inclusion in STEP in 2022.

Four journals from the study cohort had been discontinued from a SCOPUS listing by January 2025 for as yet unpublished reasons following a quality assurance review.

DISCUSSION

Observations on the findings of this study

This study has provided reassurance on the alignment of the independent  journal evaluation processes between the MEDLINE and Scopus in the accession of quality assured journals in the Medicine and Health Sciences categories, as defined by the All Sciences Journal Classification. Importantly, despite their differing business and evaluation models, both organisations are cooperatively focussed on journal-level quality assurance of the academic publication process and its outputs, given that publication malpractice can have dangerous consequences for human health and wellbeing.

Bibliometric databases are dynamic entities, which accrue new journals and which shed discontinued or discredited titles on a continuous basis. Academic publishing is in a state of flux, with more than 5000 publishers with varying commercial models and with some of questionable probity. Journals are taken over, or fail, or are lost to the global data systems when their digital signatures change.

Moreover, the unique SCOPUS Radar and Re-evaluation tools are in continuous use actively to filter out the small number of journals which “go bad” for a variety of reasons. There is unlikely to be a point at which two generally aligned but separately managed systems will be in precise alignment.

We also note that there are slight differences in the article classification systems used in MEDLINE and Scopus, which may create discrepancies when the data from MEDLINE, from the individual journal publishers, and from Scopus itself are “merged”. Overall, some 50% of MEDLINE-originated journals continue to have MEDLINE-unique content (including MeSH headings and the PubMed Identifier –PMID), while the other ~50% also possess Scopus-enriched metadata.

We have not yet sought to align, reciprocate or to automate journal accession between Scopus and MEDLINE. This might be a valuable further step towards the wider standardisation of journal evaluation between all trusted citation systems. However, we recognise that there may be complexities, policies, commercial and operational considerations on both sides which we have not yet fully understood.

Nevertheless, this project provides confidence that there are seemingly no major impediments so such an evolution, given that the quality assurance processes are similar in both MEDLINE and Scopus, and as evidenced in similar judgements of quality. We have yet to consider the arrangements that would be needed to maintain equivalence if Scopus were to discontinue to title on the basis of received evidence of impropriety, but where the MEDLINE team did not chose to do so, or vice versa.

Machine-based Journal Evaluations

We may consider whether machine based evaluations of journal quality will supersede human judgement, training, practice and experience, in the evaluation of low quality and even fraudulent academic journals, and in the setting of consistent and defensible quality assessments.

For the foreseeable future, and for all its imperfections, the trained and experienced subject expert, using a structured journal evaluation system and working in tandem with an alert community of public spirited readers, reviewers and observers, seems likely to secure the best attainable quality assurance for the global academic community. However, machine intelligence is evolving rapidly, and we will keep this matter under close review.

Insights from this study into Journal Classification

This study has prompted our reflection into the present state of the detailed and accurate Classification of Medicine and Healthcare Journals.  There is presently much discussion about moving from journal based to article-based and “AI enhanced” classifications according to the specific content of each and every article, rather than to the stated aims and purpose of the journal in which the article appears.

Nevertheless, a contemporaneous, granular and standardised Journal Classification system with applied across across platforms which include MEDLINE and Scopus might still  offer considerable long term advantages in a range of user cases for authors;  for publishers; for healthcare institutions and organisations; and for the general reader of academic journals.

Our audit exercise has also surfaced small inconsistencies in factors such as duplication of a journal title when it is submitted through multiple sources in different formats, and  where publishers apply different permissions to different systems. These issues are under review.

In conclusion

We consider that this study has been a useful point audit of the continually evolving MEDLINE –derived  Journal holdings in Scopus. This study in turn has highlighted the mutuality and interdependency of public and commercial sector systems in the complex and important ecosystem of accreditation and assurance of academic healthcare journals, where openness, public oversight and consistency is very important.

Acknowledgements

We thank Mr Wim Meester and Mrs Tracy Chen of Elsevier BV for performing the initial searches of the Scopus data systems and for making available the data available to us.  We are grateful to Mr Robert Schrauwen, Scopus Data Lead for Elsevier BV, for review and advice on the draft manuscript.

DAR and ML are appointed external members of the SCOPUS Content Selection Advisory Board, for which they receive annual stipends from Elsevier BV, owners of SCOPUS

References

1. MEDLINE Indexing Method

https://www.nlm.nih.gov/pubs/techbull/ja18/ja18_indexing_method.html
and
https://www.nlm.nih.gov/medline/medline_journal_selection.html

2. Sarkar IN. Methods in Biomedical Informatics: A Pragmatic Approach (2014)
ISBN 978-0-12-401678-1; Academic Press

3. MeSH headings: NLM
https://www.nlm.nih.gov/mesh/meshhome.html accessed 6th August 2023

4. Scopus content Coverage Guide (Elsevier) March 2023
https://blog.scopus.com/posts/the-scopus-content-coverage-guide-a-complete-overview-of-the-content-coverage-in-scopus-and

5. SCOPUS Content Selection
https://www.elsevier.com/solutions/scopus/how-scopus-works/content/scopus-content-selection-and-advisory-board