Evidence Summary: PubMed is Slightly More Sensitive but More Time Intensive to Search than Ovid MEDLINE

Objective — To compare the results of searching the MEDLINE database through Ovid and the free online version of PubMed administered by the National Library of Medicine for randomized controlled trials on the subject of the drug methotrexate (MTX) for patients suffering from rheumatoid arthritis. Design — Comparative analysis of search results. Setting — Searches conducted by researchers affiliated with Mahidol University in Bangkok, Thailand, and the University of Toronto and the University Health Network in Toronto, Ontario. Subjects — A total of 3966 search results obtained from Ovid MEDLINE and PubMed. Methods — This study employs an Ovid MEDLINE search strategy originally created for a published systematic review that identified randomized controlled trials on MTX and rheumatoid arthritis (Katchamart, Trudeau, Phumethum, & Bombardier, 2009). Two of the authors of the original systematic review (Katchamart and Bombardier) are among the authors of this current study. Appropriate medical subject heading (MeSH) terms and their synonyms were identified for the three main concepts (rheumatoid arthritis, MTX, and randomized controlled trials). The search was performed in Ovid MEDLINE, seeking articles in any language that met the search criteria, from the earliest date covered by MEDLINE to January 2009. Each MeSH or keyword term within a concept was searched separately, and then combined with other like terms using the Boolean operator OR. The searches for the three concepts were finally combined using AND. The Ovid MEDLINE search was then translated for use in PubMed by an information professional. The formatting and terminology used in some of the original Ovid MEDLINE search statements had to be changed so they would work in the new database environment, but the researchers tried to ensure that the two searches were as similar as possible. The translated search was then executed in PubMed. The final results, as well as the number of articles retrieved for each key search concept (rheumatoid arthritis, MTX, and randomized controlled trials), were then compared. The final results were further analyzed for measures of sensitivity, precision, and number needed to read. Sensitivity is calculated by the number of eligible studies found in a database divided by the “total number of eligible studies in the review” multiplied by 100 (Katchamart, Faulkner, Feldman, Tomlinson, & Bombardier, p. 806). Eligible studies were identified using the inclusion/exclusion criteria developed by Katchamart et al. The figure for “total number of eligible studies in the review” is taken from that same study, which forms the “gold standard” for this analysis (Katchamart et al., p. 806). Precision is calculated by dividing the total number of eligible citations from a database by the total number of citations returned by the database for the search multiplied by 100 (Katchamart et al., p. 806). The number needed to read (NNR) formula used by the authors is 1/precision, taken from a study by Bachman, Coray, Estermann, and Ter Riet (2002). Main Results — The PubMed search found more results than Ovid MEDLINE for each of the three key concepts – rheumatoid arthritis, MTX and randomized controlled trials. Once the three concepts were combined, PubMed found 106 more articles than Ovid MEDLINE (2036 vs. 1930). Once the review eligibility criteria were applied to the search results from PubMed, 18 eligible articles were identified, one more article than in Ovid MEDLINE. The authors indicated that the additional article located in PubMed was from a journal that was not yet indexed by MEDLINE at the time the relevant article was published. To determine database sensitivity, these numbers were then divided by 20, the total number of eligible studies located in the Katachamart et al. 2009 review, which employed tools like EMBASE and strategies like hand searching in addition to MEDLINE in order to identify relevant studies. Because of the additional study it located, the sensitivity of PubMed was determined to be slightly higher than Ovid MEDLINE (90% vs. 85%). There was little difference between the two databases in terms of precision and NNR. Precision for Ovid MEDLINE was calculated at 0.881% and at 0.884% for PubMed. The NNR was 114 for Ovid MEDLINE and 113 for PubMed. Conclusion — The authors state that while PubMed had a higher calculated sensitivity than Ovid MEDLINE in the context of this particular search because it contained content not indexed by Ovid MEDLINE that proved to be relevant for this topic, its precision and NNR were almost equal to MEDLINE’s. Some technical limitations of the PubMed interface were experienced by researchers during the study, such as periodic instability and the inability to save and modify searches and their results line by line. These same issues did not arise while using Ovid MEDLINE. The need for a skilled translation of Ovid MEDLINE searches for use in the PubMed interface was also emphasized by the authors, as differences in syntax and formatting that are not properly addressed could impact PubMed’s sensitivity and precision.