1. Human Biology and Medicine
Download icon

Meta-Research: Reader engagement with medical content on Wikipedia

  1. Lauren A Maggio  Is a corresponding author
  2. Ryan M Steinberg
  3. Tiziano Piccardi
  4. John M Willinsky
  1. Department of Medicine, Uniformed Services University of the Health Sciences, United States
  2. Lane Medical Library, Stanford University School of Medicine, United States
  3. École Polytechnique Fédérale de Lausanne, Switzerland
  4. Graduate School of Education, Stanford University, United States
Feature Article
Cite this article as: eLife 2020;9:e52426 doi: 10.7554/eLife.52426
3 figures, 6 tables and 1 additional file

Figures

Page length of WikiProject Medicine (WPM) and other Wikipedia (W) pages.

Distribution of page lengths in characters for WikiProject Medicine (WPM) pages and the rest of Wikipedia (W) on April 20th, 2019. The difference between the two distributions is statistically significant according to Mann–Whitney U test (p<0.001, two-tailed).

Open access reference displayed in Wikipedia.

A research study cited on Wikipedia’s “diabetes” page displayed on desktop device with “freely accessible” icon for PubMed Central (PMC) link and on a mobile device without the open access icon. There is an “up click”( ^ ) in front of the citation on the desktop device, which returns reader back to footnote 46 in the text.

Example of information displayed when hovering over footnotes in Wikipedia.

(A) An external link on the Hepatitis page, revealed by “hovering” over the footnote number, indicating that the source is a systematic review conducted by the Cochrane Collaboration (with a green icon PMC link signaling open access to the source). (B) Hovering over a different footnote number on the same page reveals a different external link for the 19th and relatively recent edition of a medical textbook from a leading publisher.

Tables

Table 1
Types of engagement with external links

Different events captured by the CitationUsage Schema that reflect how readers of Wikipedia pages engage with external links.

Engagement typeGeneral description
External clickA click on a link located on a Wikipedia page leading to a web page outside of Wikipedia.
HoverAn event that occurs when a reader hovers over a link for at least 1000 milliseconds on a Wikipedia page.
Footnote clickA click of an internal footnote link – [1] – that takes the reader to the reference section at the bottom of the Wikipedia page.
Up clickA click of an arrow – ^ – that takes the reader from the reference at the bottom of the Wikipedia page back to the citation in the main text.
Table 2
Factors that distinguish WPM pages from the rest of Wikipedia.

Differences between WPM pages and pages in the rest of Wikipedia (W) based on data collected on April 20th, 2019.

WPMW
Wikipedia pages34,3245,839,083
Pages with external links32,6095,210,746
External links945,64560,851,396
Links per page (with links)29.011.7
Page length (characters)13,084.97,676.4
Characters per link (pages with links)450.3657.3
Table 3
Comparing pageviews between WikiProject Medicine (WPM) and other Wikipedia (W) pages.

The average number of pageviews WikiProject Medicine (WPM) and the rest of Wikipedia (W) received per day for different types of devices between March 22nd and April 22nd, 2019.

WPM (%)W (%)
Pageviews on desktop device1,957,821.6 (33.3)97,956,273.1 (42.9)
Pageviews on mobile device3,917,648.8 (66.7)130,488,855.3 (57.1)
Total number of pageviews5,875,470.4 (100)228,445,128.4 (100)
Table 4
Frequency of different types of link engagement per day.

The average number of times per day that readers of WPM and W pages engaged with external links using one of the event types captured from March 22nd to April 22nd, 2019.

Event typeWPMW
 Total (%)Desktop (%)Mobile (%)Total (%)Desktop (%)Mobile (%)
Hover over link48,748.9 (46.9)45,814.8 (60.3)2,934.1 (10.5)1,122,704.0 (32.7)1,057,982.0 (47.2)64,722.0 (5.4)
Footnote click27,739.4 (26.7)10,948.8 (14.4)16,790.6 (60.3)722,131.0 (21.0)235,245.0 (10.5)486,886.7 (40.7)
External click25,811.9 (24.9)17,792.3 (23.4)8,019.7 (28.8)1,557,125.0 (45.3)915,445.1 (40.9)641,676.4 (53.6)
Up click1,539.5 (1.5)1,422.8 (1.9)116.4 (0.4)34,738.0 (1.0)31,230.1 (1.4)3,508.1 (0.3)
All events103,839.7 (100)75,978.7 (100)27,860.8 (100)3,436,698.0 (100)2,239,902.2 (100)1,196,793.2 (100)
Table 5
Number of pageviews per engagement event.

The frequency of each event per day was divided by the average number of daily pageviews for WPM and W pages from March 22nd to April 22nd, 2019. The lower the number of pageviews per event the greater the event frequency. Difference between each pair of WPM and W distributions is statistically significant as derived from Fisher’s exact test (p<0.001, two-tailed).

Pageviews/eventWPMW
 TotalDesktopMobileTotalDesktopMobile
Hover over link120.542.71,335.2203.592.62,016.1
Footnote click211.8178.8233.3316.3416.4268.0
External click227.6110.0488.5146.7107.0203.4
Up click3,816.61,376.133,655.06,576.23,136.637,196.5
All events56.625.8140.666.543.7109.0
Table 6
Summary of data collected.

Summary statistics selected from the tables above, comparing WikiProject Medicine (WPM) pages and readers to the rest of Wikipedia (W) pages and readers. (a) The “4.4 more pageviews” reflects the ratio of WPM to W pages (0.6%) compared to the ratio of WPM to W pageviews/day (2.6%). (b)“WPM readers” and “W readers” refer to the behaviors of those reading a WPM and/or W page during the data collection period.

WPM pagesW pagesWPM pages, compared to other W pages,...
Page length (characters)13,0857676…are 70.5% longer by character count.
Characters/external link (on pages with links)450.3657.3…possess a 31.5% greater external link density.
Pageviews/day5,875,470.40228,445,128.40…receive 4.4 more pageviews per day.a
WPM readersW readersWPM readers, compared to rest of W readers,...b
Time before engagement (sec)47.532.8…take 44.8% longer before engaging in a link activity.
Pageviews/link engagement56.6066.47…engage 17.5% more often with links per pageview.
Pageviews/hover120.5203.5…hover over links 68.9% more often per pageview.
Pageviews/footnote click211.8316.3…click footnote numbers 49.3% more often per pageview.
Pageviews/external click227.6146.7…click external links 35.5% less often per pageview.

Data availability

With the approval and support of the Wikimedia Foundation (WMF), we accessed the data presented in this study via the Wikimedia's Event Logging system and the production MediaWiki database. Due to WMF data policies intended to protect the privacy of readers and editors, all of the data collected and analyzed has had to remain within their protected data environment. Therefore, the data is not publicly available. While this data is not publicly available, data may be available upon request from the WMF research team. To gain access to the data, researchers should review the WMF Research Team's current procedures for data requests (https://www.mediawiki.org/wiki/wikimedia_research/research_and_data). The WMF Research Team will evaluate the request based on Wikimedia's data access criteria. The code utilized to collect and analyze the data, however, is organized and made publicly available in a collated series of Jupyter notebooks at: https://github.com/ryanmax/wiki-citation-usage/ (Steinberg et al., 2019; copy archived at https://github.com/elifesciences-publications/wiki-citation-usage).

Additional files

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)