Transparency of research practices in cardiovascular literature
- Western Kentucky University, United States
- Stanford Cardiovascular Institute, United States
- Cornell University, United States
- University of California, Los Angeles, United States
- University of California, Davis, United States
Figures
Figure 1
Distribution summary of all screened publications.
Summary of all screened publications on their (A) study type, (B) study type clarity, (C) journal in which it was published, (D) presence of funding source statement, (E) presence of conflict of interest (COI) statement, and (F) open access status.
Figure 2
Summary of screened papers for pre-registration, protocol, material, data, and analysis script availability.
Only study types considered empirical research in which materials are theoretically available were summarized. (A) Summary of the presence and accessibility of pre-registrations and protocols and the summary of components (hypotheses, methods, and analysis plan) of pre-registration and protocol for papers that had a pre-registration or protocol statement. N/A represents papers that could not answer the criteria because data, materials, and analysis plans were not available to begin with (top panel). (B) Summary of papers and material, data, and analysis script availability. For papers that had a statement, how materials, data, and analysis script were available, whether they were accessible, and whether it was clearly documented or present in its entirety are also summarized.
Figure 3
Summary of potential replicability and reproducibility of empirical research studies (n = 393 studies).
An article is considered ‘partially replicable’ if any of material availability, analysis script availability, and methods criteria are satisfied and ‘fully replicable’ if all three criteria are satisfied. An article is considered ‘partially reproducible’ if any of data availability, analysis script availability, and methods are satisfied and ‘fully reproducible’ if all three criteria are satisfied. Note that these data describe the potential for a study to be partially or fully reproduced or replicated based on the availability of the study’s resources (methods, materials, data, and code), not whether the study was itself replicated or reproduced.
Figure 4
Accessibility score distribution accross the dataset, by study type and by journal type.
(A) Accessibility score distribution across the entire dataset. The number of articles represented by each bar falls within the specified accessibility score range. No articles obtained an accessibility score fraction greater than 0.60. (B) Average accessibility scores across all articles screened, based on study type. Error bars correspond to standard deviation of the collected data points. Sample sizes are noted at the base of each bar. All scores are based on a total possible score of 1. (C) Average accessibility score by journal type. Screened articles were obtained from three different journals, including Journals of the American College of Cardiology (JACC), European Heart Journal from the European Society of Cardiology, and American Heart Association (AHA) Circulation. The standard deviation bars are indicative of the range of distribution of the obtained accessibility score fractions for each journal. Sample sizes are noted at the base of each bar. All score fractions are based out of a total possible score of 1.
Figure 5
Proportion of papers with presence (yes) or absence (no) of specific accessibility criteria (Material, Methods, Data, and Analysis Code) for specific study types.
Methods presence is determined by presence of a pre-registration or linked protocol. Presence is indicated in green, absence in yellow. For each value, both the percentage and the count for that category and study type are shown.
Appendix 1—figure 1
PRISMA 2020 flow diagram showing the inclusion and exclusion criteria for our screening process (Page et al., 2021).
*Consider, if feasible to do so, reporting the number of records identified from each database or register searched (rather than the total number across all databases/registers). **If automation tools were used, indicate how many records were excluded by a human and how many were excluded by automation tools.
Tables
Table 1
Criteria determining accessibility score and criteria-satisfying responses.
| Criteria screened | Criteria-satisfying responses |
|---|---|
| How clearly stated was the study type? | Stated (e.g. editorial comment, clinical trial) |
| Does the article state whether or not materials are available? | Yes the statement says that the materials (or some of the materials are available).* |
| Can you access, download, and open the materials files? | Yes.* |
| Does the article state whether or not data are available? | Yes – the statement says that the data (or some of the data) are available. |
| Can you access, download, and open the data files? | Yes. |
| Are the data files clearly documented? | Yes. |
| Do the data files appear to contain all of the raw data necessary to reproduce the reported findings? | Yes. |
| Does the article state whether or not analysis scripts are available? | Yes – the statement says that the analysis scripts (or some of the analysis scripts) are available. |
| Can you access, download, and open the analysis files? | Yes. |
| Does the article state whether or not the study (or some aspect of the study) was pre-registered? | Yes – the statement says that there was a pre-registration. |
| Can you access and open the pre-registration? | Yes. |
| What aspects of the study appear to be pre-registered? (select all that apply) | Hypotheses, Methods, AND Analysis Plan all available. |
| Does the article link to an accessible protocol? | Yes. |
| What aspects of the study appear to be included in the protocol? (select all that apply) | Hypotheses, Methods, AND Analysis Plan all available. |
| Does the article include a statement indicating whether there were any conflicts of interest? | Any of the following can be selected: Yes – the statement says that there are one or more conflicts of interest. Yes – the statement says that there is no conflict of interest. |
| Does the article include a statement indicating whether there were funding sources? | Any of the following can be selected: Yes – the statement says that there was funding from a private organization. Yes – the statement says that there was funding from a public organization. Yes – the statement says that there was funding from both public and private organizations. Yes – the statement says that no funding was provided. |
| Is the article open access? | Any of the following can be selected: Yes – found via Open Access Button. Yes – found via other means. |
-
*
This criterion was not included for publications for which it was not possible to share materials.
Table 2
p-values of chi-square tests of study type for each category of resource.
We tested for a significant relationship between study type and availability of resources. ‘Yes’ refers to a paper including the resource and ‘No’ refers to a paper not including the resource. Observed values were compared to expected values for each category of resource.
| Yes | No | Study type chi-square p-value with Bonferroni correction | |
|---|---|---|---|
| Materials | 46 | 308 | 3.05E−13 |
| Methods | 107 | 247 | 4.91E−35 (case studies omitted) |
| Data | 107 | 247 | 2.31E−12 |
| Analysis | 36 | 318 | 1.58E−12 |
Table 3
p-values for chi-squared tests and Fisher’s exact tests comparing the presence or absence of specific accessibility categories (Materials, Methods, Data, and Analysis Code) for each study type compared to every other study type (Clinical Case Study or Series, Clinical Observational Study, Clinical Trial, and Laboratory Animal Study).
p-values significant at an alpha level of 0.05 with Bonferroni correction are shown in bold. Use of Bonferroni correction to adjust the p-values for multiple comparisons resulted in some p-values being greater than 1. Given that a probability greater than 100% cannot occur, any values greater than 1 were adjusted to a ceiling value of 1.
| Materials | Methods | Data | Analysis Code | |
|---|---|---|---|---|
| Clinical Case Study or Series vs Clinical Observational Study | 1.0 | 1.0 | 1.0 | 1.0 |
| Clinical Case Study or Series vs Clinical Trial | 1.0 | 3.14E−13 | 0.34 | 1.0 |
| Clinical Case Study or Series vs Laboratory Animal Study | 0.04 | 1.0 | 1.86E−5 | 0.18 |
| Clinical Observational Study vs Clinical Trial | 1.0 | 1.35E−26 | 0.25 | 1.0 |
| Clinical Observational Study vs Laboratory Animal Study | 1.58E−12 | 1.0 | 1.74E−11 | 2.66E−12 |
| Clinical Trial vs Laboratory Animal Study | 1.42E−3 | 3.96E−19 | 2.07E−3 | 1.92E−3 |
Table 4
Summary table of problematic and informative COI and funding statement examples that we repeatedly came across.
Ambiguous or problematic statements are in red and clear statements are in green.
| Conflict of interest (COI) statements | |
|---|---|
| Example | Comment |
| When the COI is mixed with funding when disclosures are listed as ‘none’. | While funding can be considered a conflict of interest, the statements should be carefully separated if ‘disclosures’ and ‘funding’ are listed separately. Support for a specific author may not always be directly funding the project but may influence the project as a conflict of interest. |
| ‘Conflict of interest: the author has received fees or grants from COMPANY.’ | ‘Fees’ and ‘grants’ are two different elements and should be clarified as both can influence a study. It is unclear whether these fees or grants funded the published study and how they influenced the study if at all. |
| ‘Both authors have reported that they have no relationships relevant to the contents of this paper to disclose.’ ‘The authors have reported that they have no relationships relevant to the contents of this paper to disclose.’ ‘Conflicts of interest: none.’ ‘Disclosures: none.’ | This statement suggests that there are potential conflicts of interests and leaves the reader wondering if there are possible conflicts of interests left out. It does not leave room to be confident that there are no conflicts of interests. |
| ‘Dr. NAME has received research grants from NAME; and has received honoraria for NAME. Dr. NAME has received research grants from NAME. Drs. NAMES are founders of COMPANY and as such have received modest honoraria from COMPANY.’‘Dr. NAME is related through family to a member of COMPANY but neither she, nor her spouse, nor children have financial involvement or equity interest in and have received no financial assistance, support, or grants from the aforementioned.’ | It is highly ambiguous as to whether this is a funding or COI statement. It is unclear if funders played any role in designing or performing the experiment. |
| There is no COI but there is an acknowledgement. | Acknowledgements and COI should be separate sections as these two sections have different purposes. Including COI as acknowledgement allows COIs to be easily overlooked. |
| No COI or funding statement. | COI and funding statements provide additional factors that can impact a study outside of experimental factors. This information should always be included to fully inform readers, particularly when concerns are raised about certain studies or when the information is applied in real-world applications. |
| ‘NAME served as the Guest Associate Editor for this paper.’ | Although many authors will not serve as editors for the journals they are applying to, it is helpful to acknowledge when that is the case and the potential conflict of interest. |
| The authors specifically state that they have no conflicts of interest. | This is a clear and definitive statement that there are no conflicts of interests and readers are not left wondering if there are conflicts of interests that are not mentioned. |
| Itemization of funders with explicit listing of the ways funders did not contribute to the study. ‘The authors have reported that they have no relationships relevant to the contents of this paper to disclose.’ | We believe this is an excellent way of recognizing that funding can be a form of conflict of interest. The authors also specifically state that they have no conflict of interests. |
| Funding statements | |
| Example | Comment |
| Funding in acknowledgements. | Funding should be separate from acknowledgements. When funding is included in acknowledgements, it is easily overlooked. |
| Long list of affiliations without any statement that the list is funding or COI. | Funding and COI should be considered as an opportunity to share how factors outside of the experiment influenced the study. Listing affiliations is not sufficient and should be followed by explanations of why they are listed. |
| Funding statement system, particularly when papers list ‘funding on page PAGENUMBER’. | Listing ‘funding on page PAGENUMBER’ is ambiguous, leading to issues with finding the funding statement. We experienced cases where we either could not find the funding statement, or it was difficult to access. Funding statements should be listed with their respective articles. |
| Funding statement found on pages outside of their respective articles ‘Sources of Funding, see page PAGENUMBER.’ | Funding statements should be directly associated with their corresponding article. Although including a funding statement elsewhere in a journal issue is better than no funding statement, the location is distant and disconnected from its corresponding article. If a pay wall is present, access may differ between the funding statement and the original article. |
| ‘Acknowledgements: Dr. NAME is a recipient of a grant from the ORGANIZATION in support of SPECIFIC research.’ | We believe that funding should be separate from acknowledgements, as these two sections have their own purposes. |
| ‘Sources of Funding: none.’ ‘Disclosures: Drs. NAMES received modest consulting fees from COMPANY for the conduct of this research. NAME is funded by grants from COMPANY. Dr. NAME reports a charitable grant from the ORGANIZATION, and personal fees from COMPANY. The other authors report no conflicts.’ | There are no funding sources listed, but numerous avenues of fees, grants, and more are then listed as disclosures. It is not clear that these fees and grants did not influence the study in any way, including whether those fees and grants were used to partially fund the study. |
| Under ‘sources of funding’, no sources of funding are listed, but the COI statement refers to (explicit) statements that describe funding for the study. | There should not be discrepancies between reports on funding and COI. This makes it difficult for the reader to gauge how factors are truly impacting the study. |
| Funding and COI are condensed into a single section below author associations. Funding statement and COI are listed together as ‘Footnotes’. | COI, funding, and author associations should be listed separately for easy understanding. By listing COI, funding, and author associations together, it is difficult to understand who impacts the study in what ways. |
| ‘Dr. NAME is supported by FUNDING from the COMPANY/ORGANIZATION. The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the article; and decision to submit the article for publication.’ | This statement clearly acknowledges how funders can play a role in a study and explicitly states that funders had no role in the experiment and publication. The reader is not left wondering if there are additional conflicts. |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Transparency of research practices in cardiovascular literature
eLife 14:e81051.
https://doi.org/10.7554/eLife.81051
