what is needed for critical appraisal of research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Clin Diagn Res
• v.11(5); 2017 May

Critical Appraisal of Clinical Research

Azzam al-jundi.

1 Professor, Department of Orthodontics, King Saud bin Abdul Aziz University for Health Sciences-College of Dentistry, Riyadh, Kingdom of Saudi Arabia.

Salah Sakka

2 Associate Professor, Department of Oral and Maxillofacial Surgery, Al Farabi Dental College, Riyadh, KSA.

Evidence-based practice is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient’s values and expectations into the decision making process for patient care. It is a fundamental skill to be able to identify and appraise the best available evidence in order to integrate it with your own clinical experience and patients values. The aim of this article is to provide a robust and simple process for assessing the credibility of articles and their value to your clinical practice.

Introduction

Decisions related to patient value and care is carefully made following an essential process of integration of the best existing evidence, clinical experience and patient preference. Critical appraisal is the course of action for watchfully and systematically examining research to assess its reliability, value and relevance in order to direct professionals in their vital clinical decision making [ 1 ].

Critical appraisal is essential to:

• Combat information overload;
• Identify papers that are clinically relevant;
• Continuing Professional Development (CPD).

Carrying out Critical Appraisal:

Assessing the research methods used in the study is a prime step in its critical appraisal. This is done using checklists which are specific to the study design.

Standard Common Questions:

• What is the research question?
• What is the study type (design)?
• Selection issues.
• What are the outcome factors and how are they measured?
• What are the study factors and how are they measured?
• What important potential confounders are considered?
• What is the statistical method used in the study?
• Statistical results.
• What conclusions did the authors reach about the research question?
• Are ethical issues considered?

The Critical Appraisal starts by double checking the following main sections:

I. Overview of the paper:

• The publishing journal and the year
• The article title: Does it state key trial objectives?
• The author (s) and their institution (s)

The presence of a peer review process in journal acceptance protocols also adds robustness to the assessment criteria for research papers and hence would indicate a reduced likelihood of publication of poor quality research. Other areas to consider may include authors’ declarations of interest and potential market bias. Attention should be paid to any declared funding or the issue of a research grant, in order to check for a conflict of interest [ 2 ].

II. ABSTRACT: Reading the abstract is a quick way of getting to know the article and its purpose, major procedures and methods, main findings, and conclusions.

• Aim of the study: It should be well and clearly written.
• Materials and Methods: The study design and type of groups, type of randomization process, sample size, gender, age, and procedure rendered to each group and measuring tool(s) should be evidently mentioned.
• Results: The measured variables with their statistical analysis and significance.
• Conclusion: It must clearly answer the question of interest.

III. Introduction/Background section:

An excellent introduction will thoroughly include references to earlier work related to the area under discussion and express the importance and limitations of what is previously acknowledged [ 2 ].

-Why this study is considered necessary? What is the purpose of this study? Was the purpose identified before the study or a chance result revealed as part of ‘data searching?’

-What has been already achieved and how does this study be at variance?

-Does the scientific approach outline the advantages along with possible drawbacks associated with the intervention or observations?

IV. Methods and Materials section : Full details on how the study was actually carried out should be mentioned. Precise information is given on the study design, the population, the sample size and the interventions presented. All measurements approaches should be clearly stated [ 3 ].

V. Results section : This section should clearly reveal what actually occur to the subjects. The results might contain raw data and explain the statistical analysis. These can be shown in related tables, diagrams and graphs.

VI. Discussion section : This section should include an absolute comparison of what is already identified in the topic of interest and the clinical relevance of what has been newly established. A discussion on a possible related limitations and necessitation for further studies should also be indicated.

Does it summarize the main findings of the study and relate them to any deficiencies in the study design or problems in the conduct of the study? (This is called intention to treat analysis).

• Does it address any source of potential bias?
• Are interpretations consistent with the results?
• How are null findings interpreted?
• Does it mention how do the findings of this study relate to previous work in the area?
• Can they be generalized (external validity)?
• Does it mention their clinical implications/applicability?
• What are the results/outcomes/findings applicable to and will they affect a clinical practice?
• Does the conclusion answer the study question?
• -Is the conclusion convincing?
• -Does the paper indicate ethics approval?
• -Can you identify potential ethical issues?
• -Do the results apply to the population in which you are interested?
• -Will you use the results of the study?

Once you have answered the preliminary and key questions and identified the research method used, you can incorporate specific questions related to each method into your appraisal process or checklist.

1-What is the research question?

For a study to gain value, it should address a significant problem within the healthcare and provide new or meaningful results. Useful structure for assessing the problem addressed in the article is the Problem Intervention Comparison Outcome (PICO) method [ 3 ].

P = Patient or problem: Patient/Problem/Population:

It involves identifying if the research has a focused question. What is the chief complaint?

E.g.,: Disease status, previous ailments, current medications etc.,

I = Intervention: Appropriately and clearly stated management strategy e.g.,: new diagnostic test, treatment, adjunctive therapy etc.,

C= Comparison: A suitable control or alternative

E.g.,: specific and limited to one alternative choice.

O= Outcomes: The desired results or patient related consequences have to be identified. e.g.,: eliminating symptoms, improving function, esthetics etc.,

The clinical question determines which study designs are appropriate. There are five broad categories of clinical questions, as shown in [ Table/Fig-1 ].

[Table/Fig-1]:

Categories of clinical questions and the related study designs.

2- What is the study type (design)?

The study design of the research is fundamental to the usefulness of the study.

In a clinical paper the methodology employed to generate the results is fully explained. In general, all questions about the related clinical query, the study design, the subjects and the correlated measures to reduce bias and confounding should be adequately and thoroughly explored and answered.

Participants/Sample Population:

Researchers identify the target population they are interested in. A sample population is therefore taken and results from this sample are then generalized to the target population.

The sample should be representative of the target population from which it came. Knowing the baseline characteristics of the sample population is important because this allows researchers to see how closely the subjects match their own patients [ 4 ].

Sample size calculation (Power calculation): A trial should be large enough to have a high chance of detecting a worthwhile effect if it exists. Statisticians can work out before the trial begins how large the sample size should be in order to have a good chance of detecting a true difference between the intervention and control groups [ 5 ].

• Is the sample defined? Human, Animals (type); what population does it represent?
• Does it mention eligibility criteria with reasons?
• Does it mention where and how the sample were recruited, selected and assessed?
• Does it mention where was the study carried out?
• Is the sample size justified? Rightly calculated? Is it adequate to detect statistical and clinical significant results?
• Does it mention a suitable study design/type?
• Is the study type appropriate to the research question?
• Is the study adequately controlled? Does it mention type of randomization process? Does it mention the presence of control group or explain lack of it?
• Are the samples similar at baseline? Is sample attrition mentioned?
• All studies report the number of participants/specimens at the start of a study, together with details of how many of them completed the study and reasons for incomplete follow up if there is any.
• Does it mention who was blinded? Are the assessors and participants blind to the interventions received?
• Is it mentioned how was the data analysed?
• Are any measurements taken likely to be valid?

Researchers use measuring techniques and instruments that have been shown to be valid and reliable.

Validity refers to the extent to which a test measures what it is supposed to measure.

(the extent to which the value obtained represents the object of interest.)

• -Soundness, effectiveness of the measuring instrument;
• -What does the test measure?
• -Does it measure, what it is supposed to be measured?
• -How well, how accurately does it measure?

Reliability: In research, the term reliability means “repeatability” or “consistency”

Reliability refers to how consistent a test is on repeated measurements. It is important especially if assessments are made on different occasions and or by different examiners. Studies should state the method for assessing the reliability of any measurements taken and what the intra –examiner reliability was [ 6 ].

3-Selection issues:

The following questions should be raised:

• - How were subjects chosen or recruited? If not random, are they representative of the population?
• - Types of Blinding (Masking) Single, Double, Triple?
• - Is there a control group? How was it chosen?
• - How are patients followed up? Who are the dropouts? Why and how many are there?
• - Are the independent (predictor) and dependent (outcome) variables in the study clearly identified, defined, and measured?
• - Is there a statement about sample size issues or statistical power (especially important in negative studies)?
• - If a multicenter study, what quality assurance measures were employed to obtain consistency across sites?
• - Are there selection biases?
• • In a case-control study, if exercise habits to be compared:
• - Are the controls appropriate?
• - Were records of cases and controls reviewed blindly?
• - How were possible selection biases controlled (Prevalence bias, Admission Rate bias, Volunteer bias, Recall bias, Lead Time bias, Detection bias, etc.,)?
• • Cross Sectional Studies:
• - Was the sample selected in an appropriate manner (random, convenience, etc.,)?
• - Were efforts made to ensure a good response rate or to minimize the occurrence of missing data?
• - Were reliability (reproducibility) and validity reported?
• • In an intervention study, how were subjects recruited and assigned to groups?
• • In a cohort study, how many reached final follow-up?
• - Are the subject’s representatives of the population to which the findings are applied?
• - Is there evidence of volunteer bias? Was there adequate follow-up time?
• - What was the drop-out rate?
• - Any shortcoming in the methodology can lead to results that do not reflect the truth. If clinical practice is changed on the basis of these results, patients could be harmed.

Researchers employ a variety of techniques to make the methodology more robust, such as matching, restriction, randomization, and blinding [ 7 ].

Bias is the term used to describe an error at any stage of the study that was not due to chance. Bias leads to results in which there are a systematic deviation from the truth. As bias cannot be measured, researchers need to rely on good research design to minimize bias [ 8 ]. To minimize any bias within a study the sample population should be representative of the population. It is also imperative to consider the sample size in the study and identify if the study is adequately powered to produce statistically significant results, i.e., p-values quoted are <0.05 [ 9 ].

4-What are the outcome factors and how are they measured?

• -Are all relevant outcomes assessed?
• -Is measurement error an important source of bias?

5-What are the study factors and how are they measured?

• -Are all the relevant study factors included in the study?
• -Have the factors been measured using appropriate tools?

Data Analysis and Results:

- Were the tests appropriate for the data?

- Are confidence intervals or p-values given?

• How strong is the association between intervention and outcome?
• How precise is the estimate of the risk?
• Does it clearly mention the main finding(s) and does the data support them?
• Does it mention the clinical significance of the result?
• Is adverse event or lack of it mentioned?
• Are all relevant outcomes assessed?
• Was the sample size adequate to detect a clinically/socially significant result?
• Are the results presented in a way to help in health policy decisions?
• Is there measurement error?
• Is measurement error an important source of bias?

Confounding Factors:

A confounder has a triangular relationship with both the exposure and the outcome. However, it is not on the causal pathway. It makes it appear as if there is a direct relationship between the exposure and the outcome or it might even mask an association that would otherwise have been present [ 9 ].

6- What important potential confounders are considered?

• -Are potential confounders examined and controlled for?
• -Is confounding an important source of bias?

7- What is the statistical method in the study?

• -Are the statistical methods described appropriate to compare participants for primary and secondary outcomes?
• -Are statistical methods specified insufficient detail (If I had access to the raw data, could I reproduce the analysis)?
• -Were the tests appropriate for the data?
• -Are confidence intervals or p-values given?
• -Are results presented as absolute risk reduction as well as relative risk reduction?

Interpretation of p-value:

The p-value refers to the probability that any particular outcome would have arisen by chance. A p-value of less than 1 in 20 (p<0.05) is statistically significant.

• When p-value is less than significance level, which is usually 0.05, we often reject the null hypothesis and the result is considered to be statistically significant. Conversely, when p-value is greater than 0.05, we conclude that the result is not statistically significant and the null hypothesis is accepted.

Confidence interval:

Multiple repetition of the same trial would not yield the exact same results every time. However, on average the results would be within a certain range. A 95% confidence interval means that there is a 95% chance that the true size of effect will lie within this range.

8- Statistical results:

• -Do statistical tests answer the research question?

Are statistical tests performed and comparisons made (data searching)?

Correct statistical analysis of results is crucial to the reliability of the conclusions drawn from the research paper. Depending on the study design and sample selection method employed, observational or inferential statistical analysis may be carried out on the results of the study.

It is important to identify if this is appropriate for the study [ 9 ].

• -Was the sample size adequate to detect a clinically/socially significant result?
• -Are the results presented in a way to help in health policy decisions?

Clinical significance:

Statistical significance as shown by p-value is not the same as clinical significance. Statistical significance judges whether treatment effects are explicable as chance findings, whereas clinical significance assesses whether treatment effects are worthwhile in real life. Small improvements that are statistically significant might not result in any meaningful improvement clinically. The following questions should always be on mind:

• -If the results are statistically significant, do they also have clinical significance?
• -If the results are not statistically significant, was the sample size sufficiently large to detect a meaningful difference or effect?

9- What conclusions did the authors reach about the study question?

Conclusions should ensure that recommendations stated are suitable for the results attained within the capacity of the study. The authors should also concentrate on the limitations in the study and their effects on the outcomes and the proposed suggestions for future studies [ 10 ].

• -Are the questions posed in the study adequately addressed?
• -Are the conclusions justified by the data?
• -Do the authors extrapolate beyond the data?
• -Are shortcomings of the study addressed and constructive suggestions given for future research?
• -Bibliography/References:

Do the citations follow one of the Council of Biological Editors’ (CBE) standard formats?

10- Are ethical issues considered?

If a study involves human subjects, human tissues, or animals, was approval from appropriate institutional or governmental entities obtained? [ 10 , 11 ].

Critical appraisal of RCTs: Factors to look for:

• Allocation (randomization, stratification, confounders).
• Follow up of participants (intention to treat).
• Data collection (bias).
• Sample size (power calculation).
• Presentation of results (clear, precise).
• Applicability to local population.

[ Table/Fig-2 ] summarizes the guidelines for Consolidated Standards of Reporting Trials CONSORT [ 12 ].

[Table/Fig-2]:

Summary of the CONSORT guidelines.

Critical appraisal of systematic reviews: provide an overview of all primary studies on a topic and try to obtain an overall picture of the results.

In a systematic review, all the primary studies identified are critically appraised and only the best ones are selected. A meta-analysis (i.e., a statistical analysis) of the results from selected studies may be included. Factors to look for:

• Literature search (did it include published and unpublished materials as well as non-English language studies? Was personal contact with experts sought?).
• Quality-control of studies included (type of study; scoring system used to rate studies; analysis performed by at least two experts).
• Homogeneity of studies.

[ Table/Fig-3 ] summarizes the guidelines for Preferred Reporting Items for Systematic reviews and Meta-Analyses PRISMA [ 13 ].

[Table/Fig-3]:

Summary of PRISMA guidelines.

Critical appraisal is a fundamental skill in modern practice for assessing the value of clinical researches and providing an indication of their relevance to the profession. It is a skills-set developed throughout a professional career that facilitates this and, through integration with clinical experience and patient preference, permits the practice of evidence based medicine and dentistry. By following a systematic approach, such evidence can be considered and applied to clinical practice.

Financial or other Competing Interests

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: May 31, 2024 1:57 PM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

University Libraries

• Ohio University Libraries
• Library Guides

Evidence-based Practice in Healthcare

Critical appraisal.

• EBP Tutorials
• Question- PICO
• Definitions
• Systematic Reviews
• Levels of Evidence
• Finding Evidence
• Filter by Study Type
• Too Much or Too Little?
• Quality Improvement (QI)
• Performing a Literature Review
• Contact - Need Help?

Critically Appraised Topics

CATs are critical summaries of a research article.  They are concise, standardized, and provide an appraisal of the research.

If a CAT already exists for an article, it can be read quickly and the clinical bottom line can be put to use as the clinician sees fit.  If a CAT does not exist, the CAT format provides a template to appraise the article of interest.

Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability and relevance.

  Some initial appraisal questions you could ask are:

• Is the evidence from a known, reputable source?
• Has the evidence been evaluated in any way? If so, how and by whom?
• How up-to-date is the evidence?

 Second, you look at the study itself and ask the following general appraisal questions:

• Is the methodology used appropriate for the researchers question? Is the aim clear?
• How was the outcome measured? Is that a reliable way to measure? How large was the sample size? Does the sample accurately reflect the population?
• Can the results be replicated?
• Have exclusions or limitations been listed?
• What implications does the study have for your practice? Is it relevant, logical?
• Can the results be applied to your organization/purpose?
• Centre for Evidence Based Medicine - Critical Appraisal Tools
• Duke University Medical Center Library - Appraising Evidence

CASP Checklists 

CASP Case Control Checklist

CASP Clinical Protection Rule Checklist

CASP Cohort Study Checklist

CASP Diagnostic Checklist

CASP Economic Evaluation Checklist

CASP Qualitative Study Checklist

CASP Randomized Controlled Trial (RCT) Checklist

CASP Systematic Review Checklist

Appraisal: Validity vs. Reliability & Calculators

Appraisal is the third step in the Evidence Based Medicine process. It requires that the evidence found be evaluated for its validity and clinical usefulness. 

What is validity?

• Internal validity is the extent to which the experiment demonstrated a cause-effect relationship between the independent and dependent variables.
• External validity is the extent to which one may safely generalize from the sample studied to the defined target population and to other populations.

What is reliability?

Reliability is the extent to which the results of the experiment are replicable.  The research methodology should be described in detail so that the experiment could be repeated with similar results.

Statistical Calculators for Appraisal

• Diagnostic Test Calculator
• Risk Reduction Calculator
• Diagnostic Test - calculates the Sensitivity, Specificity, PPV, NPV, LR+, and LR-
• Prospective Study - calculates the Relative Risk (RR), Absolute Relative Risk (ARR), and Number Needed to Treat (NNT)
• Case-control Study - calculates the Odds Ratio (OR)
• Randomized Control Trial (RCT) - calculates the Relative Risk Reduction (RRR), ARR, and NNT
• Chi-Square Calculator
• Likelihood Ratio (LR) Calculations - The LR is used to assess how good a diagnostic test is and to help in selecting an appropriate diagnostic test(s) or sequence of tests. They have advantages over sensitivity and specificity because they are less likely to change with the prevalence of the disorder, they can be calculated for several levels of the symptom/sign or test, they can be used to combine the results of multiple diagnostic test and the can be used to calculate post-test probability for a target disorder.
• Odds Ratio - In statistics, the odds ratio (usually abbreviated "OR") is one of three main ways to quantify how strongly the presence or absence of property A is associated with the presence or absence of property B in a given population.
• Odds Ratio to NNT Converter - To convert odds ratios to NNTs, enter a number that is > 1 or < 1 in the odds ratio textbox and a number that is not equal to 0 or 1 for the Patient's Expected Event Rate (PEER). After entering the numbers, click "Calculate" to convert the odds ratio to NNT.
• One Factor ANOVA
• Relative Risk Calculator - In statistics and epidemiology, relative risk or risk ratio (RR) is the ratio of the probability of an event occurring (for example, developing a disease, being injured) in an exposed group to the probability of the event occurring in a comparison, non-exposed group.
• Two Factor ANOVA
• << Previous: Resource Evaluation
• Next: Quality Improvement (QI) >>

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 31 January 2022

The fundamentals of critically appraising an article

• Sneha Chotaliya 1  

BDJ Student volume  29 ,  pages 12–13 ( 2022 ) Cite this article

1963 Accesses

Metrics details

Sneha Chotaliya

We are often surrounded by an abundance of research and articles, but the quality and validity can vary massively. Not everything will be of a good quality - or even valid. An important part of reading a paper is first assessing the paper. This is a key skill for all healthcare professionals as anything we read can impact or influence our practice. It is also important to stay up to date with the latest research and findings.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

We are sorry, but there is no personal subscription option available for your country.

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Chambers R, 'Clinical Effectiveness Made Easy', Oxford: Radcliffe Medical Press , 1998

Loney P L, Chambers L W, Bennett K J, Roberts J G and Stratford P W. Critical appraisal of the health research literature: prevalence or incidence of a health problem. Chronic Dis Can 1998; 19 : 170-176.

Brice R. CASP CHECKLISTS - CASP - Critical Appraisal Skills Programme . 2021. Available at: https://casp-uk.net/casp-tools-checklists/ (Accessed 22 July 2021).

White S, Halter M, Hassenkamp A and Mein G. 2021. Critical Appraisal Techniques for Healthcare Literature . St George's, University of London.

Download references

Author information

Authors and affiliations.

Academic Foundation Dentist, London, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sneha Chotaliya .

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Chotaliya, S. The fundamentals of critically appraising an article. BDJ Student 29 , 12–13 (2022). https://doi.org/10.1038/s41406-021-0275-6

Download citation

Published : 31 January 2022

Issue Date : 31 January 2022

DOI : https://doi.org/10.1038/s41406-021-0275-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• CASP Checklists
• How to use our CASP Checklists
• Referencing and Creative Commons
• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

What is Critical Appraisal?

Critical Appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. It is an essential skill for evidence-based medicine because it allows people to find and use research evidence reliably and efficiently. All of us would like to enjoy the best possible health we can. To achieve this, we need reliable information about what might harm or help us when we make healthcare decisions.

Why is Critical Appraisal important?

Critical appraisal skills are important as they enable you to assess systematically the trustworthiness, relevance and results of published papers. Where an article is published, or who wrote it should not be an indication of its trustworthiness and relevance.

Randomised Controlled Trials (RCTs): An experiment that randomises participants into two groups: one that receives the treatment and another that serves as the control. RCTs are often used in healthcare to test the efficacy of different treatments.

Learn more about how to critically appraise an RCT.

Systematic Reviews : A thorough and structured analysis of all relevant studies on a particular research question. These are often used in evidence-based practice to evaluate the effects of health and social interventions.

Discover what systematic reviews are, and why they are important .

Cohort Studies : This is an observational study where two or more groups (cohorts) of individuals are followed over time and their outcomes are compared. It's used often in medical research to investigate the potential causes of disease.

Learn more about cohort studies .

Case-Control Studies : This is an observational study where two groups differing in outcome are identified and compared on the basis of some supposed causal attribute. These are often used in epidemiological research.

Check out this article to better understand what a case-control study is in research .

Cross-Sectional Studies : An observational study that examines the relationship between health outcomes and other variables of interest in a defined population at a single point in time. They're useful for determining prevalence and risk factors.

Discover what a cross-sectonal study is and when to use one .

Qualitative Research : An in-depth analysis of a phenomenon based on unstructured data, such as interviews, observations, or written material. It's often used to gain insights into behaviours, value systems, attitudes, motivations, or culture.

This guide will help you increase your knowledge of qualitative research .

Economic Evaluation : A comparison of two or more alternatives in terms of their costs and consequences. Often used in healthcare decision making to maximise efficiency and equity.

Diagnostic Studies : Evaluates the performance of a diagnostic test in predicting the presence or absence of a disease. It is commonly used to validate the accuracy and utility of a new diagnostic procedure.

Case Series : Describes characteristics of a group of patients with a particular disease or who have undergone a specific procedure. Used in clinical medicine to present preliminary observations.

Case Studies : Detailed examination of a single individual or group. Common in psychology and social sciences, this can provide in-depth understanding of complex phenomena in their real-life context.

Aren’t we already doing it?

To some extent, the answer to this question is “yes”. Evidence-based journals can give us reliable, relevant summaries of recent research; guidelines, protocols, and pathways can synthesise the best evidence and present it in the context of a clinical problem. However, we still need to be able to assess research quality to be able to adapt what we read to what we do.

There are still significant gaps in access to evidence.

The main issues we need to address are:

Health and Social Care provision must be based on sound decisions.

In order to make well-informed and sensible choices, we need evidence that is rigorous in methodology and robust in findings.

What types of questions does a critical appraisal encourage you to ask?

• What is the main objective of the research?
• Who conducted the research and are they reputable?
• How was the research funded? Are there any potential conflicts of interest?
• How was the study designed?
• Was the sample size large enough to provide accurate results?
• Were the participants or subjects selected appropriately?
• What data collection methods were used and were they reliable and valid?
• Was the data analysed accurately and rigorously?
• Were the results and conclusions drawn directly from the data or were there assumptions made?
• Can the findings be generalised to the broader population?
• How does this research contribute to existing knowledge in this field?
• Were ethical standards maintained throughout the study?
• Were any potential biases accounted for in the design, data collection or data analysis?
• Have the researchers made suggestions for future research based on their findings?
• Are the findings of the research replicable?
• Are there any implications for policy or practice based on the research findings?
• Were all aspects of the research clearly explained and detailed?

How do you critically appraise a paper?

Critically appraising a paper involves examining the quality, validity, and relevance of a published work to identify its strengths and weaknesses.

This allows the reader to judge its trustworthiness and applicability to their area of work or research. Below are general steps for critically appraising a paper:

Decide how trustworthy a piece of research is (Validity)

• Determine what the research is telling us (Results)
• Weigh up how useful the research will be in your context (Relevance)

You need to understand the research question, do a methodology evaluation, analyse the results, check the conclusion and review the implications and limitations.

That's just a quick summary but we provide a range of in-depth  training courses  and  workshops  to help you improve your knowledge around how to successfully perform critical appraisals so book onto one today or contact us for more information.

Is Critical Appraisal In Research Different To Front-Line Usage In Nursing, Etc?

Critical appraisal in research is different from front-line usage in nursing.

Critical appraisal in research involves a careful analysis of a study's methodology, results, and conclusions to assess the quality and validity of the study. This helps researchers to determine if the study's findings are robust, reliable and applicable in their own research context. It requires a specific set of skills including understanding of research methodology, statistics, and evidence-based practices.

Front-line usage in nursing refers to the direct application of evidence-based practice and research findings in patient care settings. Nurses need to appraise the evidence critically too but their focus is on the direct implications of the research on patient care and health outcomes. The skills required here would be the ability to understand the clinical implications of research findings, communicate these effectively to patients, and incorporate these into their practice.

Both require critical appraisal but the purpose, context, and skills involved are different. Critical appraisal in research is more about evaluating research for validity and reliability whereas front-line usage in nursing is about effectively applying valid and reliable research findings to improve patient care.

How do you know if you're performing critical appraisals correctly?

Thorough Understanding : You've thoroughly read and understood the research, its aims, methodology, and conclusions. You should also be aware of the limitations or potential bias in the research.

Using a Framework or Checklist : Various frameworks exist for critically appraising research (including CASP’s own!). Using these can provide structure and make sure all key points are considered. By keeping a record of your appraisal you will be able to show your reasoning behind whether you’ve implemented a decision based on research.

Identifying Research Methods : Recognising the research design, methods used, sample size, and how data was collected and analysed are crucial in assessing the research's validity and reliability.

Checking Results and Conclusions : Check if the conclusions drawn from the research are justified by the results and data provided, and if any biases could have influenced these conclusions.

Relevance and applicability : Determine if the research's results and conclusions can be applied to other situations, particularly those relevant to your context or question.

Updating Skills : Continually updating your skills in research methods and statistical analysis will improve your confidence and ability in critically appraising research.

Finally, getting feedback from colleagues or mentors on your critical appraisals can also provide a good check on how well you're doing. They can provide an additional perspective and catch anything you might have missed. If possible, we would always recommend doing appraisals in small groups or pairs, working together is always helpful for another perspective, or if you can – join and take part in a journal club.

Ready to Learn more?

Critical Appraisal Training Courses

Critical Appraisal Workshops

• CASP Checklist

Need more information?

• Online Learning
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

Best Practice for Literature Searching

• Literature Search Best Practice
• What is literature searching?
• What are literature reviews?
• Hierarchies of evidence
• 1. Managing references
• 2. Defining your research question
• 3. Where to search
• 4. Search strategy
• 5. Screening results
• 6. Paper acquisition
• 7. Critical appraisal
• Further resources
• Training opportunities and videos
• Join FSTA student advisory board This link opens in a new window
• Chinese This link opens in a new window
• Italian This link opens in a new window
• Persian This link opens in a new window
• Portuguese This link opens in a new window
• Spanish This link opens in a new window

What is critical appraisal?

We critically appraise information constantly, formally or informally, to determine if something is going to be valuable for our purpose and whether we trust the content it provides.

In the context of a literature search, critical appraisal is the process of systematically evaluating and assessing the research you have found in order to determine its quality and validity. It is essential to evidence-based practice.

More formally, critical appraisal is a systematic evaluation of research papers in order to answer the following questions:

• Does this study address a clearly focused question?
• Did the study use valid methods to address this question?
• Are there factors, based on the study type, that might have confounded its results?
• Are the valid results of this study important?
• What are the confines of what can be concluded from the study?
• Are these valid, important, though possibly limited, results applicable to my own research?

What is quality and how do you assess it?

In research we commissioned in 2018, researchers told us that they define ‘high quality evidence’ by factors such as:

• Publication in a journal they consider reputable or with a high Impact Factor.
• The peer review process, coordinated by publishers and carried out by other researchers.
• Research institutions and authors who undertake quality research, and with whom they are familiar.

In other words, researchers use their own experience and expertise to assess quality.

However, students and early career researchers are unlikely to have built up that level of experience, and no matter how experienced a researcher is, there are certain times (for instance, when conducting a systematic review) when they will need to take a very close look at the validity of research articles.

There are checklists available to help with critical appraisal.  The checklists outline the key questions to ask for a specific study design.  Examples can be found in the  Critical Appraisal  section of this guide, and the Further Resources section.  

You may also find it beneficial to discuss issues such as quality and reputation with:

• Your principal investigator (PI)
• Your supervisor or other senior colleagues
• Journal clubs. These are sometimes held by faculty or within organisations to encourage researchers to work together to discover and critically appraise information.
• Topic-specific working groups

The more you practice critical appraisal, the quicker and more confident you will become at it.

• << Previous: What are literature reviews?
• Next: Hierarchies of evidence >>
• Last Updated: May 17, 2024 5:48 PM
• URL: https://ifis.libguides.com/literature_search_best_practice

Critical Appraisal of Quantitative Research

• Living reference work entry
• Latest version View entry history
• First Online: 12 June 2018
• Cite this living reference work entry

• Rocco Cavaleri 2 ,
• Sameer Bhole 3 , 5 &
• Amit Arora 2 , 4 , 5  

1281 Accesses

1 Citations

2 Altmetric

Critical appraisal skills are important for anyone wishing to make informed decisions or improve the quality of healthcare delivery. A good critical appraisal provides information regarding the believability and usefulness of a particular study. However, the appraisal process is often overlooked, and critically appraising quantitative research can be daunting for both researchers and clinicians. This chapter introduces the concept of critical appraisal and highlights its importance in evidence-based practice. Readers are then introduced to the most common quantitative study designs and key questions to ask when appraising each type of study. These studies include systematic reviews, experimental studies (randomized controlled trials and non-randomized controlled trials), and observational studies (cohort, case-control, and cross-sectional studies). This chapter also provides the tools most commonly used to appraise the methodological and reporting quality of quantitative studies. Overall, this chapter serves as a step-by-step guide to appraising quantitative research in healthcare settings.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Altman DG, Bland JM. Treatment allocation in controlled trials: why randomise? BMJ. 1999;318(7192):1209.

Article   Google Scholar  

Arora A, Scott JA, Bhole S, Do L, Schwarz E, Blinkhorn AS. Early childhood feeding practices and dental caries in preschool children: a multi-centre birth cohort study. BMC Public Health. 2011;11(1):28.

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, … Lijmer JG. The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med. 2003;138(1):W1–12.

Google Scholar  

Cavaleri R, Schabrun S, Te M, Chipchase L. Hand therapy versus corticosteroid injections in the treatment of de quervain’s disease: a systematic review and meta-analysis. J Hand Ther. 2016;29(1):3–11. https://doi.org/10.1016/j.jht.2015.10.004 .

Centre for Evidence-based Management. Critical appraisal tools. 2017. Retrieved 20 Dec 2017, from https://www.cebma.org/resources-and-tools/what-is-critical-appraisal/ .

Centre for Evidence-based Medicine. Critical appraisal worksheets. 2017. Retrieved 3 Dec 2017, from http://www.cebm.net/blog/2014/06/10/critical-appraisal/ .

Clark HD, Wells GA, Huët C, McAlister FA, Salmi LR, Fergusson D, Laupacis A. Assessing the quality of randomized trials: reliability of the jadad scale. Control Clin Trials. 1999;20(5):448–52. https://doi.org/10.1016/S0197-2456(99)00026-4 .

Critical Appraisal Skills Program. Casp checklists. 2017. Retrieved 5 Dec 2017, from http://www.casp-uk.net/casp-tools-checklists .

Dawes M, Davies P, Gray A, Mant J, Seers K, Snowball R. Evidence-based practice: a primer for health care professionals. London: Elsevier; 2005.

Dumville JC, Torgerson DJ, Hewitt CE. Research methods: reporting attrition in randomised controlled trials. BMJ. 2006;332(7547):969.

Greenhalgh T, Donald A. Evidence-based health care workbook: understanding research for individual and group learning. London: BMJ Publishing Group; 2000.

Guyatt GH, Sackett DL, Cook DJ, Guyatt G, Bass E, Brill-Edwards P, … Gerstein H. Users’ guides to the medical literature: II. How to use an article about therapy or prevention. JAMA. 1993;270(21):2598–601.

Guyatt GH, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, … Jaeschke R. GRADE guidelines: 1. Introduction – GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4), 383–94.

Herbert R, Jamtvedt G, Mead J, Birger Hagen K. Practical evidence-based physiotherapy. London: Elsevier Health Sciences; 2005.

Hewitt CE, Torgerson DJ. Is restricted randomisation necessary? BMJ. 2006;332(7556):1506–8.

Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.0.2. The cochrane collaboration. 2009. Retrieved 3 Dec 2017, from http://www.cochrane-handbook.org .

Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. Chatswood: Elsevier Health Sciences; 2013.

Hoffmann T, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, … Johnston M. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ, 2014;348: g1687.

Joanna Briggs Institute. Critical appraisal tools. 2017. Retrieved 4 Dec 2017, from http://joannabriggs.org/research/critical-appraisal-tools.html .

Mhaskar R, Emmanuel P, Mishra S, Patel S, Naik E, Kumar A. Critical appraisal skills are essential to informed decision-making. Indian J Sex Transm Dis. 2009;30(2):112–9. https://doi.org/10.4103/0253-7184.62770 .

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Med Res Methodol. 2001;1(1):2. https://doi.org/10.1186/1471-2288-1-2 .

Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement. PLoS Med. 2009;6(7):e1000097.

National Health and Medical Research Council. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. Retrieved from https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf .

National Heart Lung and Blood Institute. Study quality assessment tools. 2017. Retrieved 17 Dec 2017, from https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools .

Physiotherapy Evidence Database. PEDro scale. 2017. Retrieved 10 Dec 2017, from https://www.pedro.org.au/english/downloads/pedro-scale/ .

Portney L, Watkins M. Foundations of clinical research: application to practice. 2nd ed. Upper Saddle River: F.A. Davis Company/Publishers; 2009.

Roberts C, Torgerson DJ. Understanding controlled trials: baseline imbalance in randomised controlled trials. BMJ. 1999;319(7203):185.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, … Kristjansson E. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. https://doi.org/10.1136/bmj.j4008 .

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, … Boutron I. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, … Thacker SB. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, Initiative S. The strengthening the reporting of observational studies in epidemiology (strobe) statement: guidelines for reporting observational studies. Int J Surg. 2014;12(12):1495–9.

Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, … Bossuyt PM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155(8):529–36.

Download references

Author information

Authors and affiliations.

School of Science and Health, Western Sydney University, Campbelltown, NSW, Australia

Rocco Cavaleri & Amit Arora

Sydney Dental School, Faculty of Medicine and Health, The University of Sydney, Surry Hills, NSW, Australia

Sameer Bhole

Discipline of Child and Adolescent Health, Sydney Medical School, The University of Sydney, Westmead, NSW, Australia

Oral Health Services, Sydney Local Health District and Sydney Dental Hospital, NSW Health, Surry Hills, NSW, Australia

Sameer Bhole & Amit Arora

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rocco Cavaleri .

Editor information

Editors and affiliations.

School of Science & Health, Western Sydney University, Penrith, New South Wales, Australia

Pranee Liamputtong

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this entry

Cite this entry.

Cavaleri, R., Bhole, S., Arora, A. (2018). Critical Appraisal of Quantitative Research. In: Liamputtong, P. (eds) Handbook of Research Methods in Health Social Sciences . Springer, Singapore. https://doi.org/10.1007/978-981-10-2779-6_120-2

Download citation

DOI : https://doi.org/10.1007/978-981-10-2779-6_120-2

Received : 20 January 2018

Accepted : 12 February 2018

Published : 12 June 2018

Publisher Name : Springer, Singapore

Print ISBN : 978-981-10-2779-6

Online ISBN : 978-981-10-2779-6

eBook Packages : Springer Reference Social Sciences Reference Module Humanities and Social Sciences Reference Module Business, Economics and Social Sciences

• Publish with us

Policies and ethics

Chapter history

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-2

DOI: https://doi.org/10.1007/978-981-10-2779-6_120-1

• Find a journal
• Track your research

Critical Appraisal : What is critical appraisal?

What is critical appraisal.

• Where to start
• Education and childhood studies
• Occupational Therapy
• Physiotherapy
• Interpreting statistics
• Further reading and resources

About this guide

This guide is designed to help students (mainly in Health Sciences, but there are checklist tools for Business and Education students) to understand the purpose and process of critical appraisal, and different methods and frameworks that can be used in different contexts.

Critical appraisal is an essential step in any evidence based process and it is defined by CASP as "t he process of assessing and interpreting evidence by systematically considering its validity , results and relevance ".

The hierarchy of evidence pyramid below provides a means to visualise the levels of evidence as well as the amount of evidence available. Systematic reviews and meta-analyses are the highest level of evidence therefore they are at the top of the pyramid but they are also the least common because they are based on the studies below them.  Moving down the pyramid, the amount of studies increases but the level of evidence decreases. The use of the hierarchy of evidence pyramid is not enough to determine the quality of research because study types can vary in quality whether it is a systematic review or a case study therefore critical appraisal skills are required to evaluate all types of evidence regardless of their level. It is important to apply your own critical appraisal skills when you evaluate research studies to decide if they merit being considered or used as reliable sources of information. Some studies have found that many research findings in published articles may in fact be false  ( Ioannidis, 2005 ).  In the worst cases, some researchers may commit research fraud to acquire research grants  ( Harvey, 2020 ).

Image from https://www.pinterest.co.uk/pin/246361042093714264/

Critical appraisal involves using a set of systematic techniques that enable you to evaluate the quality of published research including the research methodology, potential bias, strengths and weaknesses and, ultimately, its trustworthiness. It is often the case that even peer-reviewed research can have methodological flaws, incorrectly interpret data, draw incorrect conclusions or exaggerate findings. A uthors' affiliations, funding sources, study design flaws, sample size and potential bias are only some of the factors that can lead you to include poor quality research in your own work if not addressed through critical appraisal.

Critical appraisal often involves the  use of checklists to guide you to look out for specific areas in the appraisal process. Checklists vary according to types of research or study designs you are evaluating.

It is important therefore that you possess a good knowledge of research methods in your field of study and a good basic understanding of statistics where statistical analysis is involved.

Please see read  What is critical appraisal?   and see the resources on this page for further information on critical appraisal.

• Next: Where to start >>
• Last Updated: May 21, 2024 2:14 PM
• URL: https://libguides.qmu.ac.uk/critical-appraisal

Please enter both an email address and a password.

Account login

• Show/Hide Password Show password Hide password
• Reset Password

Need to reset your password?  Enter the email address which you used to register on this site (or your membership/contact number) and we'll email you a link to reset it. You must complete the process within 2hrs of receiving the link.

We've sent you an email.

An email has been sent to Simply follow the link provided in the email to reset your password. If you can't find the email please check your junk or spam folder and add [email protected] to your address book.

• About RCS England

MRCS Part A

• Dissecting the literature: the importance of critical appraisal

08 Dec 2017

Kirsty Morrison

This post was updated  in 2023.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like Medline or EMBASE often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research relevant to your practice.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Current Awareness Updates .

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked? A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question? You can learn more about different study types and the hierarchy of evidence here .
• Did the methodology address important potential sources of bias? Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Was the study performed according to the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial .
• Does the study test a stated hypothesis? Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted.
• Were the statistical analyses performed correctly? The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions? Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest? Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources:

• How to Read a Paper by Trisha Greenhalgh
• The Doctor’s Guide to Critical Appraisal by Narinder Kaur Gosall
• CASP checklists
• CEBM Critical Appraisal Tools
• Critical Appraisal: a checklist
• Critical Appraisal of a Journal Article (PDF)
• Introduction to...Critical appraisal of literature
• Reporting guidelines for the main study types

Kirsty Morrison, Information Specialist

Share this page:

• Library Blog

Critical appraisal: how to evaluate research for use in clinical practice

Aleksandr Davydov / Alamy Stock Photo

After reading this article, you should be able to:

• Appreciate the importance of critical appraisal skills;
• Understand and apply principles of critical appraisal to support evidence-based practice;
• Recognise the different types of studies found in research and their design;
• Determine the quality, value and applicability of a research paper to clinical practice.

Not all data in healthcare research are of equal quality ​[1]​ . To incorporate evidence-based medicine (EBM) into practice, pharmacists must be able to assess the quality and reliability of evidence ​[2]​ . This requires the development of critical appraisal skills.

Critical appraisal

The critical appraisal of health-related literature by healthcare professionals is a multi-step process that requires ​[2]​ :

• Formulation of a question that is important for improving patient health while advancing scientific and medical knowledge;
• Searching the relevant literature to find the best available evidence;
• Appraising research critically to evaluate quality and reliability, as well as applicability to the formulated question;
• Applying the evidence to practice;
• Monitoring the interventions to ensure the outcomes are reproducible and effective. 

Assessment and evaluation of publications can be daunting. However, this article aims to assist pharmacists when critically reviewing a research paper to support clinical decision making and evidence-based practice. 

This article focuses on the theory behind critical appraisal.

Types of studies in health research 

Cohort studies, case-control studies.

• Cross-sectional studies
• Randomised clinical trials

Systematic reviews

To undertake critical analysis, it is important to first understand the types of studies that are used to generate evidence, and how the data are analysed to provide standardised measurements of outcomes ​[3]​ . These can then be compared to evaluate whether an intervention is effective. A summary of the main research studies used in healthcare research, including their advantages and limitations, can be found in Table 1 .

The most common types of studies used to report healthcare research include:

These are observational studies that can either be retrospective (e.g examine historical records) or prospective. Here a group of people are selected for inclusion who do not have the outcome of interest (e.g. exploring the association between major depression and increased risk of advanced complications in type 2 diabetes) ​[4,5]​ . Over a period, they are observed to see if they develop the outcome of interest and, therefore, the relative risk can be determined when compared with a control group ​[6]​ . One of the biggest problems with cohort studies is the loss of participants (e.g. owing to personal reasons, or their condition not improving post-treatment). This can significantly affect the results and outcomes ​[7]​ . Most importantly, cohort studies are the best way to test a hypothesis, without experimental intervention ​[8]​ . 

A type of observational study and typically retrospective, where patients in a group with a particular outcome of interest are compared with another group that does not have the outcome, but the same degree of exposure as the test group ​[6,9]​ . Case-control studies determine the relative importance of a predictor variable in relation to the presence or absence of the disease ​[6,9]​ . An example of a case-control study is investigating the association of low serum vitamin D levels with migraine ​[10]​ .

Cross-sectional studies  

These studies commonly employ interviews, questionnaires and surveys to collect data ​[10]​ . Although not rigorous enough to assess and measure clinical and medical interventions, they can be used to determine attitudes of a cross-section of the population that is representative of the outcome of interest ​[11]​ . For example, one cross-sectional study aimed to identify the main competencies and training needs of primary care pharmacists to inform a National Health Service Executive training programme ​[11]​ .

Randomised clinical trials (RCT)

The most rigorous and robust research methods for determining whether a cause–effect relationship exists between a new treatment or intervention and its outcome ​[12]​ . Although no study alone is likely to prove causality, randomisation reduces bias and the studies are often blinded, so the clinicians, patients and researchers do not know whether patients are in the control or intervention groups ​[12,13]​ . RCTs are considered the gold standard in clinical research studies and are positioned at the top of the evidence pyramid ​[14]​ (see Figure).

The greatest advantage of RCTs is the minimisation of bias providing strong clinical evidence, which is favoured by healthcare professionals; however, there are some limitations to this type of study ​[15]​ (see Table 1 ).

These studies are robust, thorough and comprehensive. They obtain a more accurate and evidence-based assessment of a research question ​[16,17]​ . By comparing a large body of data, from a wide range of sources from primary literature, the results are analysed collectively (e.g. meta-analysis) to assess for consistency and reproducibility ​[16,17]​ . Study inclusion is set by an explicit selection criterion and reviews are typically, although not always, quantitatively analysed for statistical significance ​[17]​ . Systematic reviews are useful to obtain current, updated information regarding contemporary topics in healthcare. For example, in one review on the safety and efficacy of COVID-19 vaccines, data from several RCTs were analysed and the results were compared to obtain a more justified argument for vaccine use. 

Other studies used to gather evidence in healthcare research include:

• i) case studies and case series — focusing on individuals or a collection of cases that are of interest to the author, but does not involve trying to find the answer to a hypothesis;
• ii) qualitative studies — well-suited for investigating the meanings, interpretations, social and cultural norms and perceptions that impact health-related practice and behaviour;
• iii) diagnostic tests — investigate the accuracy of a diagnostic test; it is common to compare to a ‘gold standard’ and measure either the specificity or sensitivity ​[17–20]​ .

Steps to follow when reviewing an article

Once an article has been identified as relevant to the topic of interest, it is essential to first determine the quality of the study by assessing its appropriateness, including whether the study design was able to answer the hypothesis/research question. 

The following steps outline the main considerations when validating a study and are summarised in Table 2.

1. Determine whether the study addressed a clearly focused issue

The introduction of the article should clearly state the aims and objectives of the research being undertaken, and background information should be provided so the reader understands the reasons why this research is needed, and how the research findings will contribute to advancing clinical and scientific knowledge.

Most research studies will evaluate one of the following:

• Therapy — efficacy of a drug treatment, surgical procedure or other intervention;
• Causation — if a suspected risk factor is related to a particular disease;
• Prognosis — outcome of a disease following treatment/diagnosis;
• Diagnosis — the validity and reliability of a new diagnostic test;
• Screening — test applied to a population to detect disease.

2. Identify the study population

Particular attention must be given to the selection criteria used for RCTs. Exclusion of groups of patient populations can lead to impaired generalisability of results and over-inflation of the outcomes of the study ​[29]​ . Women, children, older people and people with medical conditions are often excluded from these studies, so caution must be applied when interpreting the results ​[30]​ .

Crucial to the selection criteria is that all study participants share common aspects other than the variable being studied so comparisons can be made ​[23]​ . For observational studies, such as cohort and cross-sectional, the individuals selected should be an accurate representation of a defined population ​[31]​ .

3. Interpret the results

Assessing the appropriateness of statistical analysis can be tricky, but for evidence-based practice it is necessary to have a basic understanding of  statistics  since errors have been known to occur in published manuscripts ​[28]​ . The ‘method’ section of the paper should be clear about the rationale for the approach and how the outcomes and results were obtained. The language used should be understandable to the journal’s readership.

There are two main uses for statistics in research. These are to provide general observations and to allow comparisons or conclusions to be made ​[32,33]​ . A previous article from The Pharmaceutical Journal offers a basic introduction to statistics, providing a practical overview of differential/inferential statistics and significance testing. These will not be discussed in detail in this article.

4) Assess for bias 

Bias can occur at any stage within a research study, and the ability to identify bias is an important skill in critical appraisal because it can lead to inaccurate results. Bias is the systematic (non-random) error in design, conduct or analysis of a study resulting in mistaken estimates. Different study designs require different steps to reduce bias. Bias can occur because of the way populations are sampled, or the way in which data are collected or analysed. Unlike random error, increasing the sample size will not decrease systematic bias ​[31]​ .

There are many types of bias, but they can be considered under three main categories:

• Selection bias is when the composition of the study subjects or participants in a research project systematically differs from the source population. A simple example would be during recruitment of participants for an influenza vaccine trial, where the participants are healthy adults. However, the sample population is not representative of a cross-section of the general population — missing out children, older people and adults with comorbidities; 
• Information bias , or ‘misclassification’, occurs when outcomes, exposures of interest (factors measured) or other data are incorrectly classified or measured. This is particularly problematic in observational studies (cross-sectional, case or cohort studies) where data are gathered using questionnaires, surveys and interviews. The method of data collection is argued as unreliable; 
• Confounding is often referred to as a ‘mixing of effects’, where the effects of the exposure under study on a given outcome are mixed in with the effects of an additional factor (or set of factors), resulting in a distortion of the true relationship. Confounding factors may mask an actual association or, more commonly, falsely demonstrate an apparent association between the treatment and outcome when no real association between them exists ​[34]​ . For example, alcohol intake has been identified as a cause of increased coronary heart disease ​[35]​ . However, there are many confounding factors that ‘blur’ the facts, such as differences in socio-economic and lifestyle characteristics, the type of drink consumed (beer, wine), and the fact that smokers are more likely to drink alcohol than non-smokers. These factors will confound the observed relationship between the amount of alcohol consumed and risk ​[36]​ . 

5) Determine whether the study can be applied to practice

Pharmacy professionals can determine the applicability of study results to clinical practice by:

• Comparing research results to relevant guidelines (e.g. National Institute for Health and Care Excellence );
• Identifying whether local or national clinical policies exist that are supported by EBM;
• Discussing recommendations and the applicability of research findings with colleagues and peers;
• Summarising and critically appraising the various interventions studied in relevant clinical trials and studies; 
• Evaluating the cost-effectiveness of the interventions ​[37]​ .

Critical appraisal skills are necessary to extract the most relevant and useful information from published literature and it is the duty of all healthcare professionals to keep up to date with current research to identify gaps in knowledge and to ensure optimal patient outcomes. It is also particularly beneficial for pharmacists, as demand for such skills increases with the rise in opportunities to deliver advanced clinical services.

Additional resources — critical appraisal tools

Several user-friendly tools are available to assist individuals with developing critical appraisal skills. Table 3 summarises a selection of useful websites that provide checklists and guidance on critical appraisal skills.

• 1 Simera I, Moher D, Hoey J, et al. A catalogue of reporting guidelines for health research. European Journal of Clinical Investigation 2010; 40 :35–53. doi: 10.1111/j.1365-2362.2009.02234.x
• 2 Umesh G, Karippacheril J, Magazine R. Critical appraisal of published literature. Indian J Anaesth 2016; 60 :670–3. doi: 10.4103/0019-5049.190624
• 3 Peinemann F, Tushabe D, Kleijnen J. Using multiple types of studies in systematic reviews of health care interventions–a systematic review. PLoS One 2013; 8 :e85035. doi: 10.1371/journal.pone.0085035
• 4 Song J, Chung K. Observational studies: cohort and case-control studies. Plast Reconstr Surg 2010; 126 :2234–42. doi: 10.1097/PRS.0b013e3181f44abc
• 5 Lin EHB, Rutter CM, Katon W, et al. Depression and Advanced Complications of Diabetes: A prospective cohort study. Diabetes Care 2009; 33 :264–9. doi: 10.2337/dc09-1068
• 6 Mann C. Observational research methods. Research design II: cohort, cross sectional, and case-control studies. Emerg Med J [Internet] 2003; 20 :54–60. http://emj.bmj.com/content/20/1/54.abstract
• 7 Fogel D. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: A review. Contemp Clin Trials Commun 2018; 11 :156–64. doi: 10.1016/j.conctc.2018.08.001
• 8 Morrow B. An overview of cohort study designs and their advantages and disadvantages. International Journal of Therapy and Rehabilitation 2010; 17 :518–23. doi: 10.12968/ijtr.2010.17.10.78810
• 9 Lu C. Observational studies: a review of study designs, challenges and strategies to reduce confounding. Int J Clin Pract 2009; 63 :691–7. doi: 10.1111/j.1742-1241.2009.02056.x
• 10 Levin KA. Study design III: Cross-sectional studies. Evid Based Dent 2006; 7 :24–5. doi: 10.1038/sj.ebd.6400375
• 11 Jesson J. Cross-sectional studies in prescribing research. J Clin Pharm Ther 2001; 26 :397–403. doi: 10.1046/j.1365-2710.2001.00373.x
• 12 Bhide A, Shah PS, Acharya G. A simplified guide to randomized controlled trials. Acta Obstet Gynecol Scand 2018; 97 :380–7. doi: 10.1111/aogs.13309
• 13 Hariton E, Locascio JJ. Randomised controlled trials – the gold standard for effectiveness research. BJOG: Int J Obstet Gy 2018; 125 :1716–1716. doi: 10.1111/1471-0528.15199
• 14 Mulimani PS. Evidence-based practice and the evidence pyramid: A 21st century orthodontic odyssey. American Journal of Orthodontics and Dentofacial Orthopedics 2017; 152 :1–8. doi: 10.1016/j.ajodo.2017.03.020
• 15 Deaton A, Cartwright N. Understanding and misunderstanding randomized controlled trials. Social Science & Medicine 2018; 210 :2–21. doi: 10.1016/j.socscimed.2017.12.005
• 16 Chandler J, Hopewell S. Cochrane methods – twenty years experience in developing systematic review methods. Syst Rev 2013; 2 . doi: 10.1186/2046-4053-2-76
• 17 Murad MH, Sultan S, Haffar S, et al. Methodological quality and synthesis of case series and case reports. BMJ EBM 2018; 23 :60–3. doi: 10.1136/bmjebm-2017-110853
• 18 Munn Z, Barker TH, Moola S, et al. Methodological quality of case series studies: an introduction to the JBI critical appraisal tool. JBI Evidence Synthesis 2019; 18 :2127–33. doi: 10.11124/jbisrir-d-19-00099
• 19 Daly J, Willis K, Small R, et al. A hierarchy of evidence for assessing qualitative health research. Journal of Clinical Epidemiology 2007; 60 :43–9. doi: 10.1016/j.jclinepi.2006.03.014
• 20 Gluud C, Gluud LL. Evidence based diagnostics. BMJ 2005; 330 :724–6. doi: 10.1136/bmj.330.7493.724
• 21 Rochon PA, Gurwitz JH, Sykora K, et al. Reader’s guide to critical appraisal of cohort studies: 1. Role and design. BMJ 2005; 330 :895–7. doi: 10.1136/bmj.330.7496.895
• 22 Mamdani M, Sykora K, Li P, et al. Reader’s guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 2005; 330 :960–2. doi: 10.1136/bmj.330.7497.960
• 23 Young JM, Solomon MJ. How to critically appraise an article. Nat Rev Gastroenterol Hepatol 2009; 6 :82–91. doi: 10.1038/ncpgasthep1331
• 24 Sutton-Tyrrell K. Assessing bias in case-control studies. Proper selection of cases and controls. Stroke 1991; 22 :938–42. doi: 10.1161/01.str.22.7.938
• 25 Sedgwick P. Bias in observational study designs: cross sectional studies. BMJ 2015; 350 :h1286–h1286. doi: 10.1136/bmj.h1286
• 26 Pannucci CJ, Wilkins EG. Identifying and Avoiding Bias in Research. Plastic and Reconstructive Surgery 2010; 126 :619–25. doi: 10.1097/prs.0b013e3181de24bc
• 27 Siedlecki SL. Understanding Descriptive Research Designs and Methods. Clin Nurse Spec 2020; 34 :8–12. doi: 10.1097/nur.0000000000000493
• 28 Mulrow CD. Systematic Reviews: Rationale for systematic reviews. BMJ 1994; 309 :597–9. doi: 10.1136/bmj.309.6954.597
• 29 Littlewood C. The RCT means nothing to me! Manual Therapy 2011; 16 :614–7. doi: 10.1016/j.math.2011.06.006
• 30 Van Spall HGC, Toren A, Kiss A, et al. Eligibility Criteria of Randomized Controlled Trials Published in High-Impact General Medical Journals. JAMA 2007; 297 :1233. doi: 10.1001/jama.297.11.1233
• 31 Pinchbeck GL, Archer DC. How to critically appraise a paper. Equine Vet Educ 2018; 32 :104–9. doi: 10.1111/eve.12896
• 32 Greenhalgh T. How to read a paper: Statistics for the non-statistician. I: Different types of data need different statistical tests. BMJ 1997; 315 :364–6. doi: 10.1136/bmj.315.7104.364
• 33 Greenhalgh T. How to read a paper: Statistics for the non-statistician. II: ‘Significant’ relations and their pitfalls. BMJ 1997; 315 :422–5. doi: 10.1136/bmj.315.7105.422
• 34 Skelly A, Dettori J, Brodt E. Assessing bias: the importance of considering confounding. Evidence-Based Spine-Care Journal 2012; 3 :9–12. doi: 10.1055/s-0031-1298595
• 35 Emberson JR, Bennett DA. Effect of alcohol on risk of coronary heart disease and stroke: causality, bias, or a bit of both? Vascular Health and Risk Management 2006; 2 :239–49. doi: 10.2147/vhrm.2006.2.3.239
• 36 Corrao G, Rubbiati L, Bagnardi V, et al. Alcohol and coronary heart disease: a meta-analysis. Addiction 2000; 95 :1505–23. doi: 10.1046/j.1360-0443.2000.951015056.x
• 37 Lewis SJ, Orland BI. The Importance and Impact of Evidence Based medicine. JMCP 2004; 10 :S3–5. doi: 10.18553/jmcp.2004.10.s5-a.s3

Please leave a comment  Cancel reply

You must be logged in to post a comment.

You might also be interested in…

Using virtual reality to motivate young people to begin a career in pharmacy

BRUKINSA: a next-generation BTK inhibitor

Potency warning labels for topical steroids to be introduced following reports of adverse reactions

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 25, Issue 1
• Critical appraisal of qualitative research: necessity, partialities and the issue of bias
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0001-5660-8224 Veronika Williams ,
• Anne-Marie Boylan ,
• http://orcid.org/0000-0003-4597-1276 David Nunan
• Nuffield Department of Primary Care Health Sciences , University of Oxford, Radcliffe Observatory Quarter , Oxford , UK
• Correspondence to Dr Veronika Williams, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK; veronika.williams{at}phc.ox.ac.uk

https://doi.org/10.1136/bmjebm-2018-111132

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• qualitative research

Introduction

Qualitative evidence allows researchers to analyse human experience and provides useful exploratory insights into experiential matters and meaning, often explaining the ‘how’ and ‘why’. As we have argued previously 1 , qualitative research has an important place within evidence-based healthcare, contributing to among other things policy on patient safety, 2 prescribing, 3 4 and understanding chronic illness. 5 Equally, it offers additional insight into quantitative studies, explaining contextual factors surrounding a successful intervention or why an intervention might have ‘failed’ or ‘succeeded’ where effect sizes cannot. It is for these reasons that the MRC strongly recommends including qualitative evaluations when developing and evaluating complex interventions. 6

Critical appraisal of qualitative research

Is it necessary.

Although the importance of qualitative research to improve health services and care is now increasingly widely supported (discussed in paper 1), the role of appraising the quality of qualitative health research is still debated. 8 10 Despite a large body of literature focusing on appraisal and rigour, 9 11–15 often referred to as ‘trustworthiness’ 16 in qualitative research, there remains debate about how to —and even whether to—critically appraise qualitative research. 8–10 17–19 However, if we are to make a case for qualitative research as integral to evidence-based healthcare, then any argument to omit a crucial element of evidence-based practice is difficult to justify. That being said, simply applying the standards of rigour used to appraise studies based on the positivist paradigm (Positivism depends on quantifiable observations to test hypotheses and assumes that the researcher is independent of the study. Research situated within a positivist paradigm isbased purely on facts and consider the world to be external and objective and is concerned with validity, reliability and generalisability as measures of rigour.) would be misplaced given the different epistemological underpinnings of the two types of data.

Given its scope and its place within health research, the robust and systematic appraisal of qualitative research to assess its trustworthiness is as paramount to its implementation in clinical practice as any other type of research. It is important to appraise different qualitative studies in relation to the specific methodology used because the methodological approach is linked to the ‘outcome’ of the research (eg, theory development, phenomenological understandings and credibility of findings). Moreover, appraisal needs to go beyond merely describing the specific details of the methods used (eg, how data were collected and analysed), with additional focus needed on the overarching research design and its appropriateness in accordance with the study remit and objectives.

Poorly conducted qualitative research has been described as ‘worthless, becomes fiction and loses its utility’. 20 However, without a deep understanding of concepts of quality in qualitative research or at least an appropriate means to assess its quality, good qualitative research also risks being dismissed, particularly in the context of evidence-based healthcare where end users may not be well versed in this paradigm.

How is appraisal currently performed?

Appraising the quality of qualitative research is not a new concept—there are a number of published appraisal tools, frameworks and checklists in existence. 21–23  An important and often overlooked point is the confusion between tools designed for appraising methodological quality and reporting guidelines designed to assess the quality of methods reporting. An example is the Consolidate Criteria for Reporting Qualitative Research (COREQ) 24 checklist, which was designed to provide standards for authors when reporting qualitative research but is often mistaken for a methods appraisal tool. 10

Broadly speaking there are two types of critical appraisal approaches for qualitative research: checklists and frameworks. Checklists have often been criticised for confusing quality in qualitative research with ‘technical fixes’ 21 25 , resulting in the erroneous prioritisation of particular aspects of methodological processes over others (eg, multiple coding and triangulation). It could be argued that a checklist approach adopts the positivist paradigm, where the focus is on objectively assessing ‘quality’ where the assumptions is that the researcher is independent of the research conducted. This may result in the application of quantitative understandings of bias in order to judge aspects of recruitment, sampling, data collection and analysis in qualitative research papers. One of the most widely used appraisal tools is the Critical Appraisal Skills Programme (CASP) 26 and along with the JBI QARI (Joanna Briggs Institute Qualitative Assessment and Assessment Instrument) 27 presents examples which tend to mimic the quantitative approach to appraisal. The CASP qualitative tool follows that of other CASP appraisal tools for quantitative research designs developed in the 1990s. The similarities are therefore unsurprising given the status of qualitative research at that time.

Frameworks focus on the overarching concepts of quality in qualitative research, including transparency, reflexivity, dependability and transferability (see box 1 ). 11–13 15 16 20 28 However, unless the reader is familiar with these concepts—their meaning and impact, and how to interpret them—they will have difficulty applying them when critically appraising a paper.

The main issue concerning currently available checklist and framework appraisal methods is that they take a broad brush approach to ‘qualitative’ research as whole, with few, if any, sufficiently differentiating between the different methodological approaches (eg, Grounded Theory, Interpretative Phenomenology, Discourse Analysis) nor different methods of data collection (interviewing, focus groups and observations). In this sense, it is akin to taking the entire field of ‘quantitative’ study designs and applying a single method or tool for their quality appraisal. In the case of qualitative research, checklists, therefore, offer only a blunt and arguably ineffective tool and potentially promote an incomplete understanding of good ‘quality’ in qualitative research. Likewise, current framework methods do not take into account how concepts differ in their application across the variety of qualitative approaches and, like checklists, they also do not differentiate between different qualitative methodologies.

On the need for specific appraisal tools

Current approaches to the appraisal of the methodological rigour of the differing types of qualitative research converge towards checklists or frameworks. More importantly, the current tools do not explicitly acknowledge the prejudices that may be present in the different types of qualitative research.

Concepts of rigour or trustworthiness within qualitative research 31

Transferability: the extent to which the presented study allows readers to make connections between the study’s data and wider community settings, ie, transfer conceptual findings to other contexts.

Credibility: extent to which a research account is believable and appropriate, particularly in relation to the stories told by participants and the interpretations made by the researcher.

Reflexivity: refers to the researchers’ engagement of continuous examination and explanation of how they have influenced a research project from choosing a research question to sampling, data collection, analysis and interpretation of data.

Transparency: making explicit the whole research process from sampling strategies, data collection to analysis. The rationale for decisions made is as important as the decisions themselves.

However, we often talk about these concepts in general terms, and it might be helpful to give some explicit examples of how the ‘technical processes’ affect these, for example, partialities related to:

Selection: recruiting participants via gatekeepers, such as healthcare professionals or clinicians, who may select them based on whether they believe them to be ‘good’ participants for interviews/focus groups.

Data collection: poor interview guide with closed questions which encourage yes/no answers and/leading questions.

Reflexivity and transparency: where researchers may focus their analysis on preconceived ideas rather than ground their analysis in the data and do not reflect on the impact of this in a transparent way.

The lack of tailored, method-specific appraisal tools has potentially contributed to the poor uptake and use of qualitative research for informing evidence-based decision making. To improve this situation, we propose the need for more robust quality appraisal tools that explicitly encompass both the core design aspects of all qualitative research (sampling/data collection/analysis) but also considered the specific partialities that can be presented with different methodological approaches. Such tools might draw on the strengths of current frameworks and checklists while providing users with sufficient understanding of concepts of rigour in relation to the different types of qualitative methods. We provide an outline of such tools in the third and final paper in this series.

As qualitative research becomes ever more embedded in health science research, and in order for that research to have better impact on healthcare decisions, we need to rethink critical appraisal and develop tools that allow differentiated evaluations of the myriad of qualitative methodological approaches rather than continuing to treat qualitative research as a single unified approach.

• Williams V ,
• Boylan AM ,
• Lingard L ,
• Orser B , et al
• Brawn R , et al
• Van Royen P ,
• Vermeire E , et al
• Barker M , et al
• McGannon KR
• Dixon-Woods M ,
• Agarwal S , et al
• Greenhalgh T ,
• Dennison L ,
• Morrison L ,
• Conway G , et al
• Barrett M ,
• Mayan M , et al
• Lockwood C ,
• Santiago-Delefosse M ,
• Bruchez C , et al
• Sainsbury P ,
• ↵ CASP (Critical Appraisal Skills Programme). date unknown . http://www.phru.nhs.uk/Pages/PHD/CASP.htm .
• ↵ The Joanna Briggs Institute . JBI QARI Critical appraisal checklist for interpretive & critical research . Adelaide : The Joanna Briggs Institute , 2014 .
• Stephens J ,

Contributors VW and DN: conceived the idea for this article. VW: wrote the first draft. AMB and DN: contributed to the final draft. All authors approve the submitted article.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Correction notice This article has been updated since its original publication to include a new reference (reference 1.)

Read the full text or download the PDF:

Find Info For

• Current Students
• Prospective Students
• Research and Partnerships
• Entrepreneurship and Commercialization

Quick Links

• Health and Life Sciences
• Info Security and AI
• Transformative Education
• Purdue Today
• Purdue Global
• Purdue in the News

June 3, 2024

What lies beneath: Mars’ subsurface ice could be a key to sustaining future habitats on other planets

Ali Bramson, an assistant professor in the Department of Earth, Atmospheric, and Planetary Sciences in Purdue University’s College of Science, holds a land-based mobile radar system. Bramson is focused on locating subsurface ice deposits on Mars and the moon for climate research and as a resource for future habitats. (Purdue University photo/Kelsey Lefever)

Purdue scientist extending her search for large subsurface ice deposits to the moon

WEST LAFAYETTE, Ind. — To survive on other planets, water is, of course, critical. We need it to drink, sustain crops and even create rocket fuel.

But on spaceflights, checked luggage is exorbitantly expensive. Anything heavy, especially liquids like water, is bulky and costly to haul by rocket, even to our closest interplanetary neighbors. The best plan, then, is to find water at the spacecraft’s destination.

Purdue University planetary scientist Ali Bramson’s research is laying the foundation for future extraterrestrial exploration. She is focused on finding ice deposits beneath the barren surfaces of the moon and Mars, providing a buried resource important for future human habitats and even space travel itself. Subsurface ice also is a compelling target for astrobiology, climatology and geology research.

ADDITIONAL INFORMATION

• Science enabling heat and air conditioning for long-term space habitats is almost fully available
• Discover Purdue’s latest and greatest in space sciences
• Autonomous grasping robots will help future astronauts maintain space habitats

Bramson is continuing work that began with Mars through NASA’s Subsurface Water Ice Mapping project. Radar analysis from spacecraft orbiting Mars probed beneath the planet’s surface, looking for indicators of where ice is likely located across the planet. Data from spectrography and visual imagery also was utilized in the project.

“We can see ice on the surface at Mars’ poles, and we’re beginning to understand how much is buried under the subsurface at lower latitudes as well,” said Bramson, an assistant professor in the Department of Earth, Atmospheric, and Planetary Sciences in Purdue University’s College of Science . “But we still don’t have a good understanding of how much subsurface ice could be on the Earth’s moon.”

Bramson’s findings will offer early ideas about where future habitats on both the moon and Mars could be located, in terms of use by astronauts as well as travel capabilities; water can be used as part of the fuel for rockets.

Mars has two large ice caps at its poles, which combined contain about the same amount of ice as Greenland. But winter at Mars’ poles lasts several months without sunlight, making temperatures at the poles less than favorable. Areas with signs of widespread ice beneath the surface were found in the middle latitudes of Mars’ northern hemisphere, making those areas more hospitable for future human habitats.

Bramson’s work is already moving from examining radar findings provided by orbiting spacecraft to exploring the potential of mobile cartlike radar systems on the ground.

“These systems can send radio waves into the ground and then listen for a signal to bounce off of materials in the subsurface,” she said. “The systems can help us learn about what is in the subsurface without having to use destructive techniques to find it.”

Research plans in 2024 include testing different versions of the land-based radar systems in Iceland by searching for buried snow deposits covered by ash from volcanic eruptions. Additional testing at Purdue will evaluate the radar’s capabilities to measure layers of ice and dust in simulated Martian conditions, using a walk-in freezer at negative 20 degrees Celsius (negative 4 degrees Fahrenheit).

Bramson will be presenting at two international conferences later this year about Mars’ ice layers and how they form. She was also honored by NASA recently as part of the Measurement Definition Team of the International Mars Ice Mapper mission concept, and was selected by the National Academies of Sciences to study the science that could be accomplished by astronauts on Mars.

Bramson’s research is funded through NASA’s Mars Data Analysis, Lunar Data Analysis and Solar System Workings programs. 

The potential for subsurface ice on the moon is seemingly low compared to other planets. Bramson said the current thinking is that there may be only 3% ice within and under the surface. That’s compared to red-hot Mercury — the closest planet to the sun — where large ice deposits have been detected.

“Even though Mercury is so close to the sun that it’s super, super hot, there are areas within craters near the poles that never see the direct sunlight because they’re in permanent shadow, and that’s cold enough to actually retain massive ice deposits,” Bramson said. “We initially thought the moon would be similar, since it has similar permanently shadowed regions, but it seems like there’s not these massive ice deposits like Mercury has.”

Bramson said the presence or lack of ice deposits on Mars and the moon raises a number of questions for her. 

“It’s really interesting from a science point of view to understand what are the conditions that led to ice being present in various locations on different planetary bodies,” Bramson said. “Today, some of these latitudes of Mars are too warm to form these ice deposits there. This ice, therefore, represents a record of the climate conditions on Mars in the past. Meanwhile, differences in the ice deposits on Mercury and the moon may tell us about different mechanisms that bring water to these objects in our solar system.”

About Purdue University

Purdue University is a public research institution demonstrating excellence at scale. Ranked among top 10 public universities and with two colleges in the top four in the United States, Purdue discovers and disseminates knowledge with a quality and at a scale second to none. More than 105,000 students study at Purdue across modalities and locations, including nearly 50,000 in person on the West Lafayette campus. Committed to affordability and accessibility, Purdue’s main campus has frozen tuition 13 years in a row. See how Purdue never stops in the persistent pursuit of the next giant leap — including its first comprehensive urban campus in Indianapolis, the new Mitchell E. Daniels, Jr. School of Business, and Purdue Computes — at https://www.purdue.edu/president/strategic-initiatives .

Writer/Media contact: Brian Huchel, [email protected]

Source: Ali Bramson, [email protected]

Research News

Communication.

• OneCampus Portal
• Brightspace
• BoilerConnect
• Faculty and Staff
• Human Resources
• Colleges and Schools

Info for Staff

• Purdue Moves
• Board of Trustees
• University Senate
• Center for Healthy Living
• Information Technology
• Ethics & Compliance
• Campus Disruptions

Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, (765) 494-4600

© 2015-24 Purdue University | An equal access/equal opportunity university | Copyright Complaints | Maintained by Office of Strategic Communications

Trouble with this page? Disability-related accessibility issue? Please contact News Service at [email protected] .