importance of pilot study in research methodology

Why Is a Pilot Study Important in Research?

Are you working on a new research project ? We know that you are excited to start, but before you dive in, make sure your study is feasible. You don’t want to end up having to process too many samples at once or realize you forgot to add an essential question to your questionnaire.

What is a Pilot Study?

You can determine the feasibility of your research design, with a pilot study before you start. This is a preliminary, small-scale “rehearsal” in which you test the methods you plan to use for your research project. You will use the results to guide the methodology of your large-scale investigation. Pilot studies should be performed for both qualitative and quantitative studies. Here, we discuss the importance of the pilot study and how it will save you time, frustration and resources.

“ You never test the depth of a river with both feet ” – African proverb

Components of a Pilot Study

Whether your research is a clinical trial of a medical treatment or a survey in the form of a questionnaire, you want your study to be informative and add value to your research field. Things to consider in your pilot study include:

• Sample size and selection. Your data needs to be representative of the target study population. You should use statistical methods to estimate the feasibility of your sample size.
• Determine the criteria for a successful pilot study based on the objectives of your study. How will your pilot study address these criteria?
• When recruiting subjects or collecting samples ensure that the process is practical and manageable.
• Always test the measurement instrument . This could be a questionnaire, equipment, or methods used. Is it realistic and workable? How can it be improved?
• Data entry and analysis . Run the trial data through your proposed statistical analysis to see whether your proposed analysis is appropriate for your data set.
• Create a flow chart of the process.

How to Conduct a Pilot Study

Conducting a pilot study is an essential step in many research projects. Here’s a general guide on how to conduct a pilot study:

Step 1: Define Objectives

Inspect what specific aspects of your main study do you want to test or evaluate in your pilot study.

Step 2: Evaluate Sample Size

Decide on an appropriate sample size for your pilot study. This can be smaller than your main study but should still be large enough to provide meaningful feedback.

Step 3: Select Participants

Choose participants who are similar to those you’ll include in the main study. Ensure they match the demographics and characteristics of your target population.

Step 4: Prepare Materials

Develop or gather all the materials needed for the study, such as surveys, questionnaires, protocols, etc.

Step 5: Explain the Purpose of the Study

Briefly explain the purpose and implementation method of the pilot study to participants. Pay attention to the study duration to help you refine your timeline for the main study.

Step 6: Gather Feedback

Gather feedback from participants through surveys, interviews, or discussions. Ask about their understanding of the questions, clarity of instructions, time taken, etc.

 Step 7: Analyze Results

Analyze the collected data and identify any trends or patterns. Take note of any unexpected issues, confusion, or problems that arise during the pilot.

Step 8: Report Findings

Write a brief report detailing the process, results, and any changes made.

Based on the results observed in the pilot study, make necessary adjustments to your study design, materials, procedures, etc. Furthermore, ensure you are following ethical guidelines for research, even in a pilot study.

Ready to test your understanding on conducting a pilot study? Take our short quiz today!

Fill the Details to Check Your Score

Importance of Pilot Study in Research

Pilot studies should be routinely incorporated into research design s because they:

• Help define the research question
• Test the proposed study design and process. This could alert you to issues which may negatively affect your project.
• Educate yourself on different techniques related to your study.
• Test the safety of the medical treatment in preclinical trials on a small number of participants. This is an essential step in clinical trials.
• Determine the feasibility of your study, so you don’t waste resources and time.
• Provide preliminary data that you can use to improve your chances for funding and convince stakeholders that you have the necessary skills and expertise to successfully carry out the research.

Are Pilot Studies Always Necessary?

We recommend pilot studies for all research. Scientific research does not always go as planned; therefore, you should optimize the process to minimize unforeseen events. Why risk disastrous and expensive mistakes that could have been discovered and corrected in a pilot study?

An Essential Component for Good Research Design

Pilot work not only gives you a chance to determine whether your project is feasible but also an opportunity to publish its results. You have an ethical and scientific obligation to get your information out to assist other researchers in making the most of their resources.

A successful pilot study does not ensure the success of a research project. However, it does help you assess your approach and practice the necessary techniques required for your project. It will give you an indication of whether your project will work. Would you start a research project without a pilot study? Let us know in the comments section below.

But it depends on the nature of the research, I suppose.

Awesome document

Good document

I totally agree with this article that pilot study helps the researcher be sure how feasible his research idea is. And is well worth the time, as it saves future time wastage.

Great article, it is always wise to carry out that test before putting out the Main stuff. It saves you time and future embarrasment.

I think that pilot study is a great way to avoid mistakes on a large scale. You can’t go wrong doing this cause there will always be some error that will arise in scientific researches.

Rate this article Cancel Reply

Your email address will not be published.

Enago Academy's Most Popular Articles

• Reporting Research

Choosing the Right Analytical Approach: Thematic analysis vs. content analysis for data interpretation

In research, choosing the right approach to understand data is crucial for deriving meaningful insights.…

Comparing Cross Sectional and Longitudinal Studies: 5 steps for choosing the right approach

The process of choosing the right research design can put ourselves at the crossroads of…

• Career Corner

Unlocking the Power of Networking in Academic Conferences

Embarking on your first academic conference experience? Fear not, we got you covered! Academic conferences…

Research Recommendations – Guiding policy-makers for evidence-based decision making

Research recommendations play a crucial role in guiding scholars and researchers toward fruitful avenues of…

• AI in Academia

Disclosing the Use of Generative AI: Best practices for authors in manuscript preparation

The rapid proliferation of generative and other AI-based tools in research writing has ignited an…

Intersectionality in Academia: Dealing with diverse perspectives

Meritocracy and Diversity in Science: Increasing inclusivity in STEM education

Avoiding the AI Trap: Pitfalls of relying on ChatGPT for PhD applications

Sign-up to read more

Subscribe for free to get unrestricted access to all our resources on research writing and academic publishing including:

• 2000+ blog articles
• 50+ Webinars
• 10+ Expert podcasts
• 50+ Infographics
• 10+ Checklists
• Research Guides

We hate spam too. We promise to protect your privacy and never spam you.

I am looking for Editing/ Proofreading services for my manuscript Tentative date of next journal submission:

As a researcher, what do you consider most when choosing an image manipulation detector?

Pilot Study in Research: Definition & Examples

Julia Simkus

Editor at Simply Psychology

BA (Hons) Psychology, Princeton University

Julia Simkus is a graduate of Princeton University with a Bachelor of Arts in Psychology. She is currently studying for a Master's Degree in Counseling for Mental Health and Wellness in September 2023. Julia's research has been published in peer reviewed journals.

Learn about our Editorial Process

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

A pilot study, also known as a feasibility study, is a small-scale preliminary study conducted before the main research to check the feasibility or improve the research design.

Pilot studies can be very important before conducting a full-scale research project, helping design the research methods and protocol.

How Does it Work?

Pilot studies are a fundamental stage of the research process. They can help identify design issues and evaluate a study’s feasibility, practicality, resources, time, and cost before the main research is conducted.

It involves selecting a few people and trying out the study on them. It is possible to save time and, in some cases, money by identifying any flaws in the procedures designed by the researcher.

A pilot study can help the researcher spot any ambiguities (i.e., unusual things), confusion in the information given to participants, or problems with the task devised.

Sometimes the task is too hard, and the researcher may get a floor effect because none of the participants can score at all or can complete the task – all performances are low.

The opposite effect is a ceiling effect, when the task is so easy that all achieve virtually full marks or top performances and are “hitting the ceiling.”

This enables researchers to predict an appropriate sample size, budget accordingly, and improve the study design before performing a full-scale project.

Pilot studies also provide researchers with preliminary data to gain insight into the potential results of their proposed experiment.

However, pilot studies should not be used to test hypotheses since the appropriate power and sample size are not calculated. Rather, pilot studies should be used to assess the feasibility of participant recruitment or study design.

By conducting a pilot study, researchers will be better prepared to face the challenges that might arise in the larger study. They will be more confident with the instruments they will use for data collection.

Multiple pilot studies may be needed in some studies, and qualitative and/or quantitative methods may be used.

To avoid bias, pilot studies are usually carried out on individuals who are as similar as possible to the target population but not on those who will be a part of the final sample.

Feedback from participants in the pilot study can be used to improve the experience for participants in the main study. This might include reducing the burden on participants, improving instructions, or identifying potential ethical issues.

Experiment Pilot Study

In a pilot study with an experimental design , you would want to ensure that your measures of these variables are reliable and valid.

You would also want to check that you can effectively manipulate your independent variables and that you can control for potential confounding variables.

A pilot study allows the research team to gain experience and training, which can be particularly beneficial if new experimental techniques or procedures are used.

Questionnaire Pilot Study

It is important to conduct a questionnaire pilot study for the following reasons:

• Check that respondents understand the terminology used in the questionnaire.
• Check that emotive questions are not used, as they make people defensive and could invalidate their answers.
• Check that leading questions have not been used as they could bias the respondent’s answer.
• Ensure that the questionnaire can be completed in a reasonable amount of time. If it’s too long, respondents may lose interest or not have enough time to complete it, which could affect the response rate and the data quality.

By identifying and addressing issues in the pilot study, researchers can reduce errors and risks in the main study. This increases the reliability and validity of the main study’s results.

Assessing the practicality and feasibility of the main study

Testing the efficacy of research instruments

Identifying and addressing any weaknesses or logistical problems

Collecting preliminary data

Estimating the time and costs required for the project

Determining what resources are needed for the study

Identifying the necessity to modify procedures that do not elicit useful data

Adding credibility and dependability to the study

Pretesting the interview format

Enabling researchers to develop consistent practices and familiarize themselves with the procedures in the protocol

Addressing safety issues and management problems

Limitations

Require extra costs, time, and resources.

Do not guarantee the success of the main study.

Contamination (ie: if data from the pilot study or pilot participants are included in the main study results).

Funding bodies may be reluctant to fund a further study if the pilot study results are published.

Do not have the power to assess treatment effects due to small sample size.

• Viscocanalostomy: A Pilot Study (Carassa, Bettin, Fiori, & Brancato, 1998)
• WHO International Pilot Study of Schizophrenia (Sartorius, Shapiro, Kimura, & Barrett, 1972)
• Stephen LaBerge of Stanford University ran a series of experiments in the 80s that investigated lucid dreaming. In 1985, he performed a pilot study that demonstrated that time perception is the same as during wakefulness. Specifically, he had participants go into a state of lucid dreaming and count out ten seconds, signaling the start and end with pre-determined eye movements measured with the EOG.
• Negative Word-of-Mouth by Dissatisfied Consumers: A Pilot Study (Richins, 1983)
• A pilot study and randomized controlled trial of the mindful self‐compassion program (Neff & Germer, 2013)
• Pilot study of secondary prevention of posttraumatic stress disorder with propranolol (Pitman et al., 2002)
• In unstructured observations, the researcher records all relevant behavior without a system. There may be too much to record, and the behaviors recorded may not necessarily be the most important, so the approach is usually used as a pilot study to see what type of behaviors would be recorded.
• Perspectives of the use of smartphones in travel behavior studies: Findings from a literature review and a pilot study (Gadziński, 2018)

Further Information

• Lancaster, G. A., Dodd, S., & Williamson, P. R. (2004). Design and analysis of pilot studies: recommendations for good practice. Journal of evaluation in clinical practice, 10 (2), 307-312.
• Thabane, L., Ma, J., Chu, R., Cheng, J., Ismaila, A., Rios, L. P., … & Goldsmith, C. H. (2010). A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology, 10 (1), 1-10.
• Moore, C. G., Carter, R. E., Nietert, P. J., & Stewart, P. W. (2011). Recommendations for planning pilot studies in clinical and translational research. Clinical and translational science, 4 (5), 332-337.

Carassa, R. G., Bettin, P., Fiori, M., & Brancato, R. (1998). Viscocanalostomy: a pilot study. European journal of ophthalmology, 8 (2), 57-61.

Gadziński, J. (2018). Perspectives of the use of smartphones in travel behaviour studies: Findings from a literature review and a pilot study. Transportation Research Part C: Emerging Technologies, 88 , 74-86.

In J. (2017). Introduction of a pilot study. Korean Journal of Anesthesiology, 70 (6), 601–605. https://doi.org/10.4097/kjae.2017.70.6.601

LaBerge, S., LaMarca, K., & Baird, B. (2018). Pre-sleep treatment with galantamine stimulates lucid dreaming: A double-blind, placebo-controlled, crossover study. PLoS One, 13 (8), e0201246.

Leon, A. C., Davis, L. L., & Kraemer, H. C. (2011). The role and interpretation of pilot studies in clinical research. Journal of psychiatric research, 45 (5), 626–629. https://doi.org/10.1016/j.jpsychires.2010.10.008

Malmqvist, J., Hellberg, K., Möllås, G., Rose, R., & Shevlin, M. (2019). Conducting the Pilot Study: A Neglected Part of the Research Process? Methodological Findings Supporting the Importance of Piloting in Qualitative Research Studies. International Journal of Qualitative Methods. https://doi.org/10.1177/1609406919878341

Neff, K. D., & Germer, C. K. (2013). A pilot study and randomized controlled trial of the mindful self‐compassion program. Journal of Clinical Psychology, 69 (1), 28-44.

Pitman, R. K., Sanders, K. M., Zusman, R. M., Healy, A. R., Cheema, F., Lasko, N. B., … & Orr, S. P. (2002). Pilot study of secondary prevention of posttraumatic stress disorder with propranolol. Biological psychiatry, 51 (2), 189-192.

Richins, M. L. (1983). Negative word-of-mouth by dissatisfied consumers: A pilot study. Journal of Marketing, 47 (1), 68-78.

Sartorius, N., Shapiro, R., Kimura, M., & Barrett, K. (1972). WHO International Pilot Study of Schizophrenia1. Psychological medicine, 2 (4), 422-425.

Teijlingen, E. R; V. Hundley (2001). The importance of pilot studies, Social research UPDATE, (35)

Related Articles

Research Methodology

Qualitative Data Coding

What Is a Focus Group?

Cross-Cultural Research Methodology In Psychology

What Is Internal Validity In Research?

Research Methodology , Statistics

What Is Face Validity In Research? Importance & How To Measure

Criterion Validity: Definition & Examples

• Open access
• Published: 06 January 2010

A tutorial on pilot studies: the what, why and how

• Lehana Thabane 1 , 2 ,
• Jinhui Ma 1 , 2 ,
• Rong Chu 1 , 2 ,
• Ji Cheng 1 , 2 ,
• Afisi Ismaila 1 , 3 ,
• Lorena P Rios 1 , 2 ,
• Reid Robson 3 ,
• Marroon Thabane 1 , 4 ,
• Lora Giangregorio 5 &
• Charles H Goldsmith 1 , 2  

BMC Medical Research Methodology volume  10 , Article number:  1 ( 2010 ) Cite this article

356k Accesses

1580 Citations

104 Altmetric

Metrics details

A Correction to this article was published on 11 March 2023

This article has been updated

Pilot studies for phase III trials - which are comparative randomized trials designed to provide preliminary evidence on the clinical efficacy of a drug or intervention - are routinely performed in many clinical areas. Also commonly know as "feasibility" or "vanguard" studies, they are designed to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre trials; to increase clinical experience with the study medication or intervention for the phase III trials. They are the best way to assess feasibility of a large, expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. The objective of this paper is to provide a detailed examination of the key aspects of pilot studies for phase III trials including: 1) the general reasons for conducting a pilot study; 2) the relationships between pilot studies, proof-of-concept studies, and adaptive designs; 3) the challenges of and misconceptions about pilot studies; 4) the criteria for evaluating the success of a pilot study; 5) frequently asked questions about pilot studies; 7) some ethical aspects related to pilot studies; and 8) some suggestions on how to report the results of pilot investigations using the CONSORT format.

1. Introduction

The Concise Oxford Thesaurus [ 1 ] defines a pilot project or study as an experimental, exploratory, test, preliminary, trial or try out investigation. Epidemiology and statistics dictionaries provide similar definitions of a pilot study as a small scale

" ... test of the methods and procedures to be used on a larger scale if the pilot study demonstrates that the methods and procedures can work" [ 2 ];

"...investigation designed to test the feasibility of methods and procedures for later use on a large scale or to search for possible effects and associations that may be worth following up in a subsequent larger study" [ 3 ].

Table 1 provides a summary of definitions found on the Internet. A closer look at these definitions reveals that they are similar to the ones above in that a pilot study is synonymous with a feasibility study intended to guide the planning of a large-scale investigation. Pilot studies are sometimes referred to as "vanguard trials" (i.e. pre-studies) intended to assess the safety of treatment or interventions; to assess recruitment potential; to assess the feasibility of international collaboration or coordination for multicentre trials; to evaluate surrogate marker data in diverse patient cohorts; to increase clinical experience with the study medication or intervention, and identify the optimal dose of treatments for the phase III trials [ 4 ]. As suggested by an African proverb from the Ashanti people in Ghana " You never test the depth of a river with both feet ", the main goal of pilot studies is to assess feasibility so as to avoid potentially disastrous consequences of embarking on a large study - which could potentially "drown" the whole research effort.

Feasibility studies are routinely performed in many clinical areas. It is fair to say that every major clinical trial had to start with some piloting or a small scale investigation to assess the feasibility of conducting a larger scale study: critical care [ 5 ], diabetes management intervention trials [ 6 ], cardiovascular trials [ 7 ], primary healthcare [ 8 ], to mention a few.

Despite their noted importance, the reality is that pilot studies receive little or no attention in scientific research training. Few epidemiology or research textbooks cover the topic with the necessary detail. In fact, we are not aware of any textbook that dedicates a chapter on this issue - many just mention it in passing or provide a cursory coverage of the topic. The objective of this paper is to provide a detailed examination of the key aspects of pilot studies. In the next section, we narrow the focus of our definition of a pilot to phase III trials. Section 3 covers the general reasons for conducting a pilot study. Section 4 deals with the relationships between pilot studies, proof-of-concept studies, and adaptive designs, while section 5 addresses the challenges of pilot studies. Evaluation of a pilot study (i.e. how to determine if a pilot study was successful) is covered in Section 6. We deal with several frequently asked questions about pilot studies in Section 7 using a "question-and-answer" approach. Section 8 covers some ethical aspects related to pilot studies; and in Section 9, we follow the CONSORT format [ 9 ] to offer some suggestions on how to report the results of pilot investigations.

2. Narrowing the focus: Pilot studies for randomized studies

Pilot studies can be conducted in both quantitative and qualitative studies. Adopting a similar approach to Lancaster et al . [ 10 ], we focus on quantitative pilot studies - particularly those done prior to full-scale phase III trials. Phase I trials are non-randomized studies designed to investigate the pharmacokinetics of a drug (i.e. how a drug is distributed and metabolized in the body) including finding a dose that can be tolerated with minimal toxicity. Phase II trials provide preliminary evidence on the clinical efficacy of a drug or intervention. They may or may not be randomized. Phase III trials are randomized studies comparing two or more drugs or intervention strategies to assess efficacy and safety. Phase IV trials, usually done after registration or marketing of a drug, are non-randomized surveillance studies to document experiences (e.g. side-effects, interactions with other drugs, etc) with using the drug in practice.

For the purposes of this paper, our approach to utilizing pilot studies relies on the model for complex interventions advocated by the British Medical Research Council - which explicitly recommends the use of feasibility studies prior to Phase III clinical trials, but stresses the iterative nature of the processes of development, feasibility and piloting, evaluation and implementation [ 11 ].

3. Reasons for Conducting Pilot Studies

Van Teijlingen et al . [ 12 ] and van Teijlingen and Hundley [ 13 ] provide a summary of the reasons for performing a pilot study. In general, the rationale for a pilot study can be grouped under several broad classifications - process, resources, management and scientific (see also http://www.childrens-mercy.org/stats/plan/pilot.asp for a different classification):

Process: This assesses the feasibility of the steps that need to take place as part of the main study. Examples include determining recruitment rates, retention rates, etc.

Resources: This deals with assessing time and budget problems that can occur during the main study. The idea is to collect some pilot data on such things as the length of time to mail or fill out all the survey forms.

Management: This covers potential human and data optimization problems such as personnel and data management issues at participating centres.

Scientific: This deals with the assessment of treatment safety, determination of dose levels and response, and estimation of treatment effect and its variance.

Table 2 summarizes this classification with specific examples.

4. Relationships between Pilot Studies, Proof-of-Concept Studies, and Adaptive Designs

A proof-of-concept (PoC) study is defined as a clinical trial carried out to determine if a treatment (drug) is biologically active or inactive [ 14 ]. PoC studies usually use surrogate markers as endpoints. In general, they are phase I/II studies - which, as noted above, investigate the safety profile, dose level and response to new drugs [ 15 ]. Thus, although designed to inform the planning of phase III trials for registration or licensing of new drugs, PoC studies may not necessarily fit our restricted definition of pilot studies aimed at assessing feasibility of phase III trials as outlined in Section 2.

An adaptive trial design refers to a design that allows modifications to be made to a trial's design or statistical procedures during its conduct, with the purpose of efficiently identifying clinical benefits/risks of new drugs or to increase the probability of success of clinical development [ 16 ]. The adaptations can be prospective (e.g. stopping a trial early due to safety or futility or efficacy at interim analysis); concurrent (e.g. changes in eligibility criteria, hypotheses or study endpoints) or retrospective (e.g. changes to statistical analysis plan prior to locking database or revealing treatment codes to trial investigators or patients). Piloting is normally built into adaptive trial designs by determining a priori decision rules to guide the adaptations based on cumulative data. For example, data from interim analyses could be used to refine sample size calculations [ 17 , 18 ]. This approach is routinely used in internal pilot studies - which are primarily designed to inform sample size calculation for the main study, with recalculation of the sample size as the key adaptation. Unlike other phase III pilots, an internal pilot investigation does not usually address any other feasibility aspects - because it is essentially part of the main study [ 10 , 19 , 20 ]..

Nonetheless, we need to emphasize that whether or not a study is a pilot, depends on its objectives. An adaptive method is used as a strategy to reach that objective. Both a pilot and a non-pilot could be adaptive.

5. Challenges of and Common Misconceptions about Pilot Studies

Pilot studies can be very informative, not only to the researchers conducting them but also to others doing similar work. However, many of them never get published, often because of the way the results are presented [ 13 ]. Quite often the emphasis is wrongly placed on statistical significance, not on feasibility - which is the main focus of the pilot study. Our experience in reviewing submissions to a research ethics board also shows that most of the pilot projects are not well designed: i.e. there are no clear feasibility objectives; no clear analytic plans; and certainly no clear criteria for success of feasibility.

In many cases, pilot studies are conducted to generate data for sample size calculations. This seems especially sensible in situations where there are no data from previous studies to inform this process. However, it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic/biased because of the limited sample sizes. Therefore if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [ 21 ] -- particularly if the pilot study was done to see if there is likely to be a treatment effect in the main study. In section 6, we provide guidance on how to proceed with caution in this regard.

There are also several misconceptions about pilot studies. Below are some of the common reasons that researchers have put forth for calling their study a pilot.

The first common reason is that a pilot study is a small single-centre study. For example, researchers often state lack of resources for a large multi-centre study as a reason for doing a pilot. The second common reason is that a pilot investigation is a small study that is similar in size to someone else's published study. In reviewing submissions to a research ethics board, we have come across sentiments such as

So-and-so did a similar study with 6 patients and got statistical significance - ours uses 12 patients (double the size)!

We did a similar pilot before (and it was published!)

The third most common reason is that a pilot is a small study done by a student or an intern - which can be completed quickly and does not require funding. Specific arguments include

I have funding for 10 patients only;

I have limited seed (start-up) funding;

This is just a student project!

My supervisor (boss) told me to do it as a pilot .

None of the above arguments qualifies as sound reasons for calling a study a pilot. A study should only be conducted if the results will be informative; studies conducted for the reasons above may result in findings of limited utility, which would be a waste of the researchers' and participants' efforts. The focus of a pilot study should be on assessment of feasibility, unless it was powered appropriately to assess statistical significance. Further, there is a vast number of poorly designed and reported studies. Assessment of the quality of a published report may be helpful to guide decisions of whether the report should be used to guide planning or designing of new studies. Finally, if a trainee or researcher is assigned a project as a pilot it is important to discuss how the results will inform the planning of the main study. In addition, clearly defined feasibility objectives and rationale to justify piloting should be provided.

Sample Size for Pilot Studies

In general, sample size calculations may not be required for some pilot studies. It is important that the sample for a pilot be representative of the target study population. It should also be based on the same inclusion/exclusion criteria as the main study. As a rule of thumb, a pilot study should be large enough to provide useful information about the aspects that are being assessed for feasibility. Note that PoC studies require sample size estimation based on surrogate markers [ 22 ], but they are usually not powered to detect meaningful differences in clinically important endpoints. The sample used in the pilot may be included in the main study, but caution is needed to ensure the key features of the main study are preserved in the pilot (e.g. blinding in randomized controlled trials). We recommend if any pooling of pilot and main study data is considered, this should be planned beforehand, described clearly in the protocol with clear discussion of the statistical consequences and methods. The goal is to avoid or minimize the potential bias that may occur due to multiple testing issues or any other opportunistic actions by investigators. In general, pooling when done appropriately can increase the efficiency of the main study [ 23 ].

As noted earlier, a carefully designed pilot study may be used to generate information for sample size calculations. Two approaches may be helpful to optimize information from a pilot study in this context: First, consider eliciting qualitative data to supplement the quantitative information obtained in the pilot. For example, consider having some discussions with clinicians using the approach suggested by Lenth [ 24 ] to illicit additional information on possible effect size and variance estimates. Second, consider creating a sample size table for various values of the effect or variance estimates to acknowledge the uncertainty surrounding the pilot estimates.

In some cases, one could use a confidence interval [CI] approach to estimate the sample size required to establish feasibility. For example, suppose we had a pilot trial designed primarily to determine adherence rates to the standardized risk assessment form to enhance venous thromboprophylaxis in hospitalized patients. Suppose it was also decided a priori that the criterion for success would be: the main trial would be ' feasibl e' if the risk assessment form is completed for ≥ 70% of eligible hospitalized patients.

6. How to Interpret the Results of a Pilot Study: Criteria for Success

It is always important to state the criteria for success of a pilot study. The criteria should be based on the primary feasibility objectives. These provide the basis for interpreting the results of the pilot study and determining whether it is feasible to proceed to the main study. In general, the outcome of a pilot study can be one of the following: (i) Stop - main study not feasible; (ii) Continue, but modify protocol - feasible with modifications; (iii) Continue without modifications, but monitor closely - feasible with close monitoring and (iv) Continue without modifications - feasible as is.

For example, the Prophylaxis of Thromboembolism in Critical Care Trial (PROTECT) was designed to assess the feasibility of a large-scale trial with the following criteria for determining success [ 25 ]:

98.5% of patients had to receive study drug within 12 hours of randomization;

91.7% of patients had to receive every scheduled dose of the study drug in a blinded manner;

90% or more of patients had to have lower limb compression ultrasounds performed at the specified times; and

> 90% of necessary dose adjustments had to have been made appropriately in response to pre-defined laboratory criteria .

In a second example, the PeriOperative Epidural Trial (POET) Pilot Study was designed to assess the feasibility of a large, multicentre trial with the following criteria for determining success [ 26 ]:

one subject per centre per week (i.e., 200 subjects from four centres over 50 weeks) can be recruited ;

at least 70% of all eligible patients can be recruited ;

no more than 5% of all recruited subjects crossed over from one modality to the other; and

complete follow-up in at least 95% of all recruited subjects .

7. Frequently asked questions about pilot studies

In this Section, we offer our thoughts on some of the frequently asked questions about pilot studies. These could be helpful to not only clinicians and trainees, but to anyone who is interested in health research.

Can I publish the results of a pilot study?

- Yes, every attempt should be made to publish.

Why is it important to publish the results of pilot studies?

- To provide information about feasibility to the research community to save resources being unnecessarily spent on studies that may not be feasible. Further, having such information can help researchers to avoid duplication of efforts in assessing feasibility.

- Finally, researchers have an ethical and scientific obligation to attempt publishing the results of every research endeavor. However, our focus should be on feasibility goals. Emphasis should not be placed on statistical significance when pilot studies are not powered to detect minimal clinically important differences. Such studies typically do not show statistically significant results - remember that underpowered studies (with no statistically significant results) are inconclusive, not negative since "no evidence of effect" is not "evidence of no effect" [ 27 ].

Can I combine data from a pilot with data from the main study?

- Yes, provided the sampling frame and methodologies are the same. This can increase the efficiency of the main study - see Section 5.

Can I combine the results of a pilot with the results of another study or in a meta-analysis?

- Yes, provided the sampling frame and methodologies are the same.

- No, if the main study is reported and it includes the pilot study.

Can the results of the pilot study be valid on their own, without existence of the main study

- Yes, if the results show that it is not feasible to proceed to the main study or there is insufficient funding.

Can I apply for funding for a pilot study?

- Yes. Like any grant, it is important to justify the need for piloting.

- The pilot has to be placed in the context of the main study.

Can I randomize patients in a pilot study?

- Yes. For a phase III pilot study, one of the goals could be to assess how a randomization procedure might work in the main study or whether the idea of randomization might be acceptable to patients [ 10 ]. In general, it is always best for a pilot to maintain the same design as the main study.

How can I use the information from a pilot to estimate the sample size?

- Use with caution, as results from pilot studies can potentially mislead sample size calculations.

- Consider supplementing the information with qualitative discussions with clinicians - see section 5; and

- Create a sample size table to acknowledge the uncertainty of the pilot information - see section 5.

Can I use the results of a pilot study to treat my patients?

- Not a good idea!

- Pilot studies are primarily for assessing feasibility.

What can I do with a failed or bad pilot study?

- No study is a complete failure; it can always be used as bad example! However, it is worth making clear that a pilot study that shows the main study is not likely to be feasible is not a failed (pilot) study. In fact, it is a success - because you avoided wasting scarce resources on a study destined for failure!

8. Ethical Aspects of Pilot Studies

Halpern et al . [ 28 ] stated that conducting underpowered trials is unethical. However, they proposed that underpowered trials are ethical in two situations: (i) small trials of interventions for rare diseases -- which require documenting explicit plans for including results with those of similar trials in a prospective meta-analysis; (ii) early-phase trials in the development of drugs or devices - provided they are adequately powered for defined purposes other than randomized treatment comparisons. Pilot studies of phase III trials (dealing with common diseases) are not addressed in their proposal. It is therefore prudent to ask: Is it ethical to conduct a study whose feasibility can not be guaranteed (i.e. with a high probability of success)?

It seems unethical to consider running a phase III study without having sufficient data or information about the feasibility. In fact, most granting agencies often require data on feasibility as part of their assessment of the scientific validity for funding decisions.

There is however one important ethical aspect about pilot studies that has received little or no attention from researchers, research ethics boards and ethicists alike. This pertains to the issue of the obligation that researchers have to patients or participants in a trial to disclose the feasibility nature of pilot studies. This is essential given that some pilot studies may not lead to further studies. A review of the commonly cited research ethics guidelines - the Nuremburg Code [ 29 ], Helsinki Declaration [ 30 ], the Belmont Report [ 31 ], ICH Good Clinical Practice [ 32 ], and the International Ethical Guidelines for Biomedical Research Involving Human Subjects [ 33 ] - shows that pilot studies are not addressed in any of these guidelines. Canadian researchers are also encouraged to follow the Tri-Council Policy Statement (TCPS) [ 34 ] - it too does not address how pilot studies need to be approached. It seems to us that given the special nature of feasibility or pilot studies, the disclosure of their purpose to study participants requires special wording - that informs them of the definition of a pilot study, the feasibility objectives of the study, and also clearly defines the criteria for success of feasibility. To fully inform participants, we suggest using the following wording in the consent form:

" The overall purpose of this pilot study is to assess the feasibility of conducting a large study to [state primary objective of the main study]. A feasibility or pilot study is a study that... [state a general definition of a feasibility study]. The specific feasibility objectives of this study are ... [state the specific feasibility objectives of the pilot study]. We will determine that it is feasible to carry on the main study if ... [state the criteria for success of feasibility] ."

9. Recommendation for Reporting the Results of Pilot Studies

Adopted from the CONSORT Statement [ 9 ], Table 3 provides a checklist of items to consider including in a report of a pilot study.

Title and abstract

Item #1: the title or abstract should indicate that the study is a "pilot" or "feasibility".

As a number one summary of the contents of any report, it is important for the title to clearly indicate that the report is for a pilot or feasibility study. This would also be helpful to other researchers during electronic information search about feasibility issues. Our quick search of PUBMED [on July 13, 2009], using the terms "pilot" OR "feasibility" OR "proof-of-concept" for revealed 24423 (16%) hits of studies that had these terms in the title or abstract compared with 149365 hits that had these terms anywhere in the text.

Item #2: Scientific background for the main study and explanation of rationale for assessing feasibility through piloting

The rationale for initiating a pilot should be based on the need to assess feasibility for the main study. Thus, the background of the main study should clearly describe what is known or not known about important feasibility aspects to provide context for piloting.

Item #3: Participants and setting of the study

The description of the inclusion-exclusion or eligibility criteria for participants should be the same as in the main study. The settings and locations where the data were collected should also be clearly described.

Item #4: Interventions

Precise details of the interventions intended for each group and how and when they were actually administered (if applicable) - state clearly if any aspects of the intervention are assessed for feasibility.

Item #5: Objectives

State the specific scientific primary and secondary objectives and hypotheses for the main study and the specific feasibility objectives. It is important to clearly indicate the feasibility objectives as the primary focus for the pilot.

Item #6: Outcomes

Clearly define primary and secondary outcome measures for the main study. Then, clearly define the feasibility outcomes and how they were operationalized - these should include key elements such as recruitment rates, consent rates, completion rates, variance estimates, etc. In some cases, a pilot study may be conducted with the aim to determine a suitable (clinical or surrogate) endpoint for the main study. In such a case, one may not be able to define the primary outcome of the main study until the pilot is finished. However, it is important that determining the primary outcome of the main study be clearly stated as part of feasibility outcomes.

Item #7: Sample Size

Describe how sample size was determined. If the pilot is a proof-of-concept study, is the sample size calculated based on primary/key surrogate marker(s)? In general if the pilot is for a phase III study, there may be no need for a formal sample size calculation. However, the confidence interval approach may be used to calculate and justify the sample size based on key feasibility objective(s).

Item #8: Feasibility criteria

Clearly describe the criteria for assessing success of feasibility - these should be based on the feasibility objectives.

Item #9: Statistical Analysis

Describe the statistical methods for the analysis of primary and secondary feasibility outcomes.

Item #10: Ethical Aspects

State whether the study received research ethics approval. Describe how informed consent was handled - given the feasibility nature of the study.

Item #11: Participant Flow

Describe the flow of participants through each stage of the study (use of a flow-diagram is strongly recommended -- see CONSORT [ 9 ] for a template). Describe protocol deviations from pilot study as planned with reasons for deviations. State the number of exclusions at each stage and corresponding reasons for exclusions.

Item #12: Recruitment

Report the dates defining the periods of recruitment and follow-up.

Item #13: Baseline Data

Report the baseline demographic and clinical characteristics of the participants.

Item #14: Outcomes and Estimation

For each primary and secondary feasibility outcomes, report the point estimate of effect and its precision ( e.g ., 95% CI) - if applicable.

Item # 15: Interpretation

Interpretation of the results should focus on feasibility, taking into account the stated criteria for success of feasibility, study hypotheses, sources of potential bias or imprecision (given the feasibility nature of the study) and the dangers associated with multiplicity - repeated testing on multiple outcomes.

Item #16: Generalizability

Discuss the generalizability (external validity) of the feasibility aspects observed in the study. State clearly what modifications in the design of the main study (if any) would be necessary to make it feasible.

Item #17: Overall evidence of feasibility

Discuss the general results in the context of overall evidence of feasibility. It is important that the focus be on feasibility.

9. Conclusions

Pilot or vanguard studies provide a good opportunity to assess feasibility of large full-scale studies. Pilot studies are the best way to assess feasibility of a large expensive full-scale study, and in fact are an almost essential pre-requisite. Conducting a pilot prior to the main study can enhance the likelihood of success of the main study and potentially help to avoid doomed main studies. Pilot studies should be well designed with clear feasibility objectives, clear analytic plans, and explicit criteria for determining success of feasibility. They should be used cautiously for determining treatment effects and variance estimates for power or sample size calculations. Finally, they should be scrutinized the same way as full scale studies, and every attempt should be taken to publish the results in peer-reviewed journals.

Change history

11 march 2023.

A Correction to this paper has been published: https://doi.org/10.1186/s12874-023-01880-1

Waite M: Concise Oxford Thesaurus. 2002, Oxford, England: Oxford University Press, 2

Google Scholar  

Last JM, editor: A Dictionary of Epidemiology. 2001, Oxford University Press, 4

Everitt B: Medical Statistics from A to Z: A Guide for Clinicians and Medical Students. 2006, Cambridge University Press: Cambridge, 2

Book   Google Scholar  

Tavel JA, Fosdick L, ESPRIT Vanguard Group. ESPRIT Executive Committee: Closeout of four phase II Vanguard trials and patient rollover into a large international phase III HIV clinical endpoint trial. Control Clin Trials. 2001, 22: 42-48. 10.1016/S0197-2456(00)00114-8.

Article   CAS   PubMed   Google Scholar  

Arnold DM, Burns KE, Adhikari NK, Kho ME, Meade MO, Cook DJ: The design and interpretation of pilot trials in clinical research in critical care. Crit Care Med. 2009, 37 (Suppl 1): 69-74. 10.1097/CCM.0b013e3181920e33.

Article   Google Scholar  

Computerization of Medical Practice for the Enhancement of Therapeutic Effectiveness. Last accessed August 8, 2009, [ http://www.compete-study.com/index.htm ]

Heart Outcomes Prevention Evaluation Study. Last accessed August 8, 2009, [ http://www.ccc.mcmaster.ca/hope.htm ]

Cardiovascular Health Awareness Program. Last accessed August 8, 2009, [ http://www.chapprogram.ca/resources.html ]

Moher D, Schulz KF, Altman DG, CONSORT Group (Consolidated Standards of Reporting Trials): The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. J Am Podiatr Med Assoc. 2001, 91: 437-442.

Lancaster GA, Dodd S, Williamson PR: Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004, 10: 307-12. 10.1111/j..2002.384.doc.x.

Article   PubMed   Google Scholar  

Craig N, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337: a1655-10.1136/bmj.a1655.

Article   PubMed   PubMed Central   Google Scholar  

Van Teijlingen ER, Rennie AM, Hundley V, Graham W: The importance of conducting and reporting pilot studies: the example of the Scottish Births Survey. J Adv Nurs. 2001, 34: 289-295. 10.1046/j.1365-2648.2001.01757.x.

Van Teijlingen ER, Hundley V: The Importance of Pilot Studies. Social Research Update. 2001, 35-[ http://sru.soc.surrey.ac.uk/SRU35.html ]

Lawrence Gould A: Timing of futility analyses for 'proof of concept' trials. Stat Med. 2005, 24: 1815-1835. 10.1002/sim.2087.

Fardon T, Haggart K, Lee DK, Lipworth BJ: A proof of concept study to evaluate stepping down the dose of fluticasone in combination with salmeterol and tiotropium in severe persistent asthma. Respir Med. 2007, 101: 1218-1228. 10.1016/j.rmed.2006.11.001.

Chow SC, Chang M: Adaptive design methods in clinical trials - a review. Orphanet J Rare Dis. 2008, 3: 11-10.1186/1750-1172-3-11.

Gould AL: Planning and revising the sample size for a trial. Stat Med. 1995, 14: 1039-1051. 10.1002/sim.4780140922.

Coffey CS, Muller KE: Properties of internal pilots with the univariate approach to repeated measures. Stat Med. 2003, 22: 2469-2485. 10.1002/sim.1466.

Zucker DM, Wittes JT, Schabenberger O, Brittain E: Internal pilot studies II: comparison of various procedures. Statistics in Medicine. 1999, 18: 3493-3509. 10.1002/(SICI)1097-0258(19991230)18:24<3493::AID-SIM302>3.0.CO;2-2.

Kieser M, Friede T: Re-calculating the sample size in internal pilot designs with control of the type I error rate. Statistics in Medicine. 2000, 19: 901-911. 10.1002/(SICI)1097-0258(20000415)19:7<901::AID-SIM405>3.0.CO;2-L.

Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA: Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry. 2006, 63: 484-489. 10.1001/archpsyc.63.5.484.

Yin Y: Sample size calculation for a proof of concept study. J Biopharm Stat. 2002, 12: 267-276. 10.1081/BIP-120015748.

Wittes J, Brittain E: The role of internal pilot studies in increasing the efficiency of clinical trials. Stat Med. 1990, 9: 65-71. 10.1002/sim.4780090113.

Lenth R: Some Practical Guidelines for Effective Sample Size Determination. The American Statistician. 2001, 55: 187-193. 10.1198/000313001317098149.

Cook DJ, Rocker G, Meade M, Guyatt G, Geerts W, Anderson D, Skrobik Y, Hebert P, Albert M, Cooper J, Bates S, Caco C, Finfer S, Fowler R, Freitag A, Granton J, Jones G, Langevin S, Mehta S, Pagliarello G, Poirier G, Rabbat C, Schiff D, Griffith L, Crowther M, PROTECT Investigators. Canadian Critical Care Trials Group: Prophylaxis of Thromboembolism in Critical Care (PROTECT) Trial: a pilot study. J Crit Care. 2005, 20: 364-372. 10.1016/j.jcrc.2005.09.010.

Choi PT, Beattie WS, Bryson GL, Paul JE, Yang H: Effects of neuraxial blockade may be difficult to study using large randomized controlled trials: the PeriOperative Epidural Trial (POET) Pilot Study. PLoS One. 2009, 4 (2): e4644-10.1371/journal.pone.0004644.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Altman DG, Bland JM: Absence of evidence is not evidence of absence. BMJ. 1995, 311: 485-

Halpern SD, Karlawish JH, Berlin JA: The continuing unethical conduct of underpowered clinical trials. JAMA. 2002, 288: 358-362. 10.1001/jama.288.3.358.

The Nuremberg Code, Research ethics guideline 2005. Last accessed August 8, 2009, [ http://www.hhs.gov/ohrp/references/nurcode.htm ]

The Declaration of Helsinki, Research ethics guideline. Last accessed December 22, 2009, [ http://www.wma.net/en/30publications/10policies/b3/index.html ]

The Belmont Report, Research ethics guideline. Last accessed August 8, 2009, [ http://ohsr.od.nih.gov/guidelines/belmont.html ]

The ICH Harmonized Tripartite Guideline-Guideline for Good Clinical Practice. Last accessed August 8, 2009, [ http://www.gcppl.org.pl/ma_struktura/docs/ich_gcp.pdf ]

The International Ethical Guidelines for Biomedical Research Involving Human Subjects. Last accessed August 8, 2009, [ http://www.fhi.org/training/fr/Retc/pdf_files/cioms.pdf ]

Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, Government of Canada. Last accessed August 8, 2009, [ http://www.pre.ethics.gc.ca/english/policystatement/policystatement.cfm ]

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/10/1/prepub

Download references

Acknowledgements

Dr Lehana Thabane is clinical trials mentor for the Canadian Institutes of Health Research. We thank the reviewers for insightful comments and suggestions which led to improvements in the manuscript.

Author information

Authors and affiliations.

Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, ON, Canada

Lehana Thabane, Jinhui Ma, Rong Chu, Ji Cheng, Afisi Ismaila, Lorena P Rios, Marroon Thabane & Charles H Goldsmith

Biostatistics Unit, St Joseph's Healthcare Hamilton, Hamilton, ON, Canada

Lehana Thabane, Jinhui Ma, Rong Chu, Ji Cheng, Lorena P Rios & Charles H Goldsmith

Department of Medical Affairs, GlaxoSmithKline Inc., Mississauga, ON, Canada

Afisi Ismaila & Reid Robson

Department of Medicine, Division of Gastroenterology, McMaster University, Hamilton, ON, Canada

Marroon Thabane

Department of Kinesiology, University of Waterloo, Waterloo, ON, Canada

Lora Giangregorio

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Lehana Thabane .

Additional information

Competing interests.

The authors declare that they have no competing interests.

Authors' contributions

LT drafted the manuscript. All authors reviewed several versions of the manuscript, read and approved the final version.

The original online version of this article was revised: the authors would like to correct the number of sample size in the fourth paragraph under the heading Sample Size for Pilot Studies from “75 patients” to “289 patients”.

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Thabane, L., Ma, J., Chu, R. et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 10 , 1 (2010). https://doi.org/10.1186/1471-2288-10-1

Download citation

Received : 09 August 2009

Accepted : 06 January 2010

Published : 06 January 2010

DOI : https://doi.org/10.1186/1471-2288-10-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Pilot Study
• Sample Size Calculation
• Research Ethic Board
• Adaptive Design

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• En español – ExME
• Em português – EME

What is a pilot study?

Posted on 31st July 2017 by Luiz Cadete

Pilot studies can play a very important role prior to conducting a full-scale research project

Pilot studies are small-scale, preliminary studies which aim to investigate whether crucial components of a main study – usually a randomized controlled trial (RCT) – will be feasible. For example, they may be used in attempt to predict an appropriate sample size for the full-scale project and/or to improve upon various aspects of the study design. Often RCTs require a lot of time and money to be carried out, so it is crucial that the researchers have confidence in the key steps they will take when conducting this type of study to avoid wasting time and resources.

Thus, a pilot study must answer a simple question: “Can the full-scale study be conducted in the way that has been planned or should some component(s) be altered?”

The reporting of pilot studies must be of high quality to allow readers to interpret the results and implications correctly. This blog will highlight some key things for readers to consider when they are appraising a pilot study.

What are the main reasons to conduct a pilot study?

Pilot studies are conducted to evaluate the feasibility of some crucial component(s) of the full-scale study. Typically, these can be divided into 4 main aspects:

• Process : where the feasibility of the key steps in the main study is assessed (e.g. recruitment rate; retention levels and eligibility criteria)
• Resources: assessing problems with time and resources that may occur during the main study (e.g. how much time the main study will take to be completed; whether use of some equipment will be feasible or whether the form(s) of evaluation selected for the main study are as good as possible)
• Management: problems with data management and with the team involved in the study (e.g. whether there were problems with collecting all the data needed for future analysis; whether the collected data are highly variable and whether data from different institutions can be analyzed together).

Reasons for not conducting a pilot study

A study should not simply be labelled a ‘pilot study’ by researchers hoping to justify a small sample size. Pilot studies should always have their objectives linked with feasibility and should inform researchers about the best way to conduct the future, full-scale project.

How to interpret a pilot study

Readers must interpret pilot studies carefully. Below are some key things to consider when assessing a pilot study:

• The objectives of pilot studies must always be linked with feasibility and the crucial component that will be tested must always be stated.
• The method section must present the criteria for success. For example: “the main study will be feasible if the retention rate of the pilot study exceeds 90%”. Sample size may vary in pilot studies (different articles present different sample size calculations) but the pilot study population, from which the sample is formed, must be the same as the main study. However, the participants in the pilot study should not be entered into the full-scale study. This is because participants may change their later behaviour if they had previously been involved in the research.
• The pilot study may or may not be a randomized trial (depending on the nature of the study). If the researchers do randomize the sample in the pilot study, it is important that the process for randomization is kept the same in the full-scale project. If the authors decide to test the randomization feasibility through a pilot study, different kinds of randomization procedures could be used.
• As well as the method section, the results of the pilot studies should be read carefully. Although pilot studies often present results related to the effectiveness of the interventions, these results should be interpreted as “potential effectiveness”. The focus in the results of pilot studies should always be on feasibility, rather than statistical significance. However, results of the pilot studies should nonetheless be provided with measures of variability (such as confidence intervals), particularly as the sample size of these studies is usually relatively small, and this might produce biased results.

After an interpretation of results, pilot studies should conclude with one of the following:

(1) the main study is not feasible;

(2) the main study is feasible, with changes to the protocol;

(3) the main study is feasible without changes to the protocol OR

(4) the main study is feasible with close monitoring.

Any recommended changes to the protocol should be clearly outlined.

Take home message

• A pilot study must provide information about whether a full-scale study is feasible and list any recommended amendments to the design of the future study.

Thabane L, Ma J, Chu R, et al. A tutorial on pilot studies: what, why and how? BMC Med Res Methodol. 2010; 10: 1.

Cocks K and Torgerson DJ. Sample Size Calculations for Randomized Pilot Trials: A Confidence Interval approach . Journal of Clinical Epidemiology. 2013.

Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004; 10 (2): 307-12.

Moore et al. Recommendations for Planning Pilot Studies in Clinical and Translational Research. Clin Transl Sci. 2011 October ; 4(5): 332–337.

Luiz Cadete

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on What is a pilot study?

I want to do pilot study what can I do. My age is 30 .

Dear Khushbu, were you wanting to get involved in research? If so, what type of research were you interested in. There are lots of resources we can point you towards.

very intersesting

How can I study pilot and how can I start at first step? What should I do at the first time.

Informative.Thank you

If i am conducting a RCT then is it necessary to give interventions before conducting pilot study???

This fantastic. I am a doctoral student preparing do a pilot study on my main study.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Popular Articles

A 20 Minute Introduction to Cluster Randomised Trials

Another 20 minute tutorial from Tim.

A beginner’s guide to interpreting odds ratios, confidence intervals and p-values

The nuts and bolts 20 minute tutorial from Tim.

Free information and resources for pupils and students interested in evidence-based health care

This new webpage from Cochrane UK is aimed at students of all ages. What is evidence-based practice? What is ‘best available research evidence’? Which resources will help you understand evidence and evidence-based practice, and search for evidence?

Pilot Study in Research

• Key Concepts
• Major Sociologists
• News & Issues
• Research, Samples, and Statistics
• Recommended Reading
• Archaeology

A pilot study is a preliminary small-scale study that researchers conduct in order to help them decide how best to conduct a large-scale research project. Using a pilot study, a researcher can identify or refine a research question, figure out what methods are best for pursuing it, and estimate how much time and resources will be necessary to complete the larger version, among other things.

Key Takeaways: Pilot Studies

• Before running a larger study, researchers can conduct a pilot study : a small-scale study that helps them refine their research topic and study methods.
• Pilot studies can be useful for determining the best research methods to use, troubleshooting unforeseen issues in the project, and determining whether a research project is feasible.
• Pilot studies can be used in both quantitative and qualitative social science research.

Large-scale research projects tend to be complex, take a lot of time to design and execute, and typically require quite a bit of funding. Conducting a pilot study beforehand allows a researcher to design and execute a large-scale project in as methodologically rigorous a way as possible, and can save time and costs by reducing the risk of errors or problems. For these reasons, pilot studies are used by both quantitative and qualitative researchers in the social sciences.

Advantages of Conducting a Pilot Study

Pilot studies are useful for a number of reasons, including:

• Identifying or refining a research question or set of questions
• Identifying or refining a hypothesis or set of hypotheses
• Identifying and evaluating a sample population, research field site , or data set
• Testing research instruments like survey questionnaires , interview, discussion guides, or statistical formulas
• Evaluating and deciding upon research methods
• Identifying and resolving as many potential problems or issues as possible
• Estimating the time and costs required for the project
• Gauging whether the research goals and design are realistic
• Producing preliminary results that can help secure funding and other forms of institutional investment

After conducting a pilot study and taking the steps listed above, a researcher will know what to do in order to proceed in a way that will make the study a success. 

Example: Quantitative Survey Research

Say you want to conduct a large-scale quantitative research project using survey data to study the relationship between race and political party affiliation . To best design and execute this research, you would first want to select a data set to use, such as the General Social Survey , for example, download one of their data sets, and then use a statistical analysis program to examine this relationship. In the process of analyzing the relationship, you are likely to realize the importance of other variables that may have an impact on political party affiliation. For example, place of residence, age, education level, socioeconomic status, and gender may impact party affiliation (either on their own or in interaction with race). You might also realize that the data set you chose does not offer you all the information that you need to best answer this question, so you might choose to use another data set, or combine another with the original that you selected. Going through this pilot study process will allow you to work out the kinks in your research design and then execute high-quality research.

Example: Qualitative Interview Studies

Pilot studies can also be useful for qualitative research studies, such as interview-based studies. For example, imagine that a researcher is interested in studying the relationship that Apple consumers have to the company's brand and products . The researcher might choose to first do a pilot study consisting of a couple of focus groups in order to identify questions and thematic areas that would be useful to pursue in-depth, one-on-one interviews. A focus group can be useful to this kind of study because while a researcher will have a notion of what questions to ask and topics to raise, she may find that other topics and questions arise when members of the target group talk among themselves. After a focus group pilot study, the researcher will have a better idea of how to craft an effective interview guide for a larger research project.

• Definition of Idiographic and Nomothetic
• An Overview of Qualitative Research Methods
• Conducting Case Study Research in Sociology
• How to Conduct a Sociology Research Interview
• Understanding Secondary Data and How to Use It in Research
• Convenience Samples for Research
• Definition and Overview of Grounded Theory
• Research in Essays and Reports
• Pros and Cons of Secondary Data Analysis
• What Is Ethnography?
• Social Surveys: Questionnaires, Interviews, and Telephone Polls
• Macro- and Microsociology
• The Different Types of Sampling Designs in Sociology
• Introduction to Sociology
• What Is Naturalistic Observation? Definition and Examples
• What Is Participant Observation Research?

The importance of pilot studies

Affiliation.

• 1 Department of Public Health, University of Aberdeen. [email protected]
• PMID: 12216297
• DOI: 10.7748/ns2002.06.16.40.33.c3214

The term 'pilot studies' refers to mini versions of a full-scale study (also called 'feasibility' studies), as well as the specific pre-testing of a particular research instrument such as a questionnaire or interview schedule. Pilot studies are a crucial element of a good study design. Conducting a pilot study does not guarantee success in the main study, but it does increase the likelihood of success. Pilot studies fulfill a range of important functions and can provide valuable insights for other researchers. There is a need for more discussion among researchers of both the process and outcomes of pilot studies.

Publication types

• Data Collection / methods
• Interviews as Topic
• Nursing Research / methods*
• Nursing Research / standards*
• Pilot Projects*
• Reproducibility of Results
• Research Design / standards*
• Surveys and Questionnaires

• Open access
• Published: 31 October 2020

Guidance for conducting feasibility and pilot studies for implementation trials

• Nicole Pearson   ORCID: orcid.org/0000-0003-2677-2327 1 , 2 ,
• Patti-Jean Naylor 3 ,
• Maureen C. Ashe 5 ,
• Maria Fernandez 4 ,
• Sze Lin Yoong 1 , 2 &
• Luke Wolfenden 1 , 2  

Pilot and Feasibility Studies volume  6 , Article number:  167 ( 2020 ) Cite this article

84k Accesses

74 Citations

24 Altmetric

Metrics details

Implementation trials aim to test the effects of implementation strategies on the adoption, integration or uptake of an evidence-based intervention within organisations or settings. Feasibility and pilot studies can assist with building and testing effective implementation strategies by helping to address uncertainties around design and methods, assessing potential implementation strategy effects and identifying potential causal mechanisms. This paper aims to provide broad guidance for the conduct of feasibility and pilot studies for implementation trials.

We convened a group with a mutual interest in the use of feasibility and pilot trials in implementation science including implementation and behavioural science experts and public health researchers. We conducted a literature review to identify existing recommendations for feasibility and pilot studies, as well as publications describing formative processes for implementation trials. In the absence of previous explicit guidance for the conduct of feasibility or pilot implementation trials specifically, we used the effectiveness-implementation hybrid trial design typology proposed by Curran and colleagues as a framework for conceptualising the application of feasibility and pilot testing of implementation interventions. We discuss and offer guidance regarding the aims, methods, design, measures, progression criteria and reporting for implementation feasibility and pilot studies.

Conclusions

This paper provides a resource for those undertaking preliminary work to enrich and inform larger scale implementation trials.

Peer Review reports

The failure to translate effective interventions for improving population and patient outcomes into policy and routine health service practice denies the community the benefits of investment in such research [ 1 ]. Improving the implementation of effective interventions has therefore been identified as a priority of health systems and research agencies internationally [ 2 , 3 , 4 , 5 , 6 ]. The increased emphasis on research translation has resulted in the rapid emergence of implementation science as a scientific discipline, with the goal of integrating effective medical and public health interventions into health care systems, policies and practice [ 1 ]. Implementation research aims to do this via the generation of new knowledge, including the evaluation of the effectiveness of implementation strategies [ 7 ]. The term “implementation strategies” is used to describe the methods or techniques (e.g. training, performance feedback, communities of practice) used to enhance the adoption, implementation and/or sustainability of evidence-based interventions (Fig. 1 ) [ 8 , 9 ].

Conceptual role of implementation strategies in improving intervention implementation and patient and public health outcomes

While there has been a rapid increase in the number of implementation trials over the past decade, the quality of trials has been criticised, and the effects of the strategies for such trials on implementation, patient or public health outcomes have been modest [ 11 , 12 , 13 ]. To improve the likelihood of impact, factors that may impede intervention implementation should be considered during intervention development and across each phase of the research translation process [ 2 ]. Feasibility and pilot studies play an important role in improving the conduct and quality of a definitive randomised controlled trial (RCT) for both intervention and implementation trials [ 10 ]. For clinical or public health interventions, pilot and feasibility studies may serve to identify potential refinements to the intervention, address uncertainties around the feasibility of intervention trial methods, or test preliminary effects of the intervention [ 10 ]. In implementation research, feasibility and pilot studies perform the same functions as those for intervention trials, however with a focus on developing or refining implementation strategies, refining research methods for an implementation intervention trial, or undertake preliminary testing of implementation strategies [ 14 , 15 ]. Despite this, reviews of implementation studies appear to suggest that few full implementation randomised controlled trials have undertaken feasibility and pilot work in advance of a larger trial [ 16 ].

A range of publications provides guidance for the conduct of feasibility and pilot studies for conventional clinical or public health efficacy trials including Guidance for Exploratory Studies of complex public health interventions [ 17 ] and the Consolidated Standards of Reporting Trials (CONSORT 2010) for Pilot and Feasibility trials [ 18 ]. However, given the differences between implementation trials and conventional clinical or public health efficacy trials, the field of implementation science has identified the need for nuanced guidance [ 14 , 15 , 16 , 19 , 20 ]. Specifically, unlike traditional feasibility and pilot studies that may include the preliminary testing of interventions on individual clinical or public health outcomes, implementation feasibility and pilot studies that explore strategies to improve intervention implementation often require assessing changes across multiple levels including individuals (e.g. service providers or clinicians) and organisational systems [ 21 ]. Due to the complexity of influencing behaviour change, the role of feasibility and pilot studies of implementation may also extend to identifying potential causal mechanisms of change and facilitate an iterative process of refining intervention strategies and optimising their impact [ 16 , 17 ]. In addition, where conventional clinical or public health efficacy trials are typically conducted under controlled conditions and directed mostly by researchers, implementation trials are more pragmatic [ 15 ]. As is the case for well conducted effectiveness trials, implementation trials often require partnerships with end-users and at times, the prioritisation of end-user needs over methods (e.g. random assignment) that seek to maximise internal validity [ 15 , 22 ]. These factors pose additional challenges for implementation researchers and underscore the need for guidance on conducting feasibility and pilot implementation studies.

Given the importance of feasibility and pilot studies in improving implementation strategies and the quality of full-scale trials of those implementation strategies, our aim is to provide practice guidance for those undertaking formative feasibility or pilot studies in the field of implementation science. Specifically, we seek to provide guidance pertaining to the three possible purposes of undertaking pilot and feasibility studies, namely (i) to inform implementation strategy development, (ii) to assess potential implementation strategy effects and (iii) to assess the feasibility of study methods.

A series of three facilitated group discussions were conducted with a group comprising of the 6 members from Canada, the U.S. and Australia (authors of the manuscript) that were mutually interested in the use of feasibility and pilot trials in implementation science. Members included international experts in implementation and behavioural science, public health and trial methods, and had considerable experience in conducting feasibility, pilot and/ or implementation trials. The group was responsible for developing the guidance document, including identification and synthesis of pertinent literature, and approving the final guidance.

To inform guidance development, a literature review was undertaken in electronic bibliographic databases and google, to identify and compile existing recommendations and guidelines for feasibility and pilot studies broadly. Through this process, we identified 30 such guidelines and recommendations relevant to our aim [ 2 , 10 , 14 , 15 , 17 , 18 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ]. In addition, seminal methods and implementation science texts recommended by the group were examined. These included the CONSORT 2010 Statement: extension to randomised pilot and feasibility trials [ 18 ], the Medical Research Council’s framework for development and evaluation of randomised controlled trials for complex interventions to improve health [ 2 ], the National Institute of Health Research (NIHR) definitions [ 39 ] and the Quality Enhancement Research Initiative (QUERI) Implementation Guide [ 4 ]. A summary of feasibility and pilot study guidelines and recommendations, and that of seminal methods and implementation science texts, was compiled by two authors. This document served as the primary discussion document in meetings of the group. Additional targeted searches of the literature were undertaken in circumstances where the identified literature did not provide sufficient guidance. The manuscript was developed iteratively over 9 months via electronic circulation and comment by the group. Any differences in views between reviewers was discussed and resolved via consensus during scheduled international video-conference calls. All members of the group supported and approved the content of the final document.

The broad guidance provided is intended to be used as supplementary resources to existing seminal feasibility and pilot study resources. We used the definitions of feasibility and pilot studies as proposed by Eldridge and colleagues [ 10 ]. These definitions propose that any type of study relating to the preparation for a main study may be classified as a “feasibility study”, and that the term “pilot” study represents a subset of feasibility studies that specifically look at a design feature proposed for the main trial, whether in part of in full, that is being conducted on a smaller scale [ 10 ]. In addition, when referring to pilot studies, unless explicitly stated otherwise, we will primarily focus on pilot trials using a randomised design. We focus on randomised trials as such designs are the most common trial design in implementation research, and randomised designs may provide the most robust estimates of the potential effect of implementation strategies [ 46 ]. Those undertaking pilot studies that employ non-randomised designs need to interpret the guidance provided in this context. We acknowledge, however, that using randomised designs can prove particularly challenging in the field of implementation science, where research is often undertaken in real-world contexts with pragmatic constraints.

We used the effectiveness-implementation hybrid trial design typology proposed by Curran and colleagues as the framework for conceptualising the application of feasibility testing of implementation interventions [ 47 ]. The typology makes an explicit distinction between the purpose and methods of implementation and conventional clinical (or public health efficacy) trials. Specifically, the first two of the three hybrid designs may be relevant for implementation feasibility or pilot studies. Hybrid Type 1 trials are those designed to test the effectiveness of an intervention on clinical or public health outcomes (primary aim) while conducting a feasibility or pilot study for future implementation via observing and gathering information regarding implementation in a real-world setting/situation (secondary aim) [ 47 ]. Hybrid Type 2 trials involve the simultaneous testing of both the clinical intervention and the testing or feasibility of a formed implementation intervention/strategy as co-primary aims. For this design, “testing” is inclusive of pilot studies with an outcome measure and related hypothesis [ 47 ]. Hybrid Type 3 trials are definitive implementation trials designed to test the effectiveness of an implementation strategy whilst also collecting secondary outcome data on clinical or public health outcomes on a population of interest [ 47 ]. As the implementation aim of the trial is a definitively powered trial, it was not considered relevant to the conduct of feasibility and pilot studies in the field and will not be discussed.

Embedding of feasibility and pilot studies within Type 1 and Type 2 effectiveness-implementation hybrid trials has been recommended as an efficient way to increase the availability of information and evidence to accelerate the field of implementation science and the development and testing of implementation strategies [ 4 ]. However, implementation feasibility and pilot studies are also undertaken as stand-alone exploratory studies and do not include effectiveness measures in terms of the patient or public health outcomes. As such, in addition to discussing feasibility and pilot trials embedded in hybrid trial designs, we will also refer to stand-alone implementation feasibility and pilot studies.

An overview of guidance (aims, design, measures, sample size and power, progression criteria and reporting) for feasibility and pilot implementation studies can be found in Table 1 .

Purpose (aims)

The primary objective of hybrid type 1 trial is to assess the effectiveness of a clinical or public health intervention (rather than an implementation strategy) on the patient or population health outcomes [ 47 ]. Implementation strategies employed in these trials are often designed to maximise the likelihood of an intervention effect [ 51 ], and may not be intended to represent the strategy that would (or could feasibly), be used to support implementation in more “real world” contexts. Specific aims of implementation feasibility or pilot studies undertaken as part of Hybrid Type 1 trials are therefore formative and descriptive as the implementation strategy has not been fully formed nor will be tested. Thus, the purpose of a Hybrid Type 1 feasibility study is generally to inform the development or refinement of the implementation strategy rather than to test potential effects or mechanisms [ 22 , 47 ]. An example of a Hybrid Type 1 trial by Cabassa and colleagues is provided in Additional file 1 [ 52 ].

In Hybrid Type 2 trial designs, there is a dual purpose to test: (i) the clinical or public health effectiveness of the intervention on clinical or public health outcomes (e.g. measure of disease or health behaviour) and (ii) test or measure the impact of the implementation strategy on implementation outcomes (e.g. adoption of health policy in a community setting) [ 53 ]. However, testing the implementation strategy on implementation outcomes may be a secondary aim in these trials and positioned as a pilot [ 22 ]. In Hybrid Type 2 trial designs, the implementation strategy is more developed than in Hybrid Type 1 trials, resembling that intended for future testing in a definitive implementation randomised controlled trial. The dual testing of the evidence-based intervention and implementation interventions or strategies in Hybrid Type 2 trial designs allows for direct assessment of potential effects of an implementation strategy and exploration of components of the strategy to further refine logic models. Additionally, such trials allow for assessments of the feasibility, utility, acceptability or quality of research methods for use in a planned definitive trial. An example of a Hybrid Type 2 trial design by Barnes and colleagues [ 54 ] is included in Additional file 2 .

Non-hybrid pilot implementation studies are undertaken in the absence of a broader effectiveness trial. Such studies typically occur when the effectiveness of a clinical or public health intervention is well established, but robust strategies to promote its broader uptake and integration into clinical or public health services remain untested [ 15 ]. In these situations, implementation pilot studies may test or explore specific trial methods for a future definitive randomised implementation trial. Similarly, a pilot implementation study may also be undertaken in a way that provides a more rigorous formative evaluation of hypothesised implementation strategy mechanisms [ 55 ], or potential impact of implementation strategies [ 56 ], using similar approaches to that employed in Hybrid Type 2 trials. Examples of potential aims for feasibility and pilot studies are outlined in Table 2 .

For implementation feasibility or pilot studies, as is the case for these types of studies in general, the selection of research design should be guided by the specific research question that the study is seeking to address [ 57 ]. Although almost any study design may be used, researchers should review the merits and potential threats to internal and external validity to help guide the selection of research design for feasibility/pilot testing [ 15 ].

As Hybrid Type 1 trials are primarily concerned with testing the effectiveness of an intervention (rather than implementation strategy), the research design will typically employ power calculations and randomisation procedures at the health outcome level to measure the effect on behaviour, symptoms, functional and/or other clinical or public health outcomes. Hybrid Type 1 feasibility studies may employ a variety of designs usually nested within the experimental group (those receiving the intervention and any form of an implementation support strategy) of the broader efficacy trial [ 47 ]. Consistent with the aims of Hybrid Type 1 feasibility and pilot studies, the research designs employed are likely to be non-comparative. Cross-sectional surveys, interviews or document review, qualitative research or mix methods approaches may be used to assess implementation contextual factors, such as barriers and enablers to implementation and/or the acceptability, perceived feasibility or utility of implementation strategies or research methods [ 47 ].

Pilot implementation studies as part of Hybrid Type 2 designs can make use of the comparative design of the broader effectiveness trial to examine the potential effects of the implementation strategy [ 47 ] and more robustly assess the implementation mechanisms, determinants and influence of broader contextual factors [ 53 ]. In this trial type, mixed method and qualitative methods may complement the findings of between group (implementation strategy arm versus comparison) quantitative comparisons, enable triangulation and provide more comprehensive evidence to inform implementation strategy development and assessment. Stand-alone implementation feasibility and pilot implementation studies are free from the constraints and opportunities of research embedded in broader effectiveness trials. As such, research can be designed in a way that best addresses the explicit implementation objectives of the study. Specifically, non-hybrid pilot studies can maximise the applicability of study findings for future definitive trials by employing methods to directly test trial methods such as recruitment or retention strategies [ 17 ], enabling estimates of implementation strategies effects [ 56 ] or capturing data to explicitly test logic models or strategy mechanisms.

The selection of outcome measures should be linked directly to the objectives of the feasibility or pilot study. Where appropriate, measures should be objective or have suitable psychometric properties, such as evidence of reliability and validity [ 58 , 59 ]. Public health evaluation frameworks often guide the choice of outcome measure in feasibility and pilot implementation work and include RE_AIM [ 60 ], PRECEDE_PROCEED [ 61 ], Proctor and colleagues framework on outcomes for implementation research [ 62 ] and more recently, the “Implementation Mapping” framework [ 63 ]. Recent work by McKay and colleagues suggests a minimum data set of implementation outcomes that includes measures of adoption, reach, dose, fidelity and sustainability [ 46 ]. We discuss selected measures below and provide a summary in Table 3 [ 46 ]. Such measures could be assessed using quantitative or qualitative or mixed methods [ 46 ].

Measures to assess potential implementation strategy effects

In addition to assessing the effects of an intervention on individual clinical or public health outcomes, Hybrid Type 2 trials (and some non-hybrid pilot studies) are interested in measures of the potential effects of an implementation strategy on desired organisational or clinician practice change such as adherence to a guideline, process, clinical standard or delivery of a program [ 62 ]. A range of potential outcomes that could be used to assess implementation strategy effects has been identified, including measures of adoption, reach, fidelity and sustainability [ 46 ]. These outcomes are described in Table 2 , including definitions and examples of how they may be applied to the implementation component of innovation being piloted. Standardised tools to assess these outcomes are often unavailable due to the unique nature of interventions being implemented and the variable (and changing) implementation context in which the research is undertaken [ 64 ]. Researchers may collect outcome data for these measures as part of environmental observations, self-completed checklists or administrative records, audio recording of client sessions or other methods suited to their study and context [ 62 ]. The limitations of such methods, however, need to be considered.

Measures to inform the design or development of the implementation strategy

Measures informing the design or development of the implementation strategy are potentially part of all types of feasibility and pilot implementation studies. An understanding of the determinants of implementation is critical to implementation strategy development. A range of theoretical determinant frameworks have been published which describe factors that may influence intervention implementation [ 65 ], and systematic reviews have been undertaken describing the psychometric properties of many of these measures [ 64 , 66 ]. McKay and colleagues have also identified a priority set of determinants for implementation trials that could be considered for use in implementation feasibility and pilot studies, including measures of context, acceptability, adaptability, feasibility, compatibility, cost, culture, dose, complexity and self-efficacy [ 46 ]. These determinants are described in Table 3 , including definitions and how such measures may be applied to an implementation feasibility or pilot study. Researchers should consider, however, the application of such measures to assess both the intervention that is being implemented (as in a conventional intervention feasibility and pilot study) and the strategy that is being employed to facilitate its implementation, given the importance of the interaction between these factors and implementation success [ 46 ]. Examples of the potential application of measures to both the intervention and its implementation strategies have been outlined elsewhere [ 46 ]. Although a range of quantitative tools could be used to measure such determinants [ 58 , 66 ], qualitative or mixed methods are generally recommended given the capacity of qualitative measures to provide depth to the interpretation of such evaluations [ 40 ].

Measures of potential implementation determinants may be included to build or enhance logic models (Hybrid Type 1 and 2 feasibility and pilot studies) and explore implementation strategy mechanisms (Hybrid Type 2 pilot studies and non-hybrid pilot studies) [ 67 ]. If exploring strategy mechanisms, a hypothesized logic model underpinning the implementation strategy should be articulated including strategy-mechanism linkages, which are required to guide the measurement of key determinants [ 55 , 63 ]. An important determinant which can complicate logic model specification and measurement is the process of adaptation—modifications to the intervention or its delivery (implementation), through the input of service providers or implementers [ 68 ]. Logic models should specify components of implementation strategies thought to be “core” to their effects and those which are thought to be “non-core” where adaptation may occur without adversely impacting on effects. Stirman and colleagues propose a method for assessing adaptations that could be considered for use in pilot and feasibility studies of implementation trials [ 69 ]. Figure 2 provides an example of some of the implementation logic model components that may be developed or refined as part of feasibility or pilot studies of implementation [ 15 , 63 ].

Example of components of an Implementation logic model

Measures to assess the feasibility of study methods

Measures of implementation feasibility and pilot study methods are similar to those of conventional studies for clinical or public health interventions. For example, standard measures of study participation and thresholds for study attrition (e.g. >20%) rates [ 73 ] can be employed in implementation studies [ 67 ]. Previous studies have also surveyed study data collectors to assess the success of blinding strategies [ 74 ]. Researchers may also consider assessing participation or adherence to implementation data collection procedures, the comprehension of survey items, data management strategies or other measures of feasibility of study methods [ 15 ].

Pilot study sample size and power

In effectiveness trials, power calculations and sample size decisions are primarily based on the detection of a clinically meaningful difference in measures of the effects of the intervention on the patient or public health outcomes such as behaviour, disease, symptomatology or functional outcomes [ 24 ]. In this context, the available study sample for implementation measures included in Hybrid Type 1 or 2 feasibility and pilot studies may be constrained by the sample and power calculations of the broader effectiveness trial in which they are embedded [ 47 ]. Nonetheless, a justification for the anticipated sample size for all implementation feasibility or pilot studies (hybrid or stand-alone) is recommended [ 18 ], to ensure that implementation measures and outcomes achieve sufficient estimates of precision to be useful. For Hybrid type 2 and relevant stand-alone implementation pilot studies, sample size calculations for implementation outcomes should seek to achieve adequate estimates of precision deemed sufficient to inform progression to a fully powered trial [ 18 ].

Progression criteria

Stating progression criteria when reporting feasibility and pilot studies is recommended as part of the CONSORT 2010 extension to randomised pilot and feasibility trials guidelines [ 18 ]. Generally, it is recommended that progression criteria should be set a priori and be specific to the feasibility measures, components and/or outcomes assessed in the study [ 18 ]. While little guidance is available, ideas around suitable progression criteria include assessment of uncertainties around feasibility, meeting recruitment targets, cost-effectiveness and refining causal hypotheses to be tested in future trials [ 17 ]. When developing progression criteria, the use of guidelines is suggested rather than strict thresholds [ 18 ], in order to allow for appropriate interpretation and exploration of potential solutions, for example, the use of a traffic light system with varying levels of acceptability [ 17 , 24 ]. For example, Thabane and colleagues recommend that, in general, the outcome of a pilot study can be one of the following: (i) stop—main study not feasible (red); (ii) continue, but modify protocol—feasible with modifications (yellow); (iii) continue without modifications, but monitor closely—feasible with close monitoring and (iv) continue without modifications (green) (44)p5.

As the goal of Hybrid Type 1 implementation component is usually formative, it may not be necessary to set additional progression criteria in terms of the implementation outcomes and measures examined. As Hybrid Type 2 trials test an intervention and can pilot an implementation strategy, criteria for these and non-hybrid pilot studies may set progression criteria based on evidence of potential effects but may also consider the feasibility of trial methods, service provider, organisational or patient (or community) acceptability, fit with organisational systems and cost-effectiveness [ 17 ]. In many instances, the progression of implementation pilot studies will often require the input and agreement of stakeholders [ 27 ]. As such, the establishment of progression criteria and the interpretation of pilot and feasibility study findings in the context of such criteria require stakeholder input [ 27 ].

Reporting suggestions

As formal reporting guidelines do not exist for hybrid trial designs, we would recommend that feasibility and pilot studies as part of hybrid designs draw upon best practice recommendations from relevant reporting standards such as the CONSORT extension for randomised pilot and feasibility trials, the Standards for Reporting Implementation Studies (STaRI) guidelines and the Template for Intervention Description and Replication (TIDieR) guide as well as any other design relevant reporting standards [ 48 , 50 , 75 ]. These, and further reporting guidelines, specific to the particular research design chosen, can be accessed as part of the EQUATOR (Enhancing the QUAility and Transparency Of health Research) network—a repository for reporting guidance [ 76 ]. In addition, researchers should specify the type of implementation feasibility or pilot study being undertaken using accepted definitions. If applicable, specification and justification behind the choice of hybrid trial design should also be stated. In line with existing recommendations for reporting of implementation trials generally, reporting on the referent of outcomes (e.g. specifying if the measure in relation to the specific intervention or the implementation strategy) [ 62 ], is also particularly pertinent when reporting hybrid trial designs.

Concerns are often raised regarding the quality of implementation trials and their capacity to contribute to the collective evidence base [ 3 ]. Although there have been many recent developments in the standardisation of guidance for implementation trials, information on the conduct of feasibility and pilot studies for implementation interventions remains limited, potentially contributing to a lack of exploratory work in this area and a limited evidence base to inform effective implementation intervention design and conduct [ 15 ]. To address this, we synthesised the existing literature and provide commentary and guidance for the conduct of implementation feasibility and pilot studies. To our knowledge, this work is the first to do so and is an important first step to the development of standardised guidelines for implementation-related feasibility and pilot studies.

Availability of data and materials

Not applicable.

Abbreviations

Randomised controlled trial

Consolidated Standards of Reporting Trials

Enhancing the QUAility and Transparency Of health Research

Standards for Reporting Implementation Studies

Strengthening the Reporting of Observational Studies in Epidemiology

Template for Intervention Description and Replication

National Institute of Health Research

Quality Enhancement Research Initiative

Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol. 2015;3:32.

Article   Google Scholar  

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655.

Eccles MP, Armstrong D, Baker R, Cleary K, Davies H, Davies S, et al. An implementation research agenda. Implement Sci. 2009;4:18.

Department of Veterans Health Administration. Implementation Guide. Health Services Research & Development, Quality Enhancement Research Initiative. Updated 2013.

Peters DH, Nhan TT, Adam T. Implementation research: a practical guide; 2013.

Google Scholar  

Neta G, Sanchez MA, Chambers DA, Phillips SM, Leyva B, Cynkin L, et al. Implementation science in cancer prevention and control: a decade of grant funding by the National Cancer Institute and future directions. Implement Sci. 2015;10:4.

Foy R, Sales A, Wensing M, Aarons GA, Flottorp S, Kent B, et al. Implementation science: a reappraisal of our journal mission and scope. Implement Sci. 2015;10:51.

Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for specifying and reporting. Implement Sci. 2013;8:139.

Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond "implementation strategies": classifying the full range of strategies used in implementation science and practice. Implement Sci. 2017;12(1):125.

Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One. 2016;11(3):e0150205.

Article   CAS   Google Scholar  

Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123–57.

Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci. 2015;10:21.

Lewis CC, Stanick C, Lyon A, Darnell D, Locke J, Puspitasari A, et al. Proceedings of the fourth biennial conference of the Society for Implementation Research Collaboration (SIRC) 2017: implementation mechanisms: what makes implementation work and why? Part 1. Implement Sci. 2018;13(Suppl 2):30.

Levati S, Campbell P, Frost R, Dougall N, Wells M, Donaldson C, et al. Optimisation of complex health interventions prior to a randomised controlled trial: a scoping review of strategies used. Pilot Feasibility Stud. 2016;2:17.

Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med. 2009;36(5):452–7.

Eccles M, Grimshaw J, Walker A, Johnston M, Pitts N. Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol. 2005;58(2):107–12.

Hallingberg B, Turley R, Segrott J, Wight D, Craig P, Moore L, et al. Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance. Pilot Feasibility Stud. 2018;4:104.

Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. Pilot Feasibility Stud. 2016;2:64.

Proctor EK, Powell BJ, Baumann AA, Hamilton AM, Santens RL. Writing implementation research grant proposals: ten key ingredients. Implement Sci. 2012;7:96.

Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl 2):S1–8.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Admin Pol Ment Health. 2011;38(1):4–23.

Johnson AL, Ecker AH, Fletcher TL, Hundt N, Kauth MR, Martin LA, et al. Increasing the impact of randomized controlled trials: an example of a hybrid effectiveness-implementation design in psychotherapy research. Transl Behav Med. 2018.

Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10(1):67.

Avery KN, Williamson PR, Gamble C, O’Connell Francischetto E, Metcalfe C, Davidson P, et al. Informing efficient randomised controlled trials: exploration of challenges in developing progression criteria for internal pilot studies. BMJ Open. 2017;7(2):e013537.

Bell ML, Whitehead AL, Julious SA. Guidance for using pilot studies to inform the design of intervention trials with continuous outcomes. J Clin Epidemiol. 2018;10:153–7.

Billingham SAM, Whitehead AL, Julious SA. An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom clinical research Network database. BMC Med Res Methodol. 2013;13(1):104.

Bugge C, Williams B, Hagen S, Logan J, Glazener C, Pringle S, et al. A process for decision-making after pilot and feasibility trials (ADePT): development following a feasibility study of a complex intervention for pelvic organ prolapse. Trials. 2013;14:353.

Charlesworth G, Burnell K, Hoe J, Orrell M, Russell I. Acceptance checklist for clinical effectiveness pilot trials: a systematic approach. BMC Med Res Methodol. 2013;13(1):78.

Eldridge SM, Costelloe CE, Kahan BC, Lancaster GA, Kerry SM. How big should the pilot study for my cluster randomised trial be? Stat Methods Med Res. 2016;25(3):1039–56.

Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation (Lond). 2016;22(3):286–303.

Hampson LV, Williamson PR, Wilby MJ, Jaki T. A framework for prospectively defining progression rules for internal pilot studies monitoring recruitment. Stat Methods Med Res. 2018;27(12):3612–27.

Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry. 2006;63(5):484–9.

Smith LJ, Harrison MB. Framework for planning and conducting pilot studies. Ostomy Wound Manage. 2009;55(12):34–48.

Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004;10(2):307–12.

Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res. 2011;45(5):626–9.

Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: Medical Research Council; 2000.

Möhler R, Bartoszek G, Meyer G. Quality of reporting of complex healthcare interventions and applicability of the CReDECI list - a survey of publications indexed in PubMed. BMC Med Res Methodol. 2013;13(1):125.

Möhler R, Köpke S, Meyer G. Criteria for reporting the development and evaluation of complex interventions in healthcare: revised guideline (CReDECI 2). Trials. 2015;16(1):204.

National Institute for Health Research. Definitions of feasibility vs pilot stuides [Available from: https://www.nihr.ac.uk/documents/guidance-on-applying-for-feasibility-studies/20474 ].

O'Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, et al. Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers. Pilot Feasibility Stud. 2015;1:32.

Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Med Res Methodol. 2011;11(1):117.

Teare MD, Dimairo M, Shephard N, Hayman A, Whitehead A, Walters SJ. Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study. Trials. 2014;15(1):264.

Thabane L, Lancaster G. Improving the efficiency of trials using innovative pilot designs: the next phase in the conduct and reporting of pilot and feasibility studies. Pilot Feasibility Stud. 2017;4(1):14.

Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

Westlund E. E.a. S. The nonuse, misuse, and proper use of pilot studies in experimental evaluation research. Am J Eval. 2016;38(2):246–61.

McKay H, Naylor PJ, Lau E, Gray SM, Wolfenden L, Milat A, et al. Implementation and scale-up of physical activity and behavioural nutrition interventions: an evaluation roadmap. Int J Behav Nutr Phys Act. 2019;16(1):102.

Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012;50(3):217–26.

Equator Network. Standards for reporting implementation studies (StaRI) statement 2017 [Available from: http://www.equator-network.org/reporting-guidelines/stari-statement/ ].

Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. PLoS Med. 2007;4(10):e297–e.

Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687.

Schliep ME, Alonzo CN, Morris MA. Beyond RCTs: innovations in research design and methods to advance implementation science. Evid Based Commun Assess Inter. 2017;11(3-4):82–98.

Cabassa LJ, Stefancic A, O'Hara K, El-Bassel N, Lewis-Fernández R, Luchsinger JA, et al. Peer-led healthy lifestyle program in supportive housing: study protocol for a randomized controlled trial. Trials. 2015;16:388.

Landes SJ, McBain SA, Curran GM. Reprint of: An introduction to effectiveness-implementation hybrid designs. J Psychiatr Res. 2020;283:112630.

Barnes C, Grady A, Nathan N, Wolfenden L, Pond N, McFayden T, Ward DS, Vaughn AE, Yoong SL. A pilot randomised controlled trial of a web-based implementation intervention to increase child intake of fruit and vegetables within childcare centres. Pilot and Feasibility Studies. 2020. https://doi.org/10.1186/s40814-020-00707-w .

Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:136.

Department of Veterans Health Affairs. Implementation Guide. Health Services Research & Development, Quality Enhancement Research Initiative. 2013.

Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108.

Lewis CC, Mettert KD, Dorsey CN, Martinez RG, Weiner BJ, Nolen E, et al. An updated protocol for a systematic review of implementation-related measures. Syst Rev. 2018;7(1):66.

Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res. 2006;21(5):688–94.

Green L, Kreuter M. Health promotion planning: an educational and ecological approach. Mountain View: Mayfield Publishing; 1999.

Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Admin Pol Ment Health. 2011;38(2):65–76.

Fernandez ME, Ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: using intervention mapping to develop implementation strategies. Front Public Health. 2019;7:158.

Lewis CC, Weiner BJ, Stanick C, Fischer SM. Advancing implementation science through measure development and evaluation: a study protocol. Implement Sci. 2015;10:102.

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50.

Clinton-McHarg T, Yoong SL, Tzelepis F, Regan T, Fielding A, Skelton E, et al. Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the consolidated framework for implementation research: a systematic review. Implement Sci. 2016;11(1):148.

Moore CG, Carter RE, Nietert PJ, Stewart PW. Recommendations for planning pilot studies in clinical and translational research. Clin Transl Sci. 2011;4(5):332–7.

Pérez D, Van der Stuyft P, Zabala MC, Castro M, Lefèvre P. A modified theoretical framework to assess implementation fidelity of adaptive public health interventions. Implement Sci. 2016;11(1):91.

Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8:65.

Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2:40.

Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41(3-4):327–50.

Saunders RP, Evans MH, Joshi P. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promot Pract. 2005;6(2):134–47.

Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Wyse RJ, Wolfenden L, Campbell E, Brennan L, Campbell KJ, Fletcher A, et al. A cluster randomised trial of a telephone-based intervention for parents to increase fruit and vegetable consumption in their 3- to 5-year-old children: study protocol. BMC Public Health. 2010;10:216.

Consort Transparent Reporting of Trials. Pilot and Feasibility Trials 2016 [Available from: http://www.consort-statement.org/extensions/overview/pilotandfeasibility ].

Equator Network. Ehancing the QUAlity and Transparency Of health Research. [Avaliable from: https://www.equator-network.org/ ].

Download references

Acknowledgements

Associate Professor Luke Wolfenden receives salary support from a NHMRC Career Development Fellowship (grant ID: APP1128348) and Heart Foundation Future Leader Fellowship (grant ID: 101175). Dr Sze Lin Yoong is a postdoctoral research fellow funded by the National Heart Foundation. A/Prof Maureen C. Ashe is supported by the Canada Research Chairs program.

Author information

Authors and affiliations.

School of Medicine and Public Health, University of Newcastle, University Drive, Callaghan, NSW 2308, Australia

Nicole Pearson, Sze Lin Yoong & Luke Wolfenden

Hunter New England Population Health, Locked Bag 10, Wallsend, NSW 2287, Australia

School of Exercise Science, Physical and Health Education, Faculty of Education, University of Victoria, PO Box 3015 STN CSC, Victoria, BC, V8W 3P1, Canada

Patti-Jean Naylor

Center for Health Promotion and Prevention Research, University of Texas Health Science Center at Houston School of Public Health, Houston, TX, 77204, USA

Maria Fernandez

Department of Family Practice, University of British Columbia (UBC) and Centre for Hip Health and Mobility, University Boulevard, Vancouver, BC, V6T 1Z3, Canada

Maureen C. Ashe

You can also search for this author in PubMed   Google Scholar

Contributions

NP and LW led the development of the manuscript. NP, LW, NP, MCA, PN, MF and SY contributed to the drafting and final approval of the manuscript.

Corresponding author

Correspondence to Nicole Pearson .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors have no financial or non-financial interests to declare .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1..

Example of a Hybrid Type 1 trial. Summary of publication by Cabassa et al.

Additional file 2.

Example of a Hybrid Type 2 trial. Summary of publication by Barnes et al.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Pearson, N., Naylor, PJ., Ashe, M.C. et al. Guidance for conducting feasibility and pilot studies for implementation trials. Pilot Feasibility Stud 6 , 167 (2020). https://doi.org/10.1186/s40814-020-00634-w

Download citation

Received : 08 January 2020

Accepted : 18 June 2020

Published : 31 October 2020

DOI : https://doi.org/10.1186/s40814-020-00634-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Feasibility
• Hybrid trial designs
• Implementation science

Pilot and Feasibility Studies

ISSN: 2055-5784

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

The Importance of Pilot Studies: Beginning the Hermeneutic Circle

• Published: June 2000
• Volume 41 , pages 385–400, ( 2000 )

Cite this article

• Adrianna Kezar 1  

1931 Accesses

11 Citations

3 Altmetric

Explore all metrics

This article suggests the importance of experiential knowledge and/or engagement of the researcher for developing understanding. One way to ground researchers' theoretical derived understanding is to conduct a pilot study. The hermeneutic circle, as described by Heidegger, provides a framework for understanding the importance of pilot studies; it suggests that a person must have a practical sense of the domain within which a phenomenon is situated in order to develop understanding. In this article, I present the many meaningful revisions to the theoretical framework and methodology that a pilot study allowed me to make within the research project. Two important implications that contribute to higher education research and practice are offered: (1) it illustrates the importance of grounding the research process in practical activity, and (2) it highlights how reflection can help to improve our research practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Reducing the Research-Practice Gap Through Problem-Based Methodology

Less Can Be More: Encouraging Mastery of Research Design in Undergraduate Research Methods

What is (good) practitioner research?

Alcoff, L. (1987). Cultural feminism versus post-structuralism: The identity crisis in feminist theory. Signs 13(3): 405–436.

Google Scholar  

Amey, M., and Tombley, S. (1992). Re-visioning leadership in community colleges. The Review of Higher Education 15(2): 125–150.

Astin, H., and Leland, C. (1991). Women of Influence, Women of Vision. San Francisco: Jossey-Bass.

Bensimon, E. (1989). A feminist reinterpretation of presidents definitions of leadership. Peabody Journal of Education 66(3): 143–156.

Bensimon, E., and Neumann, A. (1993). Redesigning Collegiate Leadership . Baltimore: Johns Hopkins University Press.

Birnbaum, R. (1992). How Academic Leadership Works. San Francisco: Jossey-Bass.

Bolman, L., and Deal, T. (1984). Modern Approaches to Understanding and Managing Organizations. San Francisco: Jossey-Bass.

Brown, D. (1986). Participatory research and community planning. In Barry Checkoway (ed.), Strategic Perspectives on Planning Practice , pp. 123–137. Lexington: Lexington Books.

Bruner, J. (1990). Acts of Meaning . Cambridge, MA: Harvard University Press.

Conrad, C., Hayworth, J., and Millar, S. (1993). A positioned subject approach to inquiry. In Clifton Conrad, Anna Neumann and Patricia Scott (eds.). Qualitative Research In Higher Education: Experiencing Alternative Perspectives And Approaches , pp. 267–277. Needham, MA: Ginn.

Denzin, N., and Lincoln. Y. (eds.). (1994). Handbook of Qualitative Research. Thousand Oaks, CA: Sage.

Dewey, J. (1938). Experience and Education . London: Collier-Macmillan.

Gaventa, J. (1988). Participatory research in North America. Convergence 21(23): 19–27.

Haraway, D. (1991). Simians, Cyborgs, and Women . New York: Routledge.

Harding, S. (1986). Introduction: Is there a feminist method? In Sandra Harding (ed.), Feminism and Methodology , pp. 1–15. Bloomington: Indiana University Press.

Harstock, N. (1986). The feminist standpoint: developing the ground for a specifically feminist historical materialism. In Sandra Harding (ed.), Feminism and Methodology, pp. 157–180. Bloomington: Indiana University.

Heidegger, M. (1962). Being and Time . New York: Harper and Row.

Helgesen, S. (1990). Women 's Way of Leading . New York: Doubleday.

Lather, P. (1986). Research as praxis. Harvard Educational Review 56(3): 257–277.

Lather, P. (1991). Getting Smart: Feminist Research and Pedagogy With/In the Postmodern . New York: Routledge.

Lincoln, Y., and Guba, E. (1985). Naturalistic Inquiry . Beverly Hills: Sage.

March, J. (1984). Emerging developments in the study of organizations. In R. Birnbaum (Ed.), ASHE reader on organization and governance in higher education (pp. 53–65). Needham Heights, MA: Ginn.

Mischler, E. (1986). Research Interviewing: Context and Narrative . Cambridge, MA: Harvard University Press.

Ochs, E. (1979). Transcription as theory. In E. Ochs and B. Schieffelin (eds.), Developmental Pragmatics, pp. 43–72. New York: Academic Press.

Reinharz, S. (1992). Feminist Methods in Social Research . New York: Oxford University.

Ricoeur, P. (1981). Hermeneutics and the Human Sciences . Cambridge: Cambridge University Press.

Riessman, C. (1993). Narrative Analysis. Beverly Hills: Sage.

Rosaldo, R. (1993). Culture and Truth . Boston: Beacon Press.

Rosener, J. (1990, November/December). Ways women lead. Harvard Business Review, 119–125.

Shotter, J. (1993). Cultural Politics of Everyday Life . Toronto: University of Toronto.

Spradley, J. (1979). The Ethnographic Interview . Florida: Harcourt Brace Jovanovich College Publishers.

Stanley, L. (1990). Feminist Praxis . New York: Routledge.

Strauss, A. (1987). Qualitative Analysis for Social Scientists . New York: Cambridge University Press.

Weick, K. (1995). Sense-making in organizations. Thousand Oaks, CA: Sage.

Whyte, W. (1991). Participatory Action Research . Newbury Park, CA: Sage Press.

Whyte, W., Greenwood, D., and Lazes, P. (1989). Participatory action research: through practice to science in social research. American Behavioral Scientist 32(5): 513–551.

Yukl, G. (1989). Managerial leadership: a review of the theory and research. Journal of Management 15(2): 251–289.

Download references

Author information

Authors and affiliations.

ERIC Clearinghouse on Higher Education, George Washington University, One Dupont Circle, Suite 630, Washington, DC, 20036

Adrianna Kezar

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

About this article

Kezar, A. The Importance of Pilot Studies: Beginning the Hermeneutic Circle. Research in Higher Education 41 , 385–400 (2000). https://doi.org/10.1023/A:1007047028758

Download citation

Issue Date : June 2000

DOI : https://doi.org/10.1023/A:1007047028758

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• High Education
• Pilot Study
• Important Implication
• Education Research
• Research Process
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 13 May 2024

What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review

• Olivia R. Phillips 1 , 2   na1 ,
• Cerian Harries 2 , 3   na1 ,
• Jo Leonardi-Bee 1 , 2 , 4   na1 ,
• Holly Knight 1 , 2 ,
• Lauren B. Sherar 2 , 3 ,
• Veronica Varela-Mato 2 , 3 &
• Joanne R. Morling 1 , 2 , 5  

Research Involvement and Engagement volume  10 , Article number:  48 ( 2024 ) Cite this article

103 Accesses

2 Altmetric

Metrics details

There is increasing interest in using patient and public involvement (PPI) in research to improve the quality of healthcare. Ordinarily, traditional methods have been used such as interviews or focus groups. However, these methods tend to engage a similar demographic of people. Thus, creative methods are being developed to involve patients for whom traditional methods are inaccessible or non-engaging.

To determine the strengths and limitations to using creative PPI methods in health and social care research.

Electronic searches were conducted over five databases on 14th April 2023 (Web of Science, PubMed, ASSIA, CINAHL, Cochrane Library). Studies that involved traditional, non-creative PPI methods were excluded. Creative PPI methods were used to engage with people as research advisors, rather than study participants. Only primary data published in English from 2009 were accepted. Title, abstract and full text screening was undertaken by two independent reviewers before inductive thematic analysis was used to generate themes.

Twelve papers met the inclusion criteria. The creative methods used included songs, poems, drawings, photograph elicitation, drama performance, visualisations, social media, photography, prototype development, cultural animation, card sorting and persona development. Analysis identified four limitations and five strengths to the creative approaches. Limitations included the time and resource intensive nature of creative PPI, the lack of generalisation to wider populations and ethical issues. External factors, such as the lack of infrastructure to support creative PPI, also affected their implementation. Strengths included the disruption of power hierarchies and the creation of a safe space for people to express mundane or “taboo” topics. Creative methods are also engaging, inclusive of people who struggle to participate in traditional PPI and can also be cost and time efficient.

‘Creative PPI’ is an umbrella term encapsulating many different methods of engagement and there are strengths and limitations to each. The choice of which should be determined by the aims and requirements of the research, as well as the characteristics of the PPI group and practical limitations. Creative PPI can be advantageous over more traditional methods, however a hybrid approach could be considered to reap the benefits of both. Creative PPI methods are not widely used; however, this could change over time as PPI becomes embedded even more into research.

Plain English Summary

It is important that patients and public are included in the research process from initial brainstorming, through design to delivery. This is known as public and patient involvement (PPI). Their input means that research closely aligns with their wants and needs. Traditionally to get this input, interviews and group discussions are held, but this can exclude people who find these activities non-engaging or inaccessible, for example those with language challenges, learning disabilities or memory issues. Creative methods of PPI can overcome this. This is a broad term describing different (non-traditional) ways of engaging patients and public in research, such as through the use or art, animation or performance. This review investigated the reasons why creative approaches to PPI could be difficult (limitations) or helpful (strengths) in health and social care research. After searching 5 online databases, 12 studies were included in the review. PPI groups included adults, children and people with language and memory impairments. Creative methods included songs, poems, drawings, the use of photos and drama, visualisations, Facebook, creating prototypes, personas and card sorting. Limitations included the time, cost and effort associated with creative methods, the lack of application to other populations, ethical issues and buy-in from the wider research community. Strengths included the feeling of equality between academics and the public, creation of a safe space for people to express themselves, inclusivity, and that creative PPI can be cost and time efficient. Overall, this review suggests that creative PPI is worthwhile, however each method has its own strengths and limitations and the choice of which will depend on the research project, PPI group characteristics and other practical limitations, such as time and financial constraints.

Peer Review reports

Introduction

Patient and public involvement (PPI) is the term used to describe the partnership between patients (including caregivers, potential patients, healthcare users etc.) or the public (a community member with no known interest in the topic) with researchers. It describes research that is done “‘with’ or ‘by’ the public, rather than ‘to,’ ‘about’ or ‘for’ them” [ 1 ]. In 2009, it became a legislative requirement for certain health and social care organisations to include patients, families, carers and communities in not only the planning of health and social care services, but the commissioning, delivery and evaluation of them too [ 2 ]. For example, funding applications for the National Institute of Health and Care Research (NIHR), a UK funding body, mandates a demonstration of how researchers plan to include patients/service users, the public and carers at each stage of the project [ 3 ]. However, this should not simply be a tokenistic, tick-box exercise. PPI should help formulate initial ideas and should be an instrumental, continuous part of the research process. Input from PPI can provide unique insights not yet considered and can ensure that research and health services are closely aligned to the needs and requirements of service users PPI also generally makes research more relevant with clearer outcomes and impacts [ 4 ]. Although this review refers to both patients and the public using the umbrella term ‘PPI’, it is important to acknowledge that these are two different groups with different motivations, needs and interests when it comes to health research and service delivery [ 5 ].

Despite continuing recognition of the need of PPI to improve quality of healthcare, researchers have also recognised that there is no ‘one size fits all’ method for involving patients [ 4 ]. Traditionally, PPI methods invite people to take part in interviews or focus groups to facilitate discussion, or surveys and questionnaires. However, these can sometimes be inaccessible or non-engaging for certain populations. For example, someone with communication difficulties may find it difficult to engage in focus groups or interviews. If individuals lack the appropriate skills to interact in these types of scenarios, they cannot take advantage of the participation opportunities it can provide [ 6 ]. Creative methods, however, aim to resolve these issues. These are a relatively new concept whereby researchers use creative methods (e.g., artwork, animations, Lego), to make PPI more accessible and engaging for those whose voices would otherwise go unheard. They ensure that all populations can engage in research, regardless of their background or skills. Seminal work has previously been conducted in this area, which brought to light the use of creative methodologies in research. Leavy (2008) [ 7 ] discussed how traditional interviews had limits on what could be expressed due to their sterile, jargon-filled and formulaic structure, read by only a few specialised academics. It was this that called for more creative approaches, which included narrative enquiry, fiction-based research, poetry, music, dance, art, theatre, film and visual art. These practices, which can be used in any stage of the research cycle, supported greater empathy, self-reflection and longer-lasting learning experiences compared to interviews [ 7 ]. They also pushed traditional academic boundaries, which made the research accessible not only to researchers, but the public too. Leavy explains that there are similarities between arts-based approaches and scientific approaches: both attempts to investigate what it means to be human through exploration, and used together, these complimentary approaches can progress our understanding of the human experience [ 7 ]. Further, it is important to acknowledge the parallels and nuances between creative and inclusive methods of PPI. Although creative methods aim to be inclusive (this should underlie any PPI activity, whether creative or not), they do not incorporate all types of accessible, inclusive methodologies e.g., using sign language for people with hearing impairments or audio recordings for people who cannot read. Given that there was not enough scope to include an evaluation of all possible inclusive methodologies, this review will focus on creative methods of PPI only.

We aimed to conduct a qualitative systematic review to highlight the strengths of creative PPI in health and social care research, as well as the limitations, which might act as a barrier to their implementation. A qualitative systematic review “brings together research on a topic, systematically searching for research evidence from primary qualitative studies and drawing the findings together” [ 8 ]. This review can then advise researchers of the best practices when designing PPI.

Public involvement

The PHIRST-LIGHT Public Advisory Group (PAG) consists of a team of experienced public contributors with a diverse range of characteristics from across the UK. The PAG was involved in the initial question setting and study design for this review.

Search strategy

For the purpose of this review, the JBI approach for conducting qualitative systematic reviews was followed [ 9 ]. The search terms were (“creativ*” OR “innovat*” OR “authentic” OR “original” OR “inclu*”) AND (“public and patient involvement” OR “patient and public involvement” OR “public and patient involvement and engagement” OR “patient and public involvement and engagement” OR “PPI” OR “PPIE” OR “co-produc*” OR “co-creat*” OR “co-design*” OR “cooperat*” OR “co-operat*”). This search string was modified according to the requirements of each database. Papers were filtered by title, abstract and keywords (see Additional file 1 for search strings). The databases searched included Web of Science (WoS), PubMed, ASSIA and CINAHL. The Cochrane Library was also searched to identify relevant reviews which could lead to the identification of primary research. The search was conducted on 14/04/23. As our aim was to report on the use of creative PPI in research, rather than more generic public engagement, we used electronic databases of scholarly peer-reviewed literature, which represent a wide range of recognised databases. These identified studies published in general international journals (WoS, PubMed), those in social sciences journals (ASSIA), those in nursing and allied health journals (CINAHL), and trials of interventions (Cochrane Library).

Inclusion criteria

Only full-text, English language, primary research papers from 2009 to 2023 were included. This was the chosen timeframe as in 2009 the Health and Social Reform Act made it mandatory for certain Health and Social Care organisations to involve the public and patients in planning, delivering, and evaluating services [ 2 ]. Only creative methods of PPI were accepted, rather than traditional methods, such as interviews or focus groups. For the purposes of this paper, creative PPI included creative art or arts-based approaches (e.g., e.g. stories, songs, drama, drawing, painting, poetry, photography) to enhance engagement. Titles were related to health and social care and the creative PPI was used to engage with people as research advisors, not as study participants. Meta-analyses, conference abstracts, book chapters, commentaries and reviews were excluded. There were no limits concerning study location or the demographic characteristics of the PPI groups. Only qualitative data were accepted.

Quality appraisal

Quality appraisal using the Critical Appraisal Skills Programme (CASP) checklist [ 10 ] was conducted by the primary authors (ORP and CH). This was done independently, and discrepancies were discussed and resolved. If a consensus could not be reached, a third independent reviewer was consulted (JRM). The full list of quality appraisal questions can be found in Additional file 2 .

Data extraction

ORP extracted the study characteristics and a subset of these were checked by CH. Discrepancies were discussed and amendments made. Extracted data included author, title, location, year of publication, year study was carried out, research question/aim, creative methods used, number of participants, mean age, gender, ethnicity of participants, setting, limitations and strengths of creative PPI and main findings.

Data analysis

The included studies were analysed using inductive thematic analysis [ 11 ], where themes were determined by the data. The familiarisation stage took place during full-text reading of the included articles. Anything identified as a strength or limitation to creative PPI methods was extracted verbatim as an initial code and inputted into the data extraction Excel sheet. Similar codes were sorted into broader themes, either under ‘strengths’ or ‘limitations’ and reviewed. Themes were then assigned a name according to the codes.

The search yielded 9978 titles across the 5 databases: Web of Science (1480 results), PubMed (94 results), ASSIA (2454 results), CINAHL (5948 results) and Cochrane Library (2 results), resulting in 8553 different studies after deduplication. ORP and CH independently screened their titles and abstracts, excluding those that did not meet the criteria. After assessment, 12 studies were included (see Fig.  1 ).

PRISMA flowchart of the study selection process

Study characteristics

The included studies were published between 2018 and 2022. Seven were conducted in the UK [ 12 , 14 , 15 , 17 , 18 , 19 , 23 ], two in Canada [ 21 , 22 ], one in Australia [ 13 ], one in Norway [ 16 ] and one in Ireland [ 20 ]. The PPI activities occurred across various settings, including a school [ 12 ], social club [ 12 ], hospital [ 17 ], university [ 22 ], theatre [ 19 ], hotel [ 20 ], or online [ 15 , 21 ], however this information was omitted in 5 studies [ 13 , 14 , 16 , 18 , 23 ]. The number of people attending the PPI sessions varied, ranging from 6 to 289, however the majority (ten studies) had less than 70 participants [ 13 , 14 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ]. Seven studies did not provide information on the age or gender of the PPI groups. Of those that did, ages ranged from 8 to 76 and were mostly female. The ethnicities of the PPI group members were also rarely recorded (see Additional file 3 for data extraction table).

Types of creative methods

The type of creative methods used to engage the PPI groups were varied. These included songs, poems, drawings, photograph elicitation, drama performance, visualisations, Facebook, photography, prototype development, cultural animation, card sorting and creating personas (see Table  1 ). These were sometimes accompanied by traditional methods of PPI such as interviews and focus group discussions.

The 12 included studies were all deemed to be of good methodological quality, with scores ranging from 6/10 to 10/10 with the CASP critical appraisal tool [ 10 ] (Table  2 ).

Thematic analysis

Analysis identified four limitations and five strengths to creative PPI (see Fig.  2 ). Limitations included the time and resource intensity of creative PPI methods, its lack of generalisation, ethical issues and external factors. Strengths included the disruption of power hierarchies, the engaging and inclusive nature of the methods and their long-term cost and time efficiency. Creative PPI methods also allowed mundane and “taboo” topics to be discussed within a safe space.

Theme map of strengths and limitations

Limitations of creative PPI

Creative ppi methods are time and resource intensive.

The time and resource intensive nature of creative PPI methods is a limitation, most notably for the persona-scenario methodology. Valaitis et al. [ 22 ] used 14 persona-scenario workshops with 70 participants to co-design a healthcare intervention, which aimed to promote optimal aging in Canada. Using the persona method, pairs composed of patients, healthcare providers, community service providers and volunteers developed a fictional character which they believed represented an ‘end-user’ of the healthcare intervention. Due to the depth and richness of the data produced the authors reported that it was time consuming to analyse. Further, they commented that the amount of information was difficult to disseminate to scientific leads and present at team meetings. Additionally, to ensure the production of high-quality data, to probe for details and lead group discussion there was a need for highly skilled facilitators. The resource intensive nature of the creative co-production was also noted in a study using the persona scenario and creative worksheets to develop a prototype decision support tool for individuals with malignant pleural effusion [ 17 ]. With approximately 50 people, this was also likely to yield a high volume of data to consider.

To prepare materials for populations who cannot engage in traditional methods of PPI was also timely. Kearns et al. [ 18 ] developed a feedback questionnaire for people with aphasia to evaluate ICT-delivered rehabilitation. To ensure people could participate effectively, the resources used during the workshops, such as PowerPoints, online images and photographs, had to be aphasia-accessible, which was labour and time intensive. The author warned that this time commitment should not be underestimated.

There are further practical limitations to implementing creative PPI, such as the costs of materials for activities as well as hiring a space for workshops. For example, the included studies in this review utilised pens, paper, worksheets, laptops, arts and craft supplies and magazines and took place in venues such as universities, a social club, and a hotel. Further, although not limited to creative PPI methods exclusively but rather most studies involving the public, a financial incentive was often offered for participation, as well as food, parking, transport and accommodation [ 21 , 22 ].

Creative PPI lacks generalisation

Another barrier to the use of creative PPI methods in health and social care research was the individual nature of its output. Those who participate, usually small in number, produce unique creative outputs specific to their own experiences, opinions and location. Craven et al. [ 13 ], used arts-based visualisations to develop a toolbox for adults with mental health difficulties. They commented, “such an approach might still not be worthwhile”, as the visualisations were individualised and highly personal. This indicates that the output may fail to meet the needs of its end-users. Further, these creative PPI groups were based in certain geographical regions such as Stoke-on-Trent [ 19 ] Sheffield [ 23 ], South Wales [ 12 ] or Ireland [ 20 ], which limits the extent the findings can be applied to wider populations, even within the same area due to individual nuances. Further, the study by Galler et al. [ 16 ], is specific to the Norwegian context and even then, maybe only a sub-group of the Norwegian population as the sample used was of higher socioeconomic status.

However, Grindell et al. [ 17 ], who used persona scenarios, creative worksheets and prototype development, pointed out that the purpose of this type of research is to improve a certain place, rather than apply findings across other populations and locations. Individualised output may, therefore, only be a limitation to research wanting to conduct PPI on a large scale.

If, however, greater generalisation within PPI is deemed necessary, then social media may offer a resolution. Fedorowicz et al. [ 15 ], used Facebook to gain feedback from the public on the use of video-recording methodology for an upcoming project. This had the benefit of including a more diverse range of people (289 people joined the closed group), who were spread geographically around the UK, as well as seven people from overseas.

Creative PPI has ethical issues

As with other research, ethical issues must be taken into consideration. Due to the nature of creative approaches, as well as the personal effort put into them, people often want to be recognised for their work. However, this compromises principles so heavily instilled in research such as anonymity and confidentiality. With the aim of exploring issues related to health and well-being in a town in South Wales, Byrne et al. [ 12 ], asked year 4/5 and year 10 pupils to create poems, songs, drawings and photographs. Community members also created a performance, mainly of monologues, to explore how poverty and inequalities are dealt with. Byrne noted the risks of these arts-based approaches, that being the possibility of over-disclosure and consequent emotional distress, as well as people’s desire to be named for their work. On one hand, the anonymity reduces the sense of ownership of the output as it does not portray a particular individual’s lived experience anymore. On the other hand, however, it could promote a more honest account of lived experience. Supporting this, Webber et al. [ 23 ], who used the persona method to co-design a back pain educational resource prototype, claimed that the anonymity provided by this creative technique allowed individuals to externalise and anonymise their own personal experience, thus creating a more authentic and genuine resource for future users. This implies that anonymity can be both a limitation and strength here.

The use of creative PPI methods is impeded by external factors

Despite the above limitations influencing the implementation of creative PPI techniques, perhaps the most influential is that creative methodologies are simply not mainstream [ 19 ]. This could be linked to the issues above, like time and resource intensity, generalisation and ethical issues but it is also likely to involve more systemic factors within the research community. Micsinszki et al. [ 21 ], who co-designed a hub for the health and well-being of vulnerable populations, commented that there is insufficient infrastructure to conduct meaningful co-design as well as a dominant medical model. Through a more holistic lens, there are “sociopolitical environments that privilege individualism over collectivism, self-sufficiency over collaboration, and scientific expertise over other ways of knowing based on lived experience” [ 21 ]. This, it could be suggested, renders creative co-design methodologies, which are based on the foundations of collectivism, collaboration and imagination an invalid technique in the research field, which is heavily dominated by more scientific methods offering reproducibility, objectivity and reliability.

Although we acknowledge that creative PPI techniques are not always appropriate, it may be that their main limitation is the lack of awareness of these methods or lack of willingness to use them. Further, there is always the risk that PPI, despite being a mandatory part of research, is used in a tokenistic or tick-box fashion [ 20 ], without considering the contribution that meaningful PPI could make to enhancing the research. It may be that PPI, let alone creative PPI, is not at the forefront of researchers’ minds when planning research.

Strengths of creative PPI

Creative ppi disrupts power hierarchies.

One of the main strengths of creative PPI techniques, cited most frequently in the included literature, was that they disrupt traditional power hierarchies [ 12 , 13 , 17 , 19 , 23 ]. For example, the use of theatre performance blurred the lines between professional and lay roles between the community and policy makers [ 12 ]. Individuals created a monologue to portray how poverty and inequality impact daily life and presented this to representatives of the National Assembly of Wales, Welsh Government, the Local Authority, Arts Council and Westminster. Byrne et al. [ 12 ], states how this medium allowed the community to engage with the people who make decisions about their lives in an environment of respect and understanding, where the hierarchies are not as visible as in other settings, e.g., political surgeries. Creative PPI methods have also removed traditional power hierarchies between researchers and adolescents. Cook et al. [ 13 ], used arts-based approaches to explore adolescents’ ideas about the “perfect” condom. They utilised the “Life Happens” resource, where adolescents drew and then decorated a person with their thoughts about sexual relationships, not too dissimilar from the persona-scenario method. This was then combined with hypothetical scenarios about sexuality. A condom-mapping exercise was then implemented, where groups shared the characteristics that make a condom “perfect” on large pieces of paper. Cook et al. [ 13 ], noted that usually power imbalances make it difficult to elicit information from adolescents, however these power imbalances were reduced due to the use of creative co-design techniques.

The same reduction in power hierarchies was noted by Grindell et al. [ 17 ], who used the person-scenario method and creative worksheets with individuals with malignant pleural effusion. This was with the aim of developing a prototype of a decision support tool for patients to help with treatment options. Although this process involved a variety of stakeholders, such as patients, carers and healthcare professionals, creative co-design was cited as a mechanism that worked to reduce power imbalances – a limitation of more traditional methods of research. Creative co-design blurred boundaries between end-users and clinical staff and enabled the sharing of ideas from multiple, valuable perspectives, meaning the prototype was able to suit user needs whilst addressing clinical problems.

Similarly, a specific creative method named cultural animation was also cited to dissolve hierarchies and encourage equal contributions from participants. Within this arts-based approach, Keleman et al. [ 19 ], explored the concept of “good health” with individuals from Stoke-on Trent. Members of the group created art installations using ribbons, buttons, cardboard and straws to depict their idea of a “healthy community”, which was accompanied by a poem. They also created a 3D Facebook page and produced another poem or song addressing the government to communicate their version of a “picture of health”. Public participants said that they found the process empowering, honest, democratic, valuable and practical.

This dissolving of hierarchies and levelling of power is beneficial as it increases the sense of ownership experienced by the creators/producers of the output [ 12 , 17 , 23 ]. This is advantageous as it has been suggested to improve its quality [ 23 ].

Creative PPI allows the unsayable to be said

Creative PPI fosters a safe space for mundane or taboo topics to be shared, which may be difficult to communicate using traditional methods of PPI. For example, the hypothetical nature of condom mapping and persona-scenarios meant that adolescents could discuss a personal topic without fear of discrimination, judgement or personal disclosure [ 13 ]. The safe space allowed a greater volume of ideas to be generated amongst peers where they might not have otherwise. Similarly, Webber et al. [ 23 ], , who used the persona method to co-design the prototype back pain educational resource, also noted how this method creates anonymity whilst allowing people the opportunity to externalise personal experiences, thoughts and feelings. Other creative methods were also used, such as drawing, collaging, role play and creating mood boards. A cardboard cube (labelled a “magic box”) was used to symbolise a physical representation of their final prototype. These creative methods levelled the playing field and made personal experiences accessible in a safe, open environment that fostered trust, as well as understanding from the researchers.

It is not only sensitive subjects that were made easier to articulate through creative PPI. The communication of mundane everyday experiences were also facilitated, which were deemed typically ‘unsayable’. This was specifically given in the context of describing intangible aspects of everyday health and wellbeing [ 11 ]. Graphic designers can also be used to visually represent the outputs of creative PPI. These captured the movement and fluidity of people and well as the relationships between them - things that cannot be spoken but can be depicted [ 21 ].

Creative PPI methods are inclusive

Another strength of creative PPI was that it is inclusive and accessible [ 17 , 19 , 21 ]. The safe space it fosters, as well as the dismantling of hierarchies, welcomed people from a diverse range of backgrounds and provided equal opportunities [ 21 ], especially for those with communication and memory difficulties who might be otherwise excluded from PPI. Kelemen et al. [ 19 ], who used creative methods to explore health and well-being in Stoke-on-Trent, discussed how people from different backgrounds came together and connected, discussed and reached a consensus over a topic which evoked strong emotions, that they all have in common. Individuals said that the techniques used “sets people to open up as they are not overwhelmed by words”. Similarly, creative activities, such as the persona method, have been stated to allow people to express themselves in an inclusive environment using a common language. Kearns et al. [ 18 ], who used aphasia-accessible material to develop a questionnaire with aphasic individuals, described how they felt comfortable in contributing to workshops (although this material was time-consuming to make, see ‘Limitations of creative PPI’ ).

Despite the general inclusivity of creative PPI, it can also be exclusive, particularly if online mediums are used. Fedorowicz et al. [ 15 ], used Facebook to create a PPI group, and although this may rectify previous drawbacks about lack of generalisation of creative methods (as Facebook can reach a greater number of people, globally), it excluded those who are not digitally active or have limited internet access or knowledge of technology. Online methods have other issues too. Maintaining the online group was cited as challenging and the volume of responses required researchers to interact outside of their working hours. Despite this, online methods like Facebook are very accessible for people who are physically disabled.

Creative PPI methods are engaging

The process of creative PPI is typically more engaging and produces more colourful data than traditional methods [ 13 ]. Individuals are permitted and encouraged to explore a creative self [ 19 ], which can lead to the exploration of new ideas and an overall increased enjoyment of the process. This increased engagement is particularly beneficial for younger PPI groups. For example, to involve children in the development of health food products, Galler et al. [ 16 ] asked 9-12-year-olds to take photos of their food and present it to other children in a “show and tell” fashion. They then created a newspaper article describing a new healthy snack. In this creative focus group, children were given lab coats to further their identity as inventors. Galler et al. [ 16 ], notes that the methods were highly engaging and facilitated teamwork and group learning. This collaborative nature of problem-solving was also observed in adults who used personas and creative worksheets to develop the resource for lower back pain [ 23 ]. Dementia patients too have been reported to enjoy the creative and informal approach to idea generation [ 20 ].

The use of cultural animation allowed people to connect with each other in a way that traditional methods do not [ 19 , 21 ]. These connections were held in place by boundary objects, such as ribbons, buttons, fabric and picture frames, which symbolised a shared meaning between people and an exchange of knowledge and emotion. Asking groups to create an art installation using these objects further fostered teamwork and collaboration, both at an individual and collective level. The exploration of a creative self increased energy levels and encouraged productive discussions and problem-solving [ 19 ]. Objects also encouraged a solution-focused approach and permitted people to think beyond their usual everyday scope [ 17 ]. They also allowed facilitators to probe deeper about the greater meanings carried by the object, which acted as a metaphor [ 21 ].

From the researcher’s point of view, co-creative methods gave rise to ideas they might not have initially considered. Valaitis et al. [ 22 ], found that over 40% of the creative outputs were novel ideas brought to light by patients, healthcare providers/community care providers, community service providers and volunteers. One researcher commented, “It [the creative methods] took me on a journey, in a way that when we do other pieces of research it can feel disconnected” [ 23 ]. Another researcher also stated they could not return to the way they used to do research, as they have learnt so much about their own health and community and how they are perceived [ 19 ]. This demonstrates that creative processes not only benefit the project outcomes and the PPI group, but also facilitators and researchers. However, although engaging, creative methods have been criticised for not demonstrating academic rigour [ 17 ]. Moreover, creative PPI may also be exclusive to people who do not like or enjoy creative activities.

Creative PPI methods are cost and time efficient

Creative PPI workshops can often produce output that is visible and tangible. This can save time and money in the long run as the output is either ready to be implemented in a healthcare setting or a first iteration has already been developed. This may also offset the time and costs it takes to implement creative PPI. For example, the prototype of the decision support tool for people with malignant pleural effusion was developed using personas and creative worksheets. The end result was two tangible prototypes to drive the initial idea forward as something to be used in practice [ 17 ]. The use of creative co-design in this case saved clinician time as well as the time it would take to develop this product without the help of its end-users. In the development of this particular prototype, analysis was iterative and informed the next stage of development, which again saved time. The same applies for the feedback questionnaire for the assessment of ICT delivered aphasia rehabilitation. The co-created questionnaire, designed with people with aphasia, was ready to be used in practice [ 18 ]. This suggests that to overcome time and resource barriers to creative PPI, researchers should aim for it to be engaging whilst also producing output.

That useable products are generated during creative workshops signals to participating patients and public members that they have been listened to and their thoughts and opinions acted upon [ 23 ]. For example, the development of the back pain resource based on patient experiences implies that their suggestions were valid and valuable. Further, those who participated in the cultural animation workshop reported that the process visualises change, and that it already feels as though the process of change has started [ 19 ].

The most cost and time efficient method of creative PPI in this review is most likely the use of Facebook to gather feedback on project methodology [ 15 ]. Although there were drawbacks to this, researchers could involve more people from a range of geographical areas at little to no cost. Feedback was instantaneous and no training was required. From the perspective of the PPI group, they could interact however much or little they wish with no time commitment.

This systematic review identified four limitations and five strengths to the use of creative PPI in health and social care research. Creative PPI is time and resource intensive, can raise ethical issues and lacks generalisability. It is also not accepted by the mainstream. These factors may act as barriers to the implementation of creative PPI. However, creative PPI disrupts traditional power hierarchies and creates a safe space for taboo or mundane topics. It is also engaging, inclusive and can be time and cost efficient in the long term.

Something that became apparent during data analysis was that these are not blanket strengths and limitations of creative PPI as a whole. The umbrella term ‘creative PPI’ is broad and encapsulates a wide range of activities, ranging from music and poems to prototype development and persona-scenarios, to more simplistic things like the use of sticky notes and ordering cards. Many different activities can be deemed ‘creative’ and the strengths and limitations of one does not necessarily apply to another. For example, cultural animation takes greater effort to prepare than the use of sticky notes and sorting cards, and the use of Facebook is cheaper and wider reaching than persona development. Researchers should use their discretion and weigh up the benefits and drawbacks of each method to decide on a technique which suits the project. What might be a limitation to creative PPI in one project may not be in another. In some cases, creative PPI may not be suitable at all.

Furthermore, the choice of creative PPI method also depends on the needs and characteristics of the PPI group. Children, adults and people living with dementia or language difficulties all have different engagement needs and capabilities. This indicates that creative PPI is not one size fits all and that the most appropriate method will change depending on the composition of the group. The choice of method will also be determined by the constraints of the research project, namely time, money and the research aim. For example, if there are time constraints, then a method which yields a lot of data and requires a lot of preparation may not be appropriate. If generalisation is important, then an online method is more suitable. Together this indicates that the choice of creative PPI method is highly individualised and dependent on multiple factors.

Although the limitations discussed in this review apply to creative PPI, they are not exclusive to creative PPI. Ethical issues are a consideration within general PPI research, especially when working with more vulnerable populations, such as children or adults living with a disability. It can also be the case that traditional PPI methods lack generalisability, as people who volunteer to be part of such a group are more likely be older, middle class and retired [ 24 ]. Most research is vulnerable to this type of bias, however, it is worth noting that generalisation is not always a goal and research remains valid and meaningful in its absence. Although online methods may somewhat combat issues related to generalisability, these methods still exclude people who do not have access to the internet/technology or who choose not to use it, implying that online PPI methods may not be wholly representative of the general population. Saying this, however, the accessibility of creative PPI techniques differs from person to person, and for some, online mediums may be more accessible (for example for those with a physical disability), and for others, this might be face-to-face. To combat this, a range of methods should be implemented. Planning multiple focus group and interviews for traditional PPI is also time and resource intensive, however the extra resources required to make this creative may be even greater. Although, the rich data provided may be worth the preparation and analysis time, which is also likely to depend on the number of participants and workshop sessions required. PPI, not just creative PPI, often requires the provision of a financial incentive, refreshments, parking and accommodation, which increase costs. These, however, are imperative and non-negotiable, as they increase the accessibility of research, especially to minority and lower-income groups less likely to participate. Adequate funding is also important for co-design studies where repeated engagement is required. One barrier to implementation, which appears to be exclusive to creative methods, however, is that creative methods are not mainstream. This cannot be said for traditional PPI as this is often a mandatory part of research applications.

Regarding the strengths of creative PPI, it could be argued that most appear to be exclusive to creative methodologies. These are inclusive by nature as multiple approaches can be taken to evoke ideas from different populations - approaches that do not necessarily rely on verbal or written communication like interviews and focus groups do. Given the anonymity provided by some creative methods, such as personas, people may be more likely to discuss their personal experiences under the guise of a general end-user, which might be more difficult to maintain when an interviewer is asking an individual questions directly. Additionally, creative methods are by nature more engaging and interactive than traditional methods, although this is a blanket statement and there may be people who find the question-and-answer/group discussion format more engaging. Creative methods have also been cited to eliminate power imbalances which exist in traditional research [ 12 , 13 , 17 , 19 , 23 ]. These imbalances exist between researchers and policy makers and adolescents, adults and the community. Lastly, although this may occur to a greater extent in creative methods like prototype development, it could be suggested that PPI in general – regardless of whether it is creative - is more time and cost efficient in the long-term than not using any PPI to guide or refine the research process. It must be noted that these are observations based on the literature. To be certain these differences exist between creative and traditional methods of PPI, direct empirical evaluation of both should be conducted.

To the best of our knowledge, this is the first review to identify the strengths and limitations to creative PPI, however, similar literature has identified barriers and facilitators to PPI in general. In the context of clinical trials, recruitment difficulties were cited as a barrier, as well as finding public contributors who were free during work/school hours. Trial managers reported finding group dynamics difficult to manage and the academic environment also made some public contributors feel nervous and lacking confidence to speak. Facilitators, however, included the shared ownership of the research – something that has been identified in the current review too. In addition, planning and the provision of knowledge, information and communication were also identified as facilitators [ 25 ]. Other research on the barriers to meaningful PPI in trial oversight committees included trialist confusion or scepticism over the PPI role and the difficulties in finding PPI members who had a basic understanding of research [ 26 ]. However, it could be argued that this is not representative of the average patient or public member. The formality of oversight meetings and the technical language used also acted as a barrier, which may imply that the informal nature of creative methods and its lack of dependency on literacy skills could overcome this. Further, a review of 42 reviews on PPI in health and social care identified financial compensation, resources, training and general support as necessary to conduct PPI, much like in the current review where the resource intensiveness of creative PPI was identified as a limitation. However, others were identified too, such as recruitment and representativeness of public contributors [ 27 ]. Like in the current review, power imbalances were also noted, however this was included as both a barrier and facilitator. Collaboration seemed to diminish hierarchies but not always, as sometimes these imbalances remained between public contributors and healthcare staff, described as a ‘them and us’ culture [ 27 ]. Although these studies compliment the findings of the current review, a direct comparison cannot be made as they do not concern creative methods. However, it does suggest that some strengths and weaknesses are shared between creative and traditional methods of PPI.

Strengths and limitations of this review

Although a general definition of creative PPI exists, it was up to our discretion to decide exactly which activities were deemed as such for this review. For example, we included sorting cards, the use of interactive whiteboards and sticky notes. Other researchers may have a more or less stringent criteria. However, two reviewers were involved in this decision which aids the reliability of the included articles. Further, it may be that some of the strengths and limitations cannot fully be attributed to the creative nature of the PPI process, but rather their co-created nature, however this is hard to disentangle as the included papers involved both these aspects.

During screening, it was difficult to decide whether the article was utilising creative qualitative methodology or creative PPI , as it was often not explicitly labelled as such. Regardless, both approaches involved the public/patients refining a healthcare product/service. This implies that if this review were to be replicated, others may do it differently. This may call for greater standardisation in the reporting of the public’s involvement in research. For example, the NIHR outlines different approaches to PPI, namely “consultation”, “collaboration”, “co-production” and “user-controlled”, which each signify an increased level of public power and influence [ 28 ]. Papers with elements of PPI could use these labels to clarify the extent of public involvement, or even explicitly state that there was no PPI. Further, given our decision to include only scholarly peer-reviewed literature, it is possible that data were missed within the grey literature. Similarly, the literature search will not have identified all papers relating to different types of accessible inclusion. However, the intent of the review was to focus solely on those within the definition of creative.

This review fills a gap in the literature and helps circulate and promote the concept of creative PPI. Each stage of this review, namely screening and quality appraisal, was conducted by two independent reviewers. However, four full texts could not be accessed during the full text reading stage, meaning there are missing data that could have altered or contributed to the findings of this review.

Research recommendations

Given that creative PPI can require effort to prepare, perform and analyse, sufficient time and funding should be allocated in the research protocol to enable meaningful and continuous PPI. This is worthwhile as PPI can significantly change the research output so that it aligns closely with the needs of the group it is to benefit. Researchers should also consider prototype development as a creative PPI activity as this might reduce future time/resource constraints. Shifting from a top-down approach within research to a bottom-up can be advantageous to all stakeholders and can help move creative PPI towards the mainstream. This, however, is the collective responsibility of funding bodies, universities and researchers, as well as committees who approve research bids.

A few of the included studies used creative techniques alongside traditional methods, such as interviews, which could also be used as a hybrid method of PPI, perhaps by researchers who are unfamiliar with creative techniques or to those who wish to reap the benefits of both. Often the characteristics of the PPI group were not included, including age, gender and ethnicity. It would be useful to include such information to assess how representative the PPI group is of the population of interest.

Creative PPI is a relatively novel approach of engaging the public and patients in research and it has both advantages and disadvantages compared to more traditional methods. There are many approaches to implementing creative PPI and the choice of technique will be unique to each piece of research and is reliant on several factors. These include the age and ability of the PPI group as well as the resource limitations of the project. Each method has benefits and drawbacks, which should be considered at the protocol-writing stage. However, given adequate funding, time and planning, creative PPI is a worthwhile and engaging method of generating ideas with end-users of research – ideas which may not be otherwise generated using traditional methods.

Data availability

No datasets were generated or analysed during the current study.

Abbreviations

Critical Appraisal Skills Programme

The Joanna Briggs Institute

National Institute of Health and Care Research

Public Advisory Group

Public and Patient Involvement

Web of Science

National Institute for Health and Care Research. What Is Patient and Public Involvement and Public Engagement? https://www.spcr.nihr.ac.uk/PPI/what-is-patient-and-public-involvement-and-engagement Accessed 01 Sept 2023.

Department of Health. Personal and Public Involvement (PPI) https://www.health-ni.gov.uk/topics/safety-and-quality-standards/personal-and-public-involvement-ppi#:~:text=The Health and Social Care Reform Act (NI) 2009 placed,delivery and evaluation of services . Accessed 01 Sept 2023.

National Institute for Health and Care Research. Policy Research Programme – Guidance for Stage 1 Applications https://www.nihr.ac.uk/documents/policy-research-programme-guidance-for-stage-1-applications-updated/26398 Accessed 01 Sept 2023.

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, Chant A. Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot. Health Expect. 2019. https://doi.org/10.1111/hex.12888

Article   PubMed   PubMed Central   Google Scholar  

Street JM, Stafinski T, Lopes E, Menon D. Defining the role of the public in health technology assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care. 2020. https://doi.org/10.1017/S0266462320000094

Article   PubMed   Google Scholar  

Morrison C, Dearden A. Beyond tokenistic participation: using representational artefacts to enable meaningful public participation in health service design. Health Policy. 2013. https://doi.org/10.1016/j.healthpol.2013.05.008

Leavy P. Method meets art: arts-Based Research Practice. New York: Guilford; 2020.

Google Scholar  

Seers K. Qualitative systematic reviews: their importance for our understanding of research relevant to pain. Br J Pain. 2015. https://doi.org/10.1177/2049463714549777

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis JBI. 2020. https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03

CASP. CASP Checklists https://casp-uk.net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/CASP-Qualitative-Checklist-2018_fillable_form.pdf (2022).

Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Res Psychol. 2006. https://doi.org/10.1191/1478088706qp063oa

Article   Google Scholar  

Byrne E, Elliott E, Saltus R, Angharad J. The creative turn in evidence for public health: community and arts-based methodologies. J Public Health. 2018. https://doi.org/10.1093/pubmed/fdx151

Cook S, Grozdanovski L, Renda G, Santoso D, Gorkin R, Senior K. Can you design the perfect condom? Engaging young people to inform safe sexual health practice and innovation. Sex Educ. 2022. https://doi.org/10.1080/14681811.2021.1891040

Craven MP, Goodwin R, Rawsthorne M, Butler D, Waddingham P, Brown S, Jamieson M. Try to see it my way: exploring the co-design of visual presentations of wellbeing through a workshop process. Perspect Public Health. 2019. https://doi.org/10.1177/1757913919835231

Fedorowicz S, Riley V, Cowap L, Ellis NJ, Chambers R, Grogan S, Crone D, Cottrell E, Clark-Carter D, Roberts L, Gidlow CJ. Using social media for patient and public involvement and engagement in health research: the process and impact of a closed Facebook group. Health Expect. 2022. https://doi.org/10.1111/hex.13515

Galler M, Myhrer K, Ares G, Varela P. Listening to children voices in early stages of new product development through co-creation – creative focus group and online platform. Food Res Int. 2022. https://doi.org/10.1016/j.foodres.2022.111000

Grindell C, Tod A, Bec R, Wolstenholme D, Bhatnagar R, Sivakumar P, Morley A, Holme J, Lyons J, Ahmed M, Jackson S, Wallace D, Noorzad F, Kamalanathan M, Ahmed L, Evison M. Using creative co-design to develop a decision support tool for people with malignant pleural effusion. BMC Med Inf Decis Mak. 2020. https://doi.org/10.1186/s12911-020-01200-3

Kearns Á, Kelly H, Pitt I. Rating experience of ICT-delivered aphasia rehabilitation: co-design of a feedback questionnaire. Aphasiology. 2020. https://doi.org/10.1080/02687038.2019.1649913

Kelemen M, Surman E, Dikomitis L. Cultural animation in health research: an innovative methodology for patient and public involvement and engagement. Health Expect. 2018. https://doi.org/10.1111/hex.12677

Keogh F, Carney P, O’Shea E. Innovative methods for involving people with dementia and carers in the policymaking process. Health Expect. 2021. https://doi.org/10.1111/hex.13213

Micsinszki SK, Buettgen A, Mulvale G, Moll S, Wyndham-West M, Bruce E, Rogerson K, Murray-Leung L, Fleisig R, Park S, Phoenix M. Creative processes in co-designing a co-design hub: towards system change in health and social services in collaboration with structurally vulnerable populations. Evid Policy. 2022. https://doi.org/10.1332/174426421X16366319768599

Valaitis R, Longaphy J, Ploeg J, Agarwal G, Oliver D, Nair K, Kastner M, Avilla E, Dolovich L. Health TAPESTRY: co-designing interprofessional primary care programs for older adults using the persona-scenario method. BMC Fam Pract. 2019. https://doi.org/10.1186/s12875-019-1013-9

Webber R, Partridge R, Grindell C. The creative co-design of low back pain education resources. Evid Policy. 2022. https://doi.org/10.1332/174426421X16437342906266

National Institute for Health and Care Research. A Researcher’s Guide to Patient and Public Involvement. https://oxfordbrc.nihr.ac.uk/wp-content/uploads/2017/03/A-Researchers-Guide-to-PPI.pdf Accessed 01 Nov 2023.

Selman L, Clement C, Douglas M, Douglas K, Taylor J, Metcalfe C, Lane J, Horwood J. Patient and public involvement in randomised clinical trials: a mixed-methods study of a clinical trials unit to identify good practice, barriers and facilitators. Trials. 2021 https://doi.org/10.1186/s13063-021-05701-y

Coulman K, Nicholson A, Shaw A, Daykin A, Selman L, Macefield R, Shorter G, Cramer H, Sydes M, Gamble C, Pick M, Taylor G, Lane J. Understanding and optimising patient and public involvement in trial oversight: an ethnographic study of eight clinical trials. Trials. 2020. https://doi.org/10.1186/s13063-020-04495-9

Ocloo J, Garfield S, Franklin B, Dawson S. Exploring the theory, barriers and enablers for patient and public involvement across health, social care and patient safety: a systematic review of reviews. Health Res Policy Sys. 2021. https://doi.org/10.1186/s12961-020-00644-3

National Institute for Health and Care Research. Briefing notes for researchers - public involvement in NHS, health and social care research. https://www.nihr.ac.uk/documents/briefing-notes-for-researchers-public-involvement-in-nhs-health-and-social-care-research/27371 Accessed 01 Nov 2023.

Download references

Acknowledgements

With thanks to the PHIRST-LIGHT public advisory group and consortium for their thoughts and contributions to the design of this work.

The research team is supported by a National Institute for Health and Care Research grant (PHIRST-LIGHT Reference NIHR 135190).

Author information

Olivia R. Phillips and Cerian Harries share joint first authorship.

Authors and Affiliations

Nottingham Centre for Public Health and Epidemiology, Lifespan and Population Health, School of Medicine, University of Nottingham, Clinical Sciences Building, City Hospital Campus, Hucknall Road, Nottingham, NG5 1PB, UK

Olivia R. Phillips, Jo Leonardi-Bee, Holly Knight & Joanne R. Morling

National Institute for Health and Care Research (NIHR) PHIRST-LIGHT, Nottingham, UK

Olivia R. Phillips, Cerian Harries, Jo Leonardi-Bee, Holly Knight, Lauren B. Sherar, Veronica Varela-Mato & Joanne R. Morling

School of Sport, Exercise and Health Sciences, Loughborough University, Epinal Way, Loughborough, Leicestershire, LE11 3TU, UK

Cerian Harries, Lauren B. Sherar & Veronica Varela-Mato

Nottingham Centre for Evidence Based Healthcare, School of Medicine, University of Nottingham, Nottingham, UK

Jo Leonardi-Bee

NIHR Nottingham Biomedical Research Centre (BRC), Nottingham University Hospitals NHS Trust, University of Nottingham, Nottingham, NG7 2UH, UK

Joanne R. Morling

You can also search for this author in PubMed   Google Scholar

Contributions

Author contributions: study design: ORP, CH, JRM, JLB, HK, LBS, VVM, literature searching and screening: ORP, CH, JRM, data curation: ORP, CH, analysis: ORP, CH, JRM, manuscript draft: ORP, CH, JRM, Plain English Summary: ORP, manuscript critical review and editing: ORP, CH, JRM, JLB, HK, LBS, VVM.

Corresponding author

Correspondence to Olivia R. Phillips .

Ethics declarations

Ethics approval and consent to participate.

The Ethics Committee of the Faculty of Medicine and Health Sciences, University of Nottingham advised that approval from the ethics committee and consent to participate was not required for systematic review studies.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

40900_2024_580_MOESM1_ESM.docx

Additional file 1: Search strings: Description of data: the search strings and filters used in each of the 5 databases in this review

Additional file 2: Quality appraisal questions: Description of data: CASP quality appraisal questions

40900_2024_580_moesm3_esm.docx.

Additional file 3: Table 1: Description of data: elements of the data extraction table that are not in the main manuscript

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Phillips, O.R., Harries, C., Leonardi-Bee, J. et al. What are the strengths and limitations to utilising creative methods in public and patient involvement in health and social care research? A qualitative systematic review. Res Involv Engagem 10 , 48 (2024). https://doi.org/10.1186/s40900-024-00580-4

Download citation

Received : 28 November 2023

Accepted : 25 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s40900-024-00580-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public and patient involvement
• Creative PPI
• Qualitative systematic review

Research Involvement and Engagement

ISSN: 2056-7529

• General enquiries: [email protected]

Pilot study of a ketogenic diet in bipolar disorder: a process evaluation

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• ORCID record for Benjamin P Rigby
• For correspondence: [email protected]
• ORCID record for Nicole Needham
• ORCID record for Ben Meadowcroft
• ORCID record for John Norrie
• ORCID record for Gerard Thompson
• ORCID record for Tessa Moses
• ORCID record for Karl Burgess
• ORCID record for Michael J Thrippleton
• ORCID record for Harry Campbell
• ORCID record for Daniel J Smith
• ORCID record for Sharon A Simpson
• Info/History
• Preview PDF

Background: Bipolar disorder is a serious mental illness, which requires new strategies for prevention and management. Recent evidence suggests that a ketogenic diet may be an effective intervention. This research aimed to explore the feasibility and acceptability of a ketogenic diet intervention for bipolar disorder, fidelity to its behavioural components and the experiences of the participants and research clinicians involved. Methods: A mixed-methods process evaluation was conducted. Semi-structured telephone interviews were carried out with 15 participants 1-2 months after completing a 6-8 week modified ketogenic diet intervention, and 4 research clinicians from the study team following the completion of data collection. Data were thematically analysed. Fidelity checklists completed by research dietitians were analysed using descriptive count and percentage statistics. Findings are reported post-hoc, following the analysis and publication of the main pilot study findings. Results: Qualitative data indicated that participants had various motives for taking part in the study, including weight loss. It was important to support people's motives while facilitating clear and realistic expectations. Despite the challenges of initiating and maintaining a ketogenic diet, including for some its disruptive effects on daily living, many participants perceived physical and psychological benefits (e.g. significant weight loss, mood stability and an enhanced ability to focus). A range of behavioural (e.g. goal setting), social (e.g. family and dietitians) and technological (e.g. apps for monitoring) support mechanisms were generally considered key facilitating factors. Meanwhile, dietary preferences, concerns about the diet and its impact, the testing burden and capacity of the delivery team were perceived as barriers for some. The importance of wider contextual influences (e.g. the cost of living and sociocultural expectations) were highlighted. Overall, descriptive analyses indicated moderate-to-good fidelity to the behaviour change components of the study. Conclusion: We provide novel insight into the experiences of people living with bipolar disorder initiating and following a ketogenic diet, as well as those of research clinicians who support the intervention. Future trials may benefit from increased clinical research capacity, better-defined entry and exit routes, additional interpersonal support, and greater understanding of how social and societal factors impact participation.

Competing Interest Statement

IHC has a diagnosis of bipolar disorder and follows a ketogenic diet to manage his symptoms. His current fellowship is funded by the Baszucki Research Fund.

Funding Statement

This study was funded by the Baszucki Brain Research Fund. BPR and SAS acknowledge funding from the UKRI Medical Research Council (MC_UU_00022/1) and the Chief Scientist Office of the Scottish Government Health Directorates (SPHSU16). MJT acknowledges funding from the Chief Scientist Office of the Scottish Government Health Directorates through the NHS Lothian Research and Development Office.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

Ethics committee of NHS Lothian Research and Development gave ethical approval for this work.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

Data generated and analysed during the current study are not publicly available as explicit consent was not sought from participants, and privacy may be compromised.

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One
• Addiction Medicine (324)
• Allergy and Immunology (628)
• Anesthesia (165)
• Cardiovascular Medicine (2377)
• Dentistry and Oral Medicine (289)
• Dermatology (207)
• Emergency Medicine (379)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (837)
• Epidemiology (11775)
• Forensic Medicine (10)
• Gastroenterology (702)
• Genetic and Genomic Medicine (3742)
• Geriatric Medicine (350)
• Health Economics (634)
• Health Informatics (2399)
• Health Policy (933)
• Health Systems and Quality Improvement (898)
• Hematology (341)
• HIV/AIDS (782)
• Infectious Diseases (except HIV/AIDS) (13317)
• Intensive Care and Critical Care Medicine (767)
• Medical Education (365)
• Medical Ethics (105)
• Nephrology (398)
• Neurology (3507)
• Nursing (198)
• Nutrition (526)
• Obstetrics and Gynecology (674)
• Occupational and Environmental Health (664)
• Oncology (1824)
• Ophthalmology (538)
• Orthopedics (219)
• Otolaryngology (287)
• Pain Medicine (232)
• Palliative Medicine (66)
• Pathology (446)
• Pediatrics (1034)
• Pharmacology and Therapeutics (426)
• Primary Care Research (420)
• Psychiatry and Clinical Psychology (3178)
• Public and Global Health (6145)
• Radiology and Imaging (1280)
• Rehabilitation Medicine and Physical Therapy (747)
• Respiratory Medicine (828)
• Rheumatology (379)
• Sexual and Reproductive Health (372)
• Sports Medicine (323)
• Surgery (402)
• Toxicology (50)
• Transplantation (172)
• Urology (146)

This paper is in the following e-collection/theme issue:

Published on 17.5.2024 in Vol 8 (2024)

Usability and Utility of a Mobile App to Deliver Health-Related Content to an Older Adult Population: Pilot Noncontrolled Quasi-Experimental Study

Authors of this article:

Original Paper

• Marta Lemos 1 * , MD   ; 
• Ana Rita Henriques 2 * , MSc   ; 
• David Gil Lopes 2 , MSc   ; 
• Nuno Mendonça 2 , PhD   ; 
• André Victorino 3 , MSc   ; 
• Andreia Costa 4, 5, 6, 7 , PhD   ; 
• Miguel Arriaga 4, 7 , PhD   ; 
• Maria João Gregório 2, 8, 9 , PhD   ; 
• Rute de Sousa 2 , MBA   ; 
• Helena Canhão 2 , MD, PhD   ; 
• Ana M Rodrigues 2 , MD, PhD  

1 Unidade de Saúde Pública do ACES Algarve II – Barlavento, Centro de Saúde de Portimão, Portimão, Portugal

2 CHRC, NOVA Medical School, NMS, Universidade NOVA de Lisboa, Lisboa, Portugal

3 Faculdade de Ciências e Tecnologia, Universidade NOVA de Lisboa, Costa da Caparica, Almada, Portugal

4 Direção-Geral de Saúde, Lisboa, Portugal

5 Instituto de Saúde Ambiental, Faculdade de Medicina, Universidade de Lisboa, Lisboa, Portugal

6 Nursing Research, Innovation and Development Centre of Lisbon (CIDNUR), Nursing School of Lisbon (ESEL), Lisboa, Portugal

7 Catolica Research Centre for Psychological, Family and Social Wellbeing, Lisboa, Portugal

8 Faculdade de Ciências da Nutrição e Alimentação, Universidade do Porto, Porto, Portugal

9 Programa Nacional para a Promoção da Alimentação Saudável, Direção-Geral da Saúde, Lisboa, Portugal

*these authors contributed equally

Corresponding Author:

Ana Rita Henriques, MSc

CHRC, NOVA Medical School

Universidade NOVA de Lisboa

Rua do Instituto Bacteriológico, nº5 Edifício Amarelo

Lisboa, 1150-190

Phone: 351 214956435 ext 27016

Email: [email protected]

Background: Digital patient-centered interventions may be important tools for improving and promoting social interaction, health, and well-being among older adults. In this regard, we developed a mobile app called DigiAdherence for an older adult population, which consisted of easy-to-access short videos and messages, to improve health-related knowledge among them and prevent common health conditions, such as falls, polypharmacy, treatment adherence, nutritional problems, and physical inactivity.

Objective: This study aimed to assess the usability and utility of the DigiAdherence app among Portuguese older adults 65 years or older.

Methods: In this pilot noncontrolled quasi-experimental study, older adults who were patients at the primary health care center in Portimão, Portugal, and owned a smartphone or tablet were recruited. Participants were assessed at baseline, given access to the DigiAdherence app for 1 month, and assessed again immediately after 30 days (first assessment) and 60 days after stopping the use of the app (second assessment). App usability and utility (primary outcomes) were analyzed in the first follow-up assessment using a structured questionnaire with 8 items. In the second follow-up assessment, our focus was on knowledge acquired through the app. Secondary outcomes such as treatment adherence and health-related quality of life were also assessed.

Results: The study included 26 older adults. Most participants rated the different functionalities of the app positively and perceived the app as useful, attractive, and user-friendly (median score of 6 on a 7-point Likert scale). In addition, after follow-up, participants reported having a sense of security and greater knowledge in preventing falls (16/24, 67%) and managing therapies and polypharmacy (16/26, 62%).

Conclusions: The DigiAdherence mobile app was useful and highly accepted by older adults, who developed more confidence regarding health-related knowledge.

International Registered Report Identifier (IRRID): RR2-10.2196/29675

Introduction

People worldwide are living longer. By 2030, 1 in 6 people in the world will be 60 years or older [ 1 ]. Portugal has one of the oldest populations in Europe due to an increased life expectancy and reduced fertility rate. In 2021, older adults comprised 22.4% of the population, a rapid increase of 3.7% from 18.7% in 2011 [ 2 ].

The World Health Organization defines health promotion as the process of enabling people to take increasing control over their health and its determinants, thereby improving their health [ 1 ], while health literacy encompasses a person’s ability to access, understand, appraise, and apply health information to make informed decisions regarding their health [ 3 ]. In light of this, it is crucial to highlight the findings from the Health Literacy Population Survey Project 2019-2021, which suggests a strategic focus on interventions that improve health literacy for health promotion, disease prevention, health care, and health system navigation. The results of this study also suggest a social gradient for health literacy; women, older age groups of individuals, people with lower economic capacity, people with lower levels of schooling, and the unemployed are at risk of low health literacy in Portugal. Therefore, particular attention should be given to this population regarding actions to promote health literacy [ 4 ].

As the world transitions into the information age, incorporating digital technologies into health promotion is becoming commonplace. In the past two decades, the use of digital media and digital products for health promotion has grown exponentially, particularly mobile app–based health promotion programs [ 5 ]. As the current generation of older adults is becoming more accustomed to using digital products [ 6 ], we must consider exploring digital patient-centered interventions as tools to improve and promote social interaction, encourage healthy lifestyles, and enhance well-being and quality of life (QoL) among this age group [ 7 - 9 ]. Evidence suggests that older adults with a higher education level and self-rated health status are more willing to use health care technologies [ 10 - 12 ]. Several research studies have been performed on mobile health (mHealth) apps, and the results suggest that, when well designed, these apps can improve health literacy skills, self-care, and treatment adherence; empower patients; and reduce the cost of health care [ 8 , 13 - 17 ]. We previously developed an informative, motivational, interactive, home-based TV application intervention program for an older adult population (Saúde.Come Senior) to promote healthy lifestyles [ 18 ]. This program has been fully implemented and tested, and shown to be well accepted and successful in reducing food insecurity and improving participants’ physical function [ 19 ].

As older adults may be potential users of mHealth apps, it is crucial to study, design, and adapt technologies to make them intuitively understandable for this age group [ 20 - 23 ]. This is challenging because the range of technical abilities of 65- to 95-year-olds is vast and very heterogeneous [ 20 ]. Because this demographic represents a diverse range of technical abilities and faces unique challenges related to aging that can impact their interaction with digital technology, addressing the usability and utility of mobile apps among older adults, along with users’ overall satisfaction and willingness to use them, is of paramount importance [ 8 , 24 ]. These challenges include cognitive changes, sensory limitations, and the need for technology that accommodates reduced dexterity and potential visual, hearing, and memory impairments [ 25 - 27 ]. Additionally, factors such as limited digital literacy and restricted access to technology can further complicate the use of mobile apps by older adults [ 28 , 29 ].

We recognize that the successful integration of technology into the lives of older adults requires careful consideration of usability, accessibility, and content relevance. In response to these multifaceted challenges, we adopted an approach that aimed to simplify the user experience by designing an app featuring simplified interfaces, cleared navigation, preloaded short video contents, and offline functionality to cater to the diverse needs and abilities of older users. In collaboration with the Portuguese Directorate-General for Health (DGS), our research group developed a free educational mobile app, called DigiAdherence, to increase health knowledge and prevent the most frequent health conditions among older adults, such as nutritional problems, physical inactivity, falls, polypharmacy, treatment adherence, and cognitive decline [ 30 ]. Our emphasis was on improving usability, ensuring the app’s simplicity, intuitiveness of use, and pleasant aesthetics, while also enhancing its utility for knowledge acquisition and application. By offering a straightforward approach to accessing health-related content, we aimed at minimizing barriers to use and promoting the engagement of older adults with digital health content. Importantly, this pilot intervention was conducted as a small-scale test to gather insights and feedback from users. Our goal was not to compete with the extensive pool of existing apps but to offer a preliminary or initial solution while aiming to refine and improve our approach based on the responses of this specific user group.

The primary aim of this study was to assess the usability and utility of the DigiAdherence app, specifically among an older adult population (65 years or older), during 30 days of use of its health-informative video-based content. The secondary aim was to assess behavioral change or knowledge improvement regarding treatment adherence, fall prevention, and health-related QoL 60 days after the intervention.

Study Design

This pilot study used a noncontrolled quasi-experimental design with outcomes measured at 2 assessment moments (first and second assessments). The usability and utility of the DigiAdherence app were evaluated 30 days after participants started using it (first assessment) and 60 days after they stopped using it (second assessment). All eligible participants were patients at the primary health care center of Portimão, Barlavento Primary Healthcare Centres (ACES Barlavento), Algarve, Portugal.

After being recruited and consenting to participation, participants attended 3 evaluation moments: 1 baseline assessment and 2 follow-up assessments (first and second assessments). The same investigator carried out all assessments and data collection.

The baseline assessment was performed before participants started using the app, between May and June 2021. Each visit had a duration of 30 to 40 minutes, during which participants were assisted in answering a questionnaire, and the mobile app was installed. If the installation was successful, participants were given a demonstration of the app’s functioning and encouraged to access its content themselves by selecting videos, adjusting the sound volume, changing the view to full-screen mode, and going back to the video selection process so any questions could be clarified. Access to the app was granted for 30 days, during which the user could easily access it offline and use it multiple times and at any time. The follow-up assessments were performed through phone call interviews 30 days after participants started using the app, between June and August 2021 (first assessment), and 60 days after participants stopped using it, between August and October 2021 (second assessment). At the end of the first assessment, participants were given instructions to uninstall the app or not to use it until the next assessment. Each follow-up assessment had a duration of 10 to 15 minutes, during which participants were assisted in answering a questionnaire, and the outcomes were assessed.

Study Population

We used a convenience sample of 26 older adults 65 years or older.

Inclusion and Exclusion Criteria

Individuals were included if they (1) were 65 years or older, (2) were living at home, (3) owned an Android smartphone or a tablet, (4) were a Portuguese speaker or understood the Portuguese language, (5) were a patient in the primary health care center of Portimão, and (6) were willing to participate and gave written informed consent.

Individuals were excluded if they (1) did not know how to use a smartphone or tablet, (2) owned a smartphone or tablet with an operating system other than Android, (3) were unable to complete app installation, (4) did not use the app during follow-up, and (5) had any cognitive impairment or were unable to answer the questionnaire, assessed by the physician.

Intervention (DigiAdherence)

Our research group developed a simple, intuitive, video-based mobile app containing 6 short health-informative videos ( Figure 1 ) as described elsewhere [ 24 ]. Going through DigiAdherence’s menu, the user could choose any of its contents, in no specific order, after 2 simple taps. The “Recipe” content showed a chef teaching users how to make a healthy cauliflower soup; the “Nutrition” content showed a nutritionist talking about sugar replacement options; the “Physical activity” content showed a personal trainer exemplifying a series of physical activity exercises that users could do while sitting in a chair; the “Cognition” content showed a psychologist talking about the importance of cognitive exercises and giving some examples of the type of exercises that users could do; the “Falls” content showed a rheumatologist listing a series of techniques that users could adopt to prevent falls on the street or at home; and the “Medication” content showed a rheumatologist talking about the risk of polypharmacy and giving tips on what users should do to take their medication as prescribed. Every professional made sure to speak clearly without using medical or technical jargon, keeping their messages simple and easy to understand. Videos were stored locally within the app, enabling offline functionality, and each had a duration of 2 to 8 minutes. The executable file (APK) was created using Android Studio’s software. The app was installed on the participant’s own smartphone or tablet by one of the investigators during a scheduled in-person visit. After installation, participants had free access to the app for 30 days. More details on the DigiAdherence app can be found in the study protocol published elsewhere [ 30 ].

Outcomes—Definition and Assessment

The primary outcomes were the DigiAdherence app’s usability and utility. The secondary outcomes were behavioral change or knowledge improvement regarding treatment adherence and health perception 60 days after the intervention.

Usability and Utility (Primary Outcomes)

The usability and utility of the app were assessed during the first assessment, whereas during the second assessment, only the utility of the app based on the knowledge acquired through it was measured.

First, participants were asked whether they use the app, with the question: “Have you used the app on your smartphone or tablet?” The first assessment was based on 8 items (5 instrumental items and 3 noninstrumental items) using a 7-point Likert scale. The 3 noninstrumental items assessed were the app’s visual aesthetics (“How do you classify the app in terms of pleasantness of its visual aesthetics?”), motivation to use the app (“How do you classify your motivation to use the app?”), and overall satisfaction with the app (“How do you classify your overall satisfaction with the app?”). The 5 instrumental items assessed were utility in taking medication correctly (“How do you classify the app in terms of usefulness to take your medication correctly?”), utility in QoL improvement (“How do you classify the app in terms of usefulness to improve your quality of life?”), utility in increasing health knowledge (“How do you classify the app in terms of usefulness to increase your health knowledge?”), utility in preventing falls (“How do you classify the app in terms of usefulness to prevent falls”), and the app’s easiness of use (“How do you classify the app in terms of easiness of use?”). Sixty days after the intervention (second assessment), participants were asked if they used knowledge acquired through the app to prevent falls (“Did you use the knowledge acquired through the app in terms of techniques for fall prevention?”—yes/no) and take their medication correctly (“Did you use the knowledge acquired through the app in terms of techniques for taking your medication correctly?”—yes/no). As an offline app, we did not quantify the app’s engagement metrics, such as frequency of use, exit rate, or use rate.

Treatment Adherence and Health-Related QoL (Secondary Outcomes)

Data measurement scales were used in all assessments to assess participants’ QoL (using the Portuguese version of the EQ-5D-3L questionnaire [ 31 , 32 ]) and treatment adherence (using the Portuguese version of the Medication Adherence Rating Scale [ 33 ]).

The EQ-5D-3L questionnaire includes 5 questions that describe health status in 5 domains: mobility, self-care, usual activities, pain and discomfort, and anxiety and depression. Each dimension had 3 levels: without problems, some problems, and extreme problems, and participants were asked to mark the option that would best describe their situation on that day. A preference-based scoring function was used to convert the descriptive system to a summary index score ranging from 1 (full health) to 0 (the worst possible state of health). Scores could also be negative down to –1, corresponding to death or states worse than death. Participants were also asked to assess their general health status on a visual analog scale from 0 (worst imaginable health status) to 100 (best imaginable health status), often called the EQ-VAS thermometer [ 31 , 32 ].

The Medication Adherence Rating Scale is a 10-item yes/no self-report instrument that was developed from 2 existing scales, the 30-item Drug Attitudes Inventory [ 34 ] and the 4-item Medication Adherence Questionnaire [ 35 ], to develop a more reliable and valid tool for assessing medication adherence behavior in psychosis. Total scores range from 0 (low likelihood of medication adherence) to 10 (high likelihood of medication adherence) [ 33 ].

Other Assessments

Information on sociodemographic data, such as sex, age, and years of education, was collected at the baseline assessment. Health literacy on fall prevention was also assessed (“Does the participant consider himself or herself familiar with fall prevention techniques?”—yes/no). When the answer was “yes,” the participant was asked to describe at least 1 strategy. The following parameters (except the app’s usability and utility, which were assessed during the follow-up assessments) were measured in the 3 assessments. Self-reported height and weight were measured, and BMI was calculated and categorized according to the World Health Organization classification (underweight: <18.5 kg/m 2 , normal weight: 18.5–24.9 kg/m 2 , overweight: 25–29.9 kg/m 2 , or obese: ≥30 kg/m 2 ). Questions concerning clinical history included self-reported comorbidities (yes/no and comorbidity count), concomitant medication (yes/no and number of medications), and frequency of falls in the previous month (yes/no and number of falls).

Statistical Analysis

A descriptive analysis of sociodemographic and clinical characteristics was conducted for the study population using frequencies and proportions for categorical variables and mean and SD for continuous variables. The app’s usability and utility are reported as median and IQR. Wilcoxon signed rank tests were used to compare treatment adherence perception and health perception between the baseline and first assessments and between the baseline and second assessments. Hypothesis testing (H0: median of the difference between the baseline and first assessments is 0; H0: median of the differences between the baseline and second assessments is 0) considered only participants with complete EQ-5D-3L and the Medication Adherence Rating Scale scores for both intervals. The significance level was set at .05. All analyses were performed using Stata/IC (version 16.1; StataCorp).

Ethical Considerations

This study was submitted and authorized by the Executive Board of ACeS Algarve II – Barlavento (No. 16/2020) and by the Ethics Committee of NOVA Medical School (99/2019/CEFCM, June 2020), NOVA University of Lisbon, Portugal. This protocol was also submitted to and approved by the Ethics Committee for Health (16/2020, September 2020) and the Executive Board of the Regional Health Administration of the Algarve (December 2020), IP (Instituto Público) (No. 16/2020). All procedures followed the principles of Good Clinical Practice and the Declaration of Helsinki (Fortaleza revision, 2013). Participants were included in the study only after obtaining their written informed consent. Although this pilot study was based on the use of a mobile phone app, this was an educational app that only delivered short videos with health-informative content. It should be noted that this technology did not collect or transmit any personal data related to participants’ health through communication networks. Thus, according to the General Regulation on Data Protection Regulation (European Union) 2016/679 of the European Parliament and the Council on April 27, 2016, and the Regulation No. 1/2018, this study was exempt from notification to the Portuguese Data Protection Authority ( Comissão Nacional de Proteção de Dados ).

Baseline Characteristics

Among a total of 47 older adults (23 male and 24 female) who initially showed interest in participating in this study and met the inclusion criteria, 37 agreed to attend an in-person visit at the primary health care center of Portimão. Of these older adults, 6 (16%) were unable to install the app because of insufficient storage space or the operating system not being compatible, and 5 (13%) installed the app but did not use it ( Figure 2 ). Table 1 presents the sociodemographic and clinical characteristics of participants at baseline who successfully installed the app and used it for 30 days (26/37, 70%). The number of men and women were similar, with a mean (SD) age of 71.7 (5.4) years. The most common level of education was the 12th grade (9/26, 35%). Most participants were overweight (11/26, 42%). All participants had at least 1 comorbidity, with a mean (SD) of 3.6 (1.6) comorbidities. Of the 26 participants, 25 (96%) were on chronic medication. At baseline, 4 (15%) participants reported having fall prevention literacy, and 1 (4%) participant had experienced a fall in the previous month.

a Sample size is not constant due to missing values for some variables: medication adherence (n=25).

b MARS: Medication Adherence Rating Scale.

Participants had a mean (SD) QoL score of 0.70 (0.19) and a mean (SD) treatment adherence score of 8.1 (1.2). Considering the mean scores for QoL and therapeutic adherence, it is evident that at baseline, the QoL is close to the national levels for the Portuguese population in the considered age group (0.790). Moreover, for therapeutic adherence, a value above 8 on a scale of 10 also indicates that the population already had good adherence at baseline.

The reasons reported by participants for not using the app were shortage of time and professional issues (n=2), depression (n=1), hospitalization (n=1), and no interest in the app (n=1). All participants had the app installed on their own smartphone (no tablets were used), except 4 who had it installed on their partner’s smartphone to avoid exclusion due to incomplete installation.

Usability and Utility

All items had an average score above 5 and a median score of 6 on a 7-point Likert scale, with the exception of the app’s easiness of use, which had a median score of 7. Overall, the items with higher values were overall satisfaction with the app, visual aesthetics, and the app’s easiness of use ( Figure 3 ).

Treatment Adherence and Health-Related QoL

Regarding treatment adherence, 16 of the 26 (62%) participants applied the knowledge acquired through the app. For treatment adherence and health-related QoL, no significant differences were found between the baseline assessment and the first and second assessments ( Table 2 ).

a Sample size is not equal between scales due to missing values for some variables: health-related QoL—second assessment (n=24); treatment adherence—baseline (n=25), first assessment (n=25), second assessment (n=23). Exact P values are displayed.

b Wilcoxon signed rank test: H0: median of the difference between the baseline and first assessments is 0.

c Wilcoxon signed rank test: H0: median of the differences between the baseline and second assessments is 0. Exact P values are displayed.

d QoL: quality of life.

e VAS: visual analog scale.

f MARS: Medication Adherence Rating Scale.

Principal Findings

We developed the DigiAdherence mobile app with the primary aim of assessing its usability and utility among a study population of 26 older adults who used its health-informative video-based content for 30 days.

Most participants rated the different functionalities of the app (eg, utility in taking medication correctly, utility in QoL improvement, utility in increasing health knowledge, and utility in preventing falls) positively, with an average score of above 5 and a median score of 6 on a 7-point Likert scale. Participants reported overall satisfaction with the app and perceived it as useful, attractive, and user-friendly.

In addition to learning information, participants felt that the app’s use offered a sense of security and knowledge in preventing falls and managing therapy and polypharmacy. At baseline, only 4 of 37 (11%) participants reported having fall prevention literacy, and after the intervention, one-third reported using the knowledge acquired through the app regarding fall prevention techniques. Although all participants had at least 1 comorbidity and most were on chronic medication (25/26, 96%), no significant differences were found in health perception or treatment adherence after using the app, probably because these measures were already high at baseline. Furthermore, half of the study population had a high level of education (12th grade or bachelor’s degree). Nevertheless, after the intervention, 62% (16/26) of participants reported applying the knowledge acquired through the app regarding treatment adherence. Regarding QoL, the short follow-up period made its assessment not relevant.

Comparison With Prior Work

In a review of mobile app–based health promotion programs by Lee et al [ 36 ], mobile app programs for the general population have mainly been used for weight control and physical activity. The most common goals of mobile app programs were to provide both health information and feedback (6/12, 50% of studies), followed by only providing feedback (3/12, 25% of studies) or information (2/12, 17% of studies) [ 36 ].

In another review of health promotion and disease prevention for older adults by Chiu et al [ 37 ], the most common intervention method was health promotion, with 322 of the 486 studies (66%) classified as evaluating health promotion interventions. Meanwhile, 264 (54%) studies were classified as evaluating screening, 114 (23%) as evaluating primary prevention, and 72 (15%) as evaluating social support [ 37 ]. The most common target was “disease-oriented” (214/486, 44% of studies), followed by “physical activity” (120/486, 24.7%), “general health” (118/486, 24.2%), “quality of life” (106/486, 21.8%), “cognitive function” (100/486, 20.5%), “frailty” (85/486, 17.5%), “nutrition” (72/486, 14.8%), “mental health” (57/486, 11.7%), “psychosocial functioning” (51/486, 10.5%), “independence” (26/486, 5.4%), “sleep quality” (13/486, 2.7%), and, finally, “addiction”(4/486, 0.8%) and “disability” (4/486, 0.8%). Concerning eHealth technology apps from 2015 to 2019, only 12 of the 486 studies evaluated the use of eHealth technology. These tools included virtual reality (n=3), smart homes and home health monitoring technologies (n=1), socially assistive robots (n=5), and electronic assistive technology (n=3).

Finally, in a scoping review, Wilson et al [ 38 ] evaluated barriers to and facilitators of the use of eHealth by older adults as reported in 14 papers. According to Wilson et al [ 38 ], the most prevalent barriers to eHealth engagement were a lack of self-efficacy, knowledge, support, functionality, and information provision about the benefits of eHealth for older adults. Key facilitators were active engagement of the target end users in the design and delivery of eHealth programs, support for overcoming privacy concerns and enhancing self-efficacy in the use of technology, and integration of eHealth programs across health services to accommodate commonplace multimorbidity.

Overall, our findings suggest that designing simple apps specifically for older adults can motivate them to use them and perceive their usage as valuable. Providing more user-friendly systems and enhancing skills in using smartphones among this age group may contribute to older adults’ willingness to use mobile app–based interventions for self-health management. These results appear to be similar to those reported by Jo et al [ 12 ]. There are some apps for older adults that promote social interaction, health, and well-being [ 7 - 9 , 37 ]. We previously developed one such app (Saúde.Come Senior), which was shown to be feasible and acceptable by users [ 18 , 19 ]. Although the range of information and communication technology (ICT) products is expanding, more research on this topic is still needed. Future research should engage older adults in developing technology based on their needs. Furthermore, factors that influence older adults’ willingness to use ICT, such as mobile apps, should be assessed to ensure its successful and effective implementation [ 39 ].

Strengths and Limitations

Several strengths of this study deserve to be mentioned. First, all assessments and data collection were conducted by the same investigator, which may have minimized interviewer bias. Second, developing a simple, intuitive app with offline access to its content after 2 simple taps may have increased participants’ technology adherence. This approach aimed to minimize the need for complex interactions and reduce the potential barriers associated with navigation. Moreover, DigiAdherence is an offline app with no interaction or dynamic elements, which was probably why no difficulties in participants engaging with the app were observed. While there may be other apps and studies targeting older adults with different features and functionalities, the simplicity and offline functionality of our app might have served as an advantage. It made the content more accessible, considering the diverse needs and limitations of older users. The local storage of videos was chosen to facilitate easy installation and usage, especially considering factors like limited data plans or technical proficiency. Third, conducting a feasibility assessment for the DigiAdherence app was deemed redundant. This decision was based on the fact that the DigiAdherence content, originating from the Saúde.Come Senior trial [ 17 ], has undergone extensive implementation, testing, and validation, establishing its feasibility and acceptability among users. Besides, our primary focus centered on evaluating the usability and utility of the app, emphasizing its user-friendliness and effectiveness in meeting users’ health-related needs. By addressing these factors, we aim to contribute to the development of effective digital tools that meet the needs and expectations of older adults. Moreover, we ensured that the concepts of usability and utility were aligned with accepted definitions and frameworks within the field of mHealth app usability assessment [ 40 ]. We drew on established definitions and guidelines to shape our evaluation criteria [ 40 ]. This reference provided us with a solid foundation for assessing the usability and utility of the DigiAdherence app and ensured that our evaluation was consistent with accepted standards in the field. Lastly, the first and second assessments were performed through a phone call interview, which may have limited the number of participants lost during follow-up.

Despite these strengths, this pilot study has limitations that need to be addressed in future trials. First, the use of a convenience sample of older adults with higher digital literacy and access to ICT may have led to selection bias. In addition, being an offline video–based app increased the need for free storage capacity. The app being exclusive to Android was also a limitation. In some instances, it was necessary to erase some of the participants’ device content. Nevertheless, 6 participants were excluded because of insufficient storage space (or the operating system not being compatible), and 4 had the app installed on their partner’s device to avoid exclusion due to incomplete installation. These participants were not excluded because we were assured that this procedure would not interfere with their access to the app. Regarding the sample size, it is important to clarify that our study’s primary objective was to assess usability and utility and gather preliminary feedback. This approach is consistent with the goals of other studies, where smaller sample sizes were valuable in providing early insights into user experiences [ 41 , 42 ]. Our intention was not to offer a definitive assessment of the app’s effectiveness for all potential users. Instead, it was a critical first step in our iterative process of refining and enhancing the app’s design, with a broader adoption and improved service in mind. Second, the first assessment occurred during summertime, between June and August, which made it difficult to establish contact with participants, probably due to a lack of motivation or availability to participate during their vacations. As a result, all phone calls were made more than 30 days after participants started using the app (mean 45.4, SD 10.2, range 34-69 days). The second assessment occurred approximately 60 days after participants stopped using the app (mean 63.5, SD 8.4, range 41-75 days). Between these 2 assessments, some participants chose not to uninstall the app, with the condition that they would not use it until the study concluded. Third, when asked to score the app’s usability and utility, some users rated it considering personal gain, whereas others considered a perception of collective gain, such as “for me it was not that useful, since I was already familiarized with most of the app’s contents, but for other people I believe it can be very useful.” A few users expressed some discontent with the fact that the video content was always the same, which may have led to a loss of motivation for its use. Lastly, regarding the use of a 7-point Likert scale, users had an average score of above 5 and a median score of 6. This may be due to overall satisfaction with the DigiAdherence app, but considering the age group in question, we should question the possibility that the use of a 7-point Likert scale may have been too complex (numerical/noncategorical, with a wide range of choices) combined with the fact that the follow-up assessments were conducted through a phone call (which made it impossible for participants to visualize the scale).

Conclusions

In conclusion, the findings of this pilot study show the easiness of use and aesthetic appeal of the DigiAdherence app, thus resulting in higher overall acceptance and satisfaction among an older adult population. Lessons on the usability and utility of this app can be used for other apps targeted at older adults. Moreover, this intervention program seems to have increased older adults’ security and confidence regarding health-related knowledge, particularly in preventing falls and managing therapy and polypharmacy.

Acknowledgments

The DigiAdherence project was generously funded by DGS ( Direção-Geral da Saúde ) through the CENIE project ( Fundo Europeu de Desenvolvimento Regional , [FEDER]), Programa Interreg V-A Espanha-Portugal (POCTEP) 2014/2020, and Fundação Ciência e Tecnologia, IP(Instituto Público), and national support through CHRC (UIDP/04923/2020). The Portuguese Health Authority (DGS) has ownership of the app and has no profitable or commercial interest in the product. We also thank ACES Barlavento and ARS Algarve, Portugal for their institutional support, Dr Ludmila Porojan for coordinating the study at ARS Algarve, and Catarina Nunes da Silva for technical support. Additionally, we thank Dr Susana Lourenço and Dr Paulo Pancrácio for their collaboration during participant recruitment as well as Chef Justa Nobre and Renato Lopes for their contribution to the app’s videos.

Data Availability

The data sets used or analyzed during this current study are available from the corresponding author upon reasonable request.

Authors' Contributions

This study was designed by AMR, HC, RDS, MA, and AC. The DigiAdherence app’s content was developed by AMR, HC, MJG, and RDS. AV was responsible for development of the DigiAdherence app architecture. ML and ARH were the major contributors to the writing of the manuscript. ML conducted all participant assessments. ARH, DGL, and NM conducted formal analysis. AC, AMR, ARH, AV, DGL, HC, ML, NM, and RDS provided essential scientific perspectives and reviewed the manuscript for publication. All authors read and approved the final manuscript.

Conflicts of Interest

AMR recived unrestricted grants from Amgen, Astrazeneca, and Novartis.

• Ageing and health. World Health Organization. 2022. URL: https://www.who.int/news-room/fact-sheets/detail/ageing-and-health [accessed 2022-03-15]
• Population structure and ageing. eurostat, Statistics Explained. URL: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Population_structure_and_ageing [accessed 2022-01-12]
• mHealth: new horizons for health through mobile technologies: second global survey on eHealth. World Health Organization. 2011. URL: https://apps.who.int/iris/bitstream/handle/10665/44607/9789241564250_eng.pdf?sequence=1&isAllowed=y [accessed 2024-04-24]
• Arriaga M, Francisco R, Nogueira P, Oliveira J, Silva C, Câmara G, et al. Health literacy in Portugal: results of the health literacy population survey project 2019-2021. Int J Environ Res Public Health. 2022;19(7):4225. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Koh A, Swanepoel DW, Ling A, Ho BL, Tan SY, Lim J. Digital health promotion: promise and peril. Health Promot Int. 2021;36(Supplement_1):i70-i80. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Instituto Nacional de Estatística. URL: https://www.ine.pt/xportal/xmain?xpgid=ine_main&xpid=INE [accessed 2022-03-15]
• Mira JJ, Navarro I, Botella F, Borrás F, Nuño-Solinís R, Orozco D, et al. A Spanish pillbox app for elderly patients taking multiple medications: randomized controlled trial. J Med Internet Res. 2014;16(4):e99. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): development and validation study. JMIR Mhealth Uhealth. 2019;7(4):e11500. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Changizi M, Kaveh MH. Effectiveness of the mHealth technology in improvement of healthy behaviors in an elderly population-a systematic review. Mhealth. 2017;3:51. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oh YS, Choi EY, Kim YS. Predictors of smartphone uses for health information seeking in the Korean elderly. Soc Work Public Health. 2018;33(1):43-54. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rasche P, Wille M, Bröhl C, Theis S, Schäfer K, Knobe M, et al. Prevalence of health app use among older adults in Germany: national survey. JMIR Mhealth Uhealth. 2018;6(1):e26. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jo HS, Hwang YS, Dronina Y. Mediating effects of smartphone utilization between attitude and willingness to use home-based healthcare ICT among older adults. Healthc Inform Res. 2021;27(2):137-145. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fairman AD, Dicianno BE, Datt N, Garver A, Parmanto B, McCue M. Outcomes of clinicians, caregivers, family members and adults with spina bifida regarding receptivity to use of the iMHere mHealth solution to promote wellness. Int J Telerehabil. 2013;5(1):3-16. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Parmanto B, Pramana G, Yu DX, Fairman AD, Dicianno BE, McCue MP. iMHere: A novel mHealth system for supporting self-care in management of complex and chronic conditions. JMIR Mhealth Uhealth. 2013;1(2):e10. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Mobile phone-based telemonitoring for heart failure management: a randomized controlled trial. J Med Internet Res. 2012;14(1):e31. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Seto E, Leonard KJ, Cafazzo JA, Barnsley J, Masino C, Ross HJ. Perceptions and experiences of heart failure patients and clinicians on the use of mobile phone-based telemonitoring. J Med Internet Res. 2012;14(1):e25. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Henshall C, Davey Z, Jacelon C, Martin C. A usability study to test the effectiveness, efficiency and simplicity of a newly developed internet-based exercise-focused health app for lung cancer survivors (iEXHALE): protocol paper. Health Informatics J. 2020;26(2):1431-1442. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rodrigues AM, Gregório MJ, Gein P, Eusébio M, Santos MJ, de Sousa RD, et al. Home-based intervention program to reduce food insecurity in elderly populations using a TV app: study protocol of the randomized controlled trial saúde.Come senior. JMIR Res Protoc. 2017;6(3):e40. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gomes LA, Gregório MJ, Iakovleva TA, de Sousa RD, Bessant J, Oliveira P, et al. A home-based eHealth intervention for an older adult population with food insecurity: feasibility and acceptability study. J Med Internet Res. 2021;23(8):e26871. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ensuring Artificial Intelligence (AI) technologies for health benefit older people. World Health Organization. URL: https:/​/www.​who.int/​news/​item/​09-02-2022-ensuring-artificial-intelligence-(ai)-technologies-for-health-benefit-older-people [accessed 2022-12-27]
• Nebeker C, Zlatar ZZ. Learning from older adults to promote independent physical activity using mobile health (mHealth). Front Public Health. 2021;9:703910. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cajita MI, Hodgson NA, Lam KW, Yoo S, Han HR. Facilitators of and barriers to mHealth adoption in older adults with heart failure. Comput Inform Nurs. 2018;36(8):376-382. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Terp R, Kayser L, Lindhardt T. Older patients' competence, preferences, and attitudes toward digital technology use: explorative study. JMIR Hum Factors. 2021;8(2):e27005. [ FREE Full text ] [ CrossRef ] [ Medline ]
• To QG, Green C, Vandelanotte C. Feasibility, usability, and effectiveness of a machine learning-based physical activity chatbot: quasi-experimental study. JMIR Mhealth Uhealth. 2021;9(11):e28577. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Woods DL, Satz. The neuropsychology of aging. In: Birren JE, Schaie KW, editors. Handbook of the Psychology of Aging 3rd Edition. San Diego. Academic Press; 1991:236-255.
• Melenhorst AS, Rogers WA, Caylor EC. The use of communication technologies by older adults: exploring the benefits from the user's perspective. J Hum Factors Ergon Soc. 2016;45(3):221-225. [ FREE Full text ] [ CrossRef ]
• Peek STM, Luijkx KG, Rijnaard MD, Nieboer ME, van der Voort CS, Aarts S, et al. Older adults' reasons for using technology while aging in place. Gerontology. 2016;62(2):226-237. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Charness N, Boot WR. Aging and information technology use. Curr Dir Psychol Sci. 2009;18(5):253-258. [ FREE Full text ] [ CrossRef ]
• Xie B. Effects of an eHealth literacy intervention for older adults. J Med Internet Res. 2011;13(4):e90. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nunes-Da-Silva C, Victorino A, Lemos M, Porojan L, Costa A, Arriaga M, et al. A video-based mobile app as a health literacy tool for older adults living at home: protocol for a utility study. JMIR Res Protoc. 2022;11(12):e29675. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ferreira LN, Ferreira PL, Pereira LN, Oppe M. The valuation of the EQ-5D in Portugal. Qual Life Res. 2014;23(2):413-423. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ferreira LN, Ferreira PL, Pereira LN, Oppe M. EQ-5D Portuguese population norms. Qual Life Res. 2014;23(2):425-430. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Vanelli I, Chendo I, Gois C, Santos J, Levy P. Adaptação e validação da versão portuguesa da escala de adesão à terapêutica [Medication adherence rating scale]. Acta Med Port. 2011;24(1):17-20. [ FREE Full text ] [ Medline ]
• Hogan TP, Awad AG, Eastwood R. A self-report scale predictive of drug compliance in schizophrenics: reliability and discriminative validity. Psychol Med. 1983;13(1):177-183. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morisky DE, Green LW, Levine DM. Concurrent and predictive validity of a self-reported measure of medication adherence. Med Care. 1986;24(1):67-74. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lee M, Lee H, Kim Y, Kim J, Cho M, Jang J, et al. Mobile app-based health promotion programs: a systematic review of the literature. Int J Environ Res Public Health. 2018;15(12):2838. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chiu CJ, Hu JC, Lo YH, Chang EY. Health promotion and disease prevention interventions for the elderly: a scoping review from 2015-2019. Int J Environ Res Public Health. 2020;17(15):5335. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wilson J, Heinsch M, Betts D, Booth D, Kay-Lambkin F. Barriers and facilitators to the use of e-health by older adults: a scoping review. BMC Public Health. 2021;21(1):1556. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nordin S, Sturge J, Ayoub M, Jones A, McKee K, Dahlberg L, et al. The role of Information and Communication Technology (ICT) for older adults' decision-making related to health, and health and social care services in daily life-a scoping review. Int J Environ Res Public Health. 2021;19(1):151. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Venkatesh V, Bala H. Technology acceptance model 3 and a research agenda on interventions. Decis Sci. 2008;39(2):273-315. [ FREE Full text ] [ CrossRef ]
• Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst. 2015;39(2):1. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Daniels J, Fels S, Kushniruk A, Lim J, Ansermino JM. A framework for evaluating usability of clinical monitoring technology. J Clin Monit Comput. 2007;21(5):323-330. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 01.02.23; peer-reviewed by L Zhou, T Patel; comments to author 16.05.23; revised version received 31.10.23; accepted 04.12.23; published 17.05.24.

©Marta Lemos, Ana Rita Henriques, David Gil Lopes, Nuno Mendonça, André Victorino, Andreia Costa, Miguel Arriaga, Maria João Gregório, Rute de Sousa, Helena Canhão, Ana M Rodrigues. Originally published in JMIR Formative Research (https://formative.jmir.org), 17.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

• Open access
• Published: 14 May 2024

Developing a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in Medical Assistance in Dying (MAiD): a mixed method modified e-Delphi study

• Jocelyn Schroeder 1 ,
• Barbara Pesut 1 , 2 ,
• Lise Olsen 2 ,
• Nelly D. Oelke 2 &
• Helen Sharp 2  

BMC Nursing volume  23 , Article number:  326 ( 2024 ) Cite this article

31 Accesses

Metrics details

Medical Assistance in Dying (MAiD) was legalized in Canada in 2016. Canada’s legislation is the first to permit Nurse Practitioners (NP) to serve as independent MAiD assessors and providers. Registered Nurses’ (RN) also have important roles in MAiD that include MAiD care coordination; client and family teaching and support, MAiD procedural quality; healthcare provider and public education; and bereavement care for family. Nurses have a right under the law to conscientious objection to participating in MAiD. Therefore, it is essential to prepare nurses in their entry-level education for the practice implications and moral complexities inherent in this practice. Knowing what nursing students think about MAiD is a critical first step. Therefore, the purpose of this study was to develop a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in MAiD in the Canadian context.

The design was a mixed-method, modified e-Delphi method that entailed item generation from the literature, item refinement through a 2 round survey of an expert faculty panel, and item validation through a cognitive focus group interview with nursing students. The settings were a University located in an urban area and a College located in a rural area in Western Canada.

During phase 1, a 56-item survey was developed from existing literature that included demographic items and items designed to measure experience with death and dying (including MAiD), education and preparation, attitudes and beliefs, influences on those beliefs, and anticipated future involvement. During phase 2, an expert faculty panel reviewed, modified, and prioritized the items yielding 51 items. During phase 3, a sample of nursing students further evaluated and modified the language in the survey to aid readability and comprehension. The final survey consists of 45 items including 4 case studies.

Systematic evaluation of knowledge-to-date coupled with stakeholder perspectives supports robust survey design. This study yielded a survey to assess nursing students’ attitudes toward MAiD in a Canadian context.

The survey is appropriate for use in education and research to measure knowledge and attitudes about MAiD among nurse trainees and can be a helpful step in preparing nursing students for entry-level practice.

Peer Review reports

Medical Assistance in Dying (MAiD) is permitted under an amendment to Canada’s Criminal Code which was passed in 2016 [ 1 ]. MAiD is defined in the legislation as both self-administered and clinician-administered medication for the purpose of causing death. In the 2016 Bill C-14 legislation one of the eligibility criteria was that an applicant for MAiD must have a reasonably foreseeable natural death although this term was not defined. It was left to the clinical judgement of MAiD assessors and providers to determine the time frame that constitutes reasonably foreseeable [ 2 ]. However, in 2021 under Bill C-7, the eligibility criteria for MAiD were changed to allow individuals with irreversible medical conditions, declining health, and suffering, but whose natural death was not reasonably foreseeable, to receive MAiD [ 3 ]. This population of MAiD applicants are referred to as Track 2 MAiD (those whose natural death is foreseeable are referred to as Track 1). Track 2 applicants are subject to additional safeguards under the 2021 C-7 legislation.

Three additional proposed changes to the legislation have been extensively studied by Canadian Expert Panels (Council of Canadian Academics [CCA]) [ 4 , 5 , 6 ] First, under the legislation that defines Track 2, individuals with mental disease as their sole underlying medical condition may apply for MAiD, but implementation of this practice is embargoed until March 2027 [ 4 ]. Second, there is consideration of allowing MAiD to be implemented through advanced consent. This would make it possible for persons living with dementia to receive MAID after they have lost the capacity to consent to the procedure [ 5 ]. Third, there is consideration of extending MAiD to mature minors. A mature minor is defined as “a person under the age of majority…and who has the capacity to understand and appreciate the nature and consequences of a decision” ([ 6 ] p. 5). In summary, since the legalization of MAiD in 2016 the eligibility criteria and safeguards have evolved significantly with consequent implications for nurses and nursing care. Further, the number of Canadians who access MAiD shows steady increases since 2016 [ 7 ] and it is expected that these increases will continue in the foreseeable future.

Nurses have been integral to MAiD care in the Canadian context. While other countries such as Belgium and the Netherlands also permit euthanasia, Canada is the first country to allow Nurse Practitioners (Registered Nurses with additional preparation typically achieved at the graduate level) to act independently as assessors and providers of MAiD [ 1 ]. Although the role of Registered Nurses (RNs) in MAiD is not defined in federal legislation, it has been addressed at the provincial/territorial-level with variability in scope of practice by region [ 8 , 9 ]. For example, there are differences with respect to the obligation of the nurse to provide information to patients about MAiD, and to the degree that nurses are expected to ensure that patient eligibility criteria and safeguards are met prior to their participation [ 10 ]. Studies conducted in the Canadian context indicate that RNs perform essential roles in MAiD care coordination; client and family teaching and support; MAiD procedural quality; healthcare provider and public education; and bereavement care for family [ 9 , 11 ]. Nurse practitioners and RNs are integral to a robust MAiD care system in Canada and hence need to be well-prepared for their role [ 12 ].

Previous studies have found that end of life care, and MAiD specifically, raise complex moral and ethical issues for nurses [ 13 , 14 , 15 , 16 ]. The knowledge, attitudes, and beliefs of nurses are important across practice settings because nurses have consistent, ongoing, and direct contact with patients who experience chronic or life-limiting health conditions. Canadian studies exploring nurses’ moral and ethical decision-making in relation to MAiD reveal that although some nurses are clear in their support for, or opposition to, MAiD, others are unclear on what they believe to be good and right [ 14 ]. Empirical findings suggest that nurses go through a period of moral sense-making that is often informed by their family, peers, and initial experiences with MAID [ 17 , 18 ]. Canadian legislation and policy specifies that nurses are not required to participate in MAiD and may recuse themselves as conscientious objectors with appropriate steps to ensure ongoing and safe care of patients [ 1 , 19 ]. However, with so many nurses having to reflect on and make sense of their moral position, it is essential that they are given adequate time and preparation to make an informed and thoughtful decision before they participate in a MAID death [ 20 , 21 ].

It is well established that nursing students receive inconsistent exposure to end of life care issues [ 22 ] and little or no training related to MAiD [ 23 ]. Without such education and reflection time in pre-entry nursing preparation, nurses are at significant risk for moral harm. An important first step in providing this preparation is to be able to assess the knowledge, values, and beliefs of nursing students regarding MAID and end of life care. As demand for MAiD increases along with the complexities of MAiD, it is critical to understand the knowledge, attitudes, and likelihood of engagement with MAiD among nursing students as a baseline upon which to build curriculum and as a means to track these variables over time.

Aim, design, and setting

The aim of this study was to develop a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in MAiD in the Canadian context. We sought to explore both their willingness to be involved in the registered nursing role and in the nurse practitioner role should they chose to prepare themselves to that level of education. The design was a mixed-method, modified e-Delphi method that entailed item generation, item refinement through an expert faculty panel [ 24 , 25 , 26 ], and initial item validation through a cognitive focus group interview with nursing students [ 27 ]. The settings were a University located in an urban area and a College located in a rural area in Western Canada.

Participants

A panel of 10 faculty from the two nursing education programs were recruited for Phase 2 of the e-Delphi. To be included, faculty were required to have a minimum of three years of experience in nurse education, be employed as nursing faculty, and self-identify as having experience with MAiD. A convenience sample of 5 fourth-year nursing students were recruited to participate in Phase 3. Students had to be in good standing in the nursing program and be willing to share their experiences of the survey in an online group interview format.

The modified e-Delphi was conducted in 3 phases: Phase 1 entailed item generation through literature and existing survey review. Phase 2 entailed item refinement through a faculty expert panel review with focus on content validity, prioritization, and revision of item wording [ 25 ]. Phase 3 entailed an assessment of face validity through focus group-based cognitive interview with nursing students.

Phase I. Item generation through literature review

The goal of phase 1 was to develop a bank of survey items that would represent the variables of interest and which could be provided to expert faculty in Phase 2. Initial survey items were generated through a literature review of similar surveys designed to assess knowledge and attitudes toward MAiD/euthanasia in healthcare providers; Canadian empirical studies on nurses’ roles and/or experiences with MAiD; and legislative and expert panel documents that outlined proposed changes to the legislative eligibility criteria and safeguards. The literature review was conducted in three online databases: CINAHL, PsycINFO, and Medline. Key words for the search included nurses , nursing students , medical students , NPs, MAiD , euthanasia , assisted death , and end-of-life care . Only articles written in English were reviewed. The legalization and legislation of MAiD is new in many countries; therefore, studies that were greater than twenty years old were excluded, no further exclusion criteria set for country.

Items from surveys designed to measure similar variables in other health care providers and geographic contexts were placed in a table and similar items were collated and revised into a single item. Then key variables were identified from the empirical literature on nurses and MAiD in Canada and checked against the items derived from the surveys to ensure that each of the key variables were represented. For example, conscientious objection has figured prominently in the Canadian literature, but there were few items that assessed knowledge of conscientious objection in other surveys and so items were added [ 15 , 21 , 28 , 29 ]. Finally, four case studies were added to the survey to address the anticipated changes to the Canadian legislation. The case studies were based upon the inclusion of mature minors, advanced consent, and mental disorder as the sole underlying medical condition. The intention was to assess nurses’ beliefs and comfort with these potential legislative changes.

Phase 2. Item refinement through expert panel review

The goal of phase 2 was to refine and prioritize the proposed survey items identified in phase 1 using a modified e-Delphi approach to achieve consensus among an expert panel [ 26 ]. Items from phase 1 were presented to an expert faculty panel using a Qualtrics (Provo, UT) online survey. Panel members were asked to review each item to determine if it should be: included, excluded or adapted for the survey. When adapted was selected faculty experts were asked to provide rationale and suggestions for adaptation through the use of an open text box. Items that reached a level of 75% consensus for either inclusion or adaptation were retained [ 25 , 26 ]. New items were categorized and added, and a revised survey was presented to the panel of experts in round 2. Panel members were again asked to review items, including new items, to determine if it should be: included, excluded, or adapted for the survey. Round 2 of the modified e-Delphi approach also included an item prioritization activity, where participants were then asked to rate the importance of each item, based on a 5-point Likert scale (low to high importance), which De Vaus [ 30 ] states is helpful for increasing the reliability of responses. Items that reached a 75% consensus on inclusion were then considered in relation to the importance it was given by the expert panel. Quantitative data were managed using SPSS (IBM Corp).

Phase 3. Face validity through cognitive interviews with nursing students

The goal of phase 3 was to obtain initial face validity of the proposed survey using a sample of nursing student informants. More specifically, student participants were asked to discuss how items were interpreted, to identify confusing wording or other problematic construction of items, and to provide feedback about the survey as a whole including readability and organization [ 31 , 32 , 33 ]. The focus group was held online and audio recorded. A semi-structured interview guide was developed for this study that focused on clarity, meaning, order and wording of questions; emotions evoked by the questions; and overall survey cohesion and length was used to obtain data (see Supplementary Material 2  for the interview guide). A prompt to “think aloud” was used to limit interviewer-imposed bias and encourage participants to describe their thoughts and response to a given item as they reviewed survey items [ 27 ]. Where needed, verbal probes such as “could you expand on that” were used to encourage participants to expand on their responses [ 27 ]. Student participants’ feedback was collated verbatim and presented to the research team where potential survey modifications were negotiated and finalized among team members. Conventional content analysis [ 34 ] of focus group data was conducted to identify key themes that emerged through discussion with students. Themes were derived from the data by grouping common responses and then using those common responses to modify survey items.

Ten nursing faculty participated in the expert panel. Eight of the 10 faculty self-identified as female. No faculty panel members reported conscientious objector status and ninety percent reported general agreement with MAiD with one respondent who indicated their view as “unsure.” Six of the 10 faculty experts had 16 years of experience or more working as a nurse educator.

Five nursing students participated in the cognitive interview focus group. The duration of the focus group was 2.5 h. All participants identified that they were born in Canada, self-identified as female (one preferred not to say) and reported having received some instruction about MAiD as part of their nursing curriculum. See Tables  1 and 2 for the demographic descriptors of the study sample. Study results will be reported in accordance with the study phases. See Fig.  1 for an overview of the results from each phase.

Fig. 1  Overview of survey development findings

Phase 1: survey item generation

Review of the literature identified that no existing survey was available for use with nursing students in the Canadian context. However, an analysis of themes across qualitative and quantitative studies of physicians, medical students, nurses, and nursing students provided sufficient data to develop a preliminary set of items suitable for adaptation to a population of nursing students.

Four major themes and factors that influence knowledge, attitudes, and beliefs about MAiD were evident from the literature: (i) endogenous or individual factors such as age, gender, personally held values, religion, religiosity, and/or spirituality [ 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ], (ii) experience with death and dying in personal and/or professional life [ 35 , 40 , 41 , 43 , 44 , 45 ], (iii) training including curricular instruction about clinical role, scope of practice, or the law [ 23 , 36 , 39 ], and (iv) exogenous or social factors such as the influence of key leaders, colleagues, friends and/or family, professional and licensure organizations, support within professional settings, and/or engagement in MAiD in an interdisciplinary team context [ 9 , 35 , 46 ].

Studies of nursing students also suggest overlap across these categories. For example, value for patient autonomy [ 23 ] and the moral complexity of decision-making [ 37 ] are important factors that contribute to attitudes about MAiD and may stem from a blend of personally held values coupled with curricular content, professional training and norms, and clinical exposure. For example, students report that participation in end of life care allows for personal growth, shifts in perception, and opportunities to build therapeutic relationships with their clients [ 44 , 47 , 48 ].

Preliminary items generated from the literature resulted in 56 questions from 11 published sources (See Table  3 ). These items were constructed across four main categories: (i) socio-demographic questions; (ii) end of life care questions; (iii) knowledge about MAiD; or (iv) comfort and willingness to participate in MAiD. Knowledge questions were refined to reflect current MAiD legislation, policies, and regulatory frameworks. Falconer [ 39 ] and Freeman [ 45 ] studies were foundational sources for item selection. Additionally, four case studies were written to reflect the most recent anticipated changes to MAiD legislation and all used the same open-ended core questions to address respondents’ perspectives about the patient’s right to make the decision, comfort in assisting a physician or NP to administer MAiD in that scenario, and hypothesized comfort about serving as a primary provider if qualified as an NP in future. Response options for the survey were also constructed during this stage and included: open text, categorical, yes/no , and Likert scales.

Phase 2: faculty expert panel review

Of the 56 items presented to the faculty panel, 54 questions reached 75% consensus. However, based upon the qualitative responses 9 items were removed largely because they were felt to be repetitive. Items that generated the most controversy were related to measuring religion and spirituality in the Canadian context, defining end of life care when there is no agreed upon time frames (e.g., last days, months, or years), and predicting willingness to be involved in a future events – thus predicting their future selves. Phase 2, round 1 resulted in an initial set of 47 items which were then presented back to the faculty panel in round 2.

Of the 47 initial questions presented to the panel in round 2, 45 reached a level of consensus of 75% or greater, and 34 of these questions reached a level of 100% consensus [ 27 ] of which all participants chose to include without any adaptations) For each question, level of importance was determined based on a 5-point Likert scale (1 = very unimportant, 2 = somewhat unimportant, 3 = neutral, 4 = somewhat important, and 5 = very important). Figure  2 provides an overview of the level of importance assigned to each item.

Ranking level of importance for survey items

After round 2, a careful analysis of participant comments and level of importance was completed by the research team. While the main method of survey item development came from participants’ response to the first round of Delphi consensus ratings, level of importance was used to assist in the decision of whether to keep or modify questions that created controversy, or that rated lower in the include/exclude/adapt portion of the Delphi. Survey items that rated low in level of importance included questions about future roles, sex and gender, and religion/spirituality. After deliberation by the research committee, these questions were retained in the survey based upon the importance of these variables in the scientific literature.

Of the 47 questions remaining from Phase 2, round 2, four were revised. In addition, the two questions that did not meet the 75% cut off level for consensus were reviewed by the research team. The first question reviewed was What is your comfort level with providing a MAiD death in the future if you were a qualified NP ? Based on a review of participant comments, it was decided to retain this question for the cognitive interviews with students in the final phase of testing. The second question asked about impacts on respondents’ views of MAiD and was changed from one item with 4 subcategories into 4 separate items, resulting in a final total of 51 items for phase 3. The revised survey was then brought forward to the cognitive interviews with student participants in Phase 3. (see Supplementary Material 1 for a complete description of item modification during round 2).

Phase 3. Outcomes of cognitive interview focus group

Of the 51 items reviewed by student participants, 29 were identified as clear with little or no discussion. Participant comments for the remaining 22 questions were noted and verified against the audio recording. Following content analysis of the comments, four key themes emerged through the student discussion: unclear or ambiguous wording; difficult to answer questions; need for additional response options; and emotional response evoked by questions. An example of unclear or ambiguous wording was a request for clarity in the use of the word “sufficient” in the context of assessing an item that read “My nursing education has provided sufficient content about the nursing role in MAiD.” “Sufficient” was viewed as subjective and “laden with…complexity that distracted me from the question.” The group recommended rewording the item to read “My nursing education has provided enough content for me to care for a patient considering or requesting MAiD.”

An example of having difficulty answering questions related to limited knowledge related to terms used in the legislation such as such as safeguards , mature minor , eligibility criteria , and conscientious objection. Students were unclear about what these words meant relative to the legislation and indicated that this lack of clarity would hamper appropriate responses to the survey. To ensure that respondents are able to answer relevant questions, student participants recommended that the final survey include explanation of key terms such as mature minor and conscientious objection and an overview of current legislation.

Response options were also a point of discussion. Participants noted a lack of distinction between response options of unsure and unable to say . Additionally, scaling of attitudes was noted as important since perspectives about MAiD are dynamic and not dichotomous “agree or disagree” responses. Although the faculty expert panel recommended the integration of the demographic variables of religious and/or spiritual remain as a single item, the student group stated a preference to have religion and spirituality appear as separate items. The student focus group also took issue with separate items for the variables of sex and gender, specifically that non-binary respondents might feel othered or “outed” particularly when asked to identify their sex. These variables had been created based upon best practices in health research but students did not feel they were appropriate in this context [ 49 ]. Finally, students agreed with the faculty expert panel in terms of the complexity of projecting their future involvement as a Nurse Practitioner. One participant stated: “I certainly had to like, whoa, whoa, whoa. Now let me finish this degree first, please.” Another stated, “I'm still imagining myself, my future career as an RN.”

Finally, student participants acknowledged the array of emotions that some of the items produced for them. For example, one student described positive feelings when interacting with the survey. “Brought me a little bit of feeling of joy. Like it reminded me that this is the last piece of independence that people grab on to.” Another participant, described the freedom that the idea of an advance request gave her. “The advance request gives the most comfort for me, just with early onset Alzheimer’s and knowing what it can do.” But other participants described less positive feelings. For example, the mature minor case study yielded a comment: “This whole scenario just made my heart hurt with the idea of a child requesting that.”

Based on the data gathered from the cognitive interview focus group of nursing students, revisions were made to 11 closed-ended questions (see Table  4 ) and 3 items were excluded. In the four case studies, the open-ended question related to a respondents’ hypothesized actions in a future role as NP were removed. The final survey consists of 45 items including 4 case studies (see Supplementary Material 3 ).

The aim of this study was to develop and validate a survey that can be used to track the growth of knowledge about MAiD among nursing students over time, inform training programs about curricular needs, and evaluate attitudes and willingness to participate in MAiD at time-points during training or across nursing programs over time.

The faculty expert panel and student participants in the cognitive interview focus group identified a need to establish core knowledge of the terminology and legislative rules related to MAiD. For example, within the cognitive interview group of student participants, several acknowledged lack of clear understanding of specific terms such as “conscientious objector” and “safeguards.” Participants acknowledged discomfort with the uncertainty of not knowing and their inclination to look up these terms to assist with answering the questions. This survey can be administered to nursing or pre-nursing students at any phase of their training within a program or across training programs. However, in doing so it is important to acknowledge that their baseline knowledge of MAiD will vary. A response option of “not sure” is important and provides a means for respondents to convey uncertainty. If this survey is used to inform curricular needs, respondents should be given explicit instructions not to conduct online searches to inform their responses, but rather to provide an honest appraisal of their current knowledge and these instructions are included in the survey (see Supplementary Material 3 ).

Some provincial regulatory bodies have established core competencies for entry-level nurses that include MAiD. For example, the BC College of Nurses and Midwives (BCCNM) requires “knowledge about ethical, legal, and regulatory implications of medical assistance in dying (MAiD) when providing nursing care.” (10 p. 6) However, across Canada curricular content and coverage related to end of life care and MAiD is variable [ 23 ]. Given the dynamic nature of the legislation that includes portions of the law that are embargoed until 2024, it is important to ensure that respondents are guided by current and accurate information. As the law changes, nursing curricula, and public attitudes continue to evolve, inclusion of core knowledge and content is essential and relevant for investigators to be able to interpret the portions of the survey focused on attitudes and beliefs about MAiD. Content knowledge portions of the survey may need to be modified over time as legislation and training change and to meet the specific purposes of the investigator.

Given the sensitive nature of the topic, it is strongly recommended that surveys be conducted anonymously and that students be provided with an opportunity to discuss their responses to the survey. A majority of feedback from both the expert panel of faculty and from student participants related to the wording and inclusion of demographic variables, in particular religion, religiosity, gender identity, and sex assigned at birth. These and other demographic variables have the potential to be highly identifying in small samples. In any instance in which the survey could be expected to yield demographic group sizes less than 5, users should eliminate the demographic variables from the survey. For example, the profession of nursing is highly dominated by females with over 90% of nurses who identify as female [ 50 ]. Thus, a survey within a single class of students or even across classes in a single institution is likely to yield a small number of male respondents and/or respondents who report a difference between sex assigned at birth and gender identity. When variables that serve to identify respondents are included, respondents are less likely to complete or submit the survey, to obscure their responses so as not to be identifiable, or to be influenced by social desirability bias in their responses rather than to convey their attitudes accurately [ 51 ]. Further, small samples do not allow for conclusive analyses or interpretation of apparent group differences. Although these variables are often included in surveys, such demographics should be included only when anonymity can be sustained. In small and/or known samples, highly identifying variables should be omitted.

There are several limitations associated with the development of this survey. The expert panel was comprised of faculty who teach nursing students and are knowledgeable about MAiD and curricular content, however none identified as a conscientious objector to MAiD. Ideally, our expert panel would have included one or more conscientious objectors to MAiD to provide a broader perspective. Review by practitioners who participate in MAiD, those who are neutral or undecided, and practitioners who are conscientious objectors would ensure broad applicability of the survey. This study included one student cognitive interview focus group with 5 self-selected participants. All student participants had held discussions about end of life care with at least one patient, 4 of 5 participants had worked with a patient who requested MAiD, and one had been present for a MAiD death. It is not clear that these participants are representative of nursing students demographically or by experience with end of life care. It is possible that the students who elected to participate hold perspectives and reflections on patient care and MAiD that differ from students with little or no exposure to end of life care and/or MAiD. However, previous studies find that most nursing students have been involved with end of life care including meaningful discussions about patients’ preferences and care needs during their education [ 40 , 44 , 47 , 48 , 52 ]. Data collection with additional student focus groups with students early in their training and drawn from other training contexts would contribute to further validation of survey items.

Future studies should incorporate pilot testing with small sample of nursing students followed by a larger cross-program sample to allow evaluation of the psychometric properties of specific items and further refinement of the survey tool. Consistent with literature about the importance of leadership in the context of MAiD [ 12 , 53 , 54 ], a study of faculty knowledge, beliefs, and attitudes toward MAiD would provide context for understanding student perspectives within and across programs. Additional research is also needed to understand the timing and content coverage of MAiD across Canadian nurse training programs’ curricula.

The implementation of MAiD is complex and requires understanding of the perspectives of multiple stakeholders. Within the field of nursing this includes clinical providers, educators, and students who will deliver clinical care. A survey to assess nursing students’ attitudes toward and willingness to participate in MAiD in the Canadian context is timely, due to the legislation enacted in 2016 and subsequent modifications to the law in 2021 with portions of the law to be enacted in 2027. Further development of this survey could be undertaken to allow for use in settings with practicing nurses or to allow longitudinal follow up with students as they enter practice. As the Canadian landscape changes, ongoing assessment of the perspectives and needs of health professionals and students in the health professions is needed to inform policy makers, leaders in practice, curricular needs, and to monitor changes in attitudes and practice patterns over time.

Availability of data and materials

The datasets used and/or analysed during the current study are not publicly available due to small sample sizes, but are available from the corresponding author on reasonable request.

Abbreviations

British Columbia College of Nurses and Midwives

Medical assistance in dying

Nurse practitioner

Registered nurse

University of British Columbia Okanagan

Nicol J, Tiedemann M. Legislative Summary: Bill C-14: An Act to amend the Criminal Code and to make related amendments to other Acts (medical assistance in dying). Available from: https://lop.parl.ca/staticfiles/PublicWebsite/Home/ResearchPublications/LegislativeSummaries/PDF/42-1/c14-e.pdf .

Downie J, Scallion K. Foreseeably unclear. The meaning of the “reasonably foreseeable” criterion for access to medical assistance in dying in Canada. Dalhousie Law J. 2018;41(1):23–57.

Nicol J, Tiedeman M. Legislative summary of Bill C-7: an act to amend the criminal code (medical assistance in dying). Ottawa: Government of Canada; 2021.

Google Scholar  

Council of Canadian Academies. The state of knowledge on medical assistance in dying where a mental disorder is the sole underlying medical condition. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2018/12/The-State-of-Knowledge-on-Medical-Assistance-in-Dying-Where-a-Mental-Disorder-is-the-Sole-Underlying-Medical-Condition.pdf .

Council of Canadian Academies. The state of knowledge on advance requests for medical assistance in dying. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2019/02/The-State-of-Knowledge-on-Advance-Requests-for-Medical-Assistance-in-Dying.pdf .

Council of Canadian Academies. The state of knowledge on medical assistance in dying for mature minors. Ottawa; 2018. Available from: https://cca-reports.ca/wp-content/uploads/2018/12/The-State-of-Knowledge-on-Medical-Assistance-in-Dying-for-Mature-Minors.pdf .

Health Canada. Third annual report on medical assistance in dying in Canada 2021. Ottawa; 2022. [cited 2023 Oct 23]. Available from: https://www.canada.ca/en/health-canada/services/medical-assistance-dying/annual-report-2021.html .

Banner D, Schiller CJ, Freeman S. Medical assistance in dying: a political issue for nurses and nursing in Canada. Nurs Philos. 2019;20(4): e12281.

Article   PubMed   Google Scholar  

Pesut B, Thorne S, Stager ML, Schiller CJ, Penney C, Hoffman C, et al. Medical assistance in dying: a review of Canadian nursing regulatory documents. Policy Polit Nurs Pract. 2019;20(3):113–30.

Article   PubMed   PubMed Central   Google Scholar  

College of Registered Nurses of British Columbia. Scope of practice for registered nurses [Internet]. Vancouver; 2018. Available from: https://www.bccnm.ca/Documents/standards_practice/rn/RN_ScopeofPractice.pdf .

Pesut B, Thorne S, Schiller C, Greig M, Roussel J, Tishelman C. Constructing good nursing practice for medical assistance in dying in Canada: an interpretive descriptive study. Global Qual Nurs Res. 2020;7:2333393620938686. https://doi.org/10.1177/2333393620938686 .

Article   Google Scholar  

Pesut B, Thorne S, Schiller CJ, Greig M, Roussel J. The rocks and hard places of MAiD: a qualitative study of nursing practice in the context of legislated assisted death. BMC Nurs. 2020;19:12. https://doi.org/10.1186/s12912-020-0404-5 .

Pesut B, Greig M, Thorne S, Burgess M, Storch JL, Tishelman C, et al. Nursing and euthanasia: a narrative review of the nursing ethics literature. Nurs Ethics. 2020;27(1):152–67.

Pesut B, Thorne S, Storch J, Chambaere K, Greig M, Burgess M. Riding an elephant: a qualitative study of nurses’ moral journeys in the context of Medical Assistance in Dying (MAiD). Journal Clin Nurs. 2020;29(19–20):3870–81.

Lamb C, Babenko-Mould Y, Evans M, Wong CA, Kirkwood KW. Conscientious objection and nurses: results of an interpretive phenomenological study. Nurs Ethics. 2018;26(5):1337–49.

Wright DK, Chan LS, Fishman JR, Macdonald ME. “Reflection and soul searching:” Negotiating nursing identity at the fault lines of palliative care and medical assistance in dying. Social Sci & Med. 2021;289: 114366.

Beuthin R, Bruce A, Scaia M. Medical assistance in dying (MAiD): Canadian nurses’ experiences. Nurs Forum. 2018;54(4):511–20.

Bruce A, Beuthin R. Medically assisted dying in Canada: "Beautiful Death" is transforming nurses' experiences of suffering. The Canadian J Nurs Res | Revue Canadienne de Recherche en Sci Infirmieres. 2020;52(4):268–77. https://doi.org/10.1177/0844562119856234 .

Canadian Nurses Association. Code of ethics for registered nurses. Ottawa; 2017. Available from: https://www.cna-aiic.ca/en/nursing/regulated-nursing-in-canada/nursing-ethics .

Canadian Nurses Association. National nursing framework on Medical Assistance in Dying in Canada. Ottawa: 2017. Available from: https://www.virtualhospice.ca/Assets/cna-national-nursing-framework-on-maidEng_20170216155827.pdf .

Pesut B, Thorne S, Greig M. Shades of gray: conscientious objection in medical assistance in dying. Nursing Inq. 2020;27(1): e12308.

Durojaiye A, Ryan R, Doody O. Student nurse education and preparation for palliative care: a scoping review. PLoS ONE. 2023. https://doi.org/10.1371/journal.pone.0286678 .

McMechan C, Bruce A, Beuthin R. Canadian nursing students’ experiences with medical assistance in dying | Les expériences d’étudiantes en sciences infirmières au regard de l’aide médicale à mourir. Qual Adv Nurs Educ - Avancées en Formation Infirmière. 2019;5(1). https://doi.org/10.17483/2368-6669.1179 .

Adler M, Ziglio E. Gazing into the oracle. The Delphi method and its application to social policy and public health. London: Jessica Kingsley Publishers; 1996

Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12.

Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. 1st ed. City: Wiley; 2011.

Willis GB. Cognitive interviewing: a tool for improving questionnaire design. 1st ed. Thousand Oaks, Calif: Sage; 2005. ISBN: 9780761928041

Lamb C, Evans M, Babenko-Mould Y, Wong CA, Kirkwood EW. Conscience, conscientious objection, and nursing: a concept analysis. Nurs Ethics. 2017;26(1):37–49.

Lamb C, Evans M, Babenko-Mould Y, Wong CA, Kirkwood K. Nurses’ use of conscientious objection and the implications of conscience. J Adv Nurs. 2018;75(3):594–602.

de Vaus D. Surveys in social research. 6th ed. Abingdon, Oxon: Routledge; 2014.

Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: A primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149 .

Puchta C, Potter J. Focus group practice. 1st ed. London: Sage; 2004.

Book   Google Scholar  

Streiner DL, Norman GR, Cairney J. Health measurement scales: a practical guide to their development and use. 5th ed. Oxford: Oxford University Press; 2015.

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Adesina O, DeBellis A, Zannettino L. Third-year Australian nursing students’ attitudes, experiences, knowledge, and education concerning end-of-life care. Int J of Palliative Nurs. 2014;20(8):395–401.

Bator EX, Philpott B, Costa AP. This moral coil: a cross-sectional survey of Canadian medical student attitudes toward medical assistance in dying. BMC Med Ethics. 2017;18(1):58.

Beuthin R, Bruce A, Scaia M. Medical assistance in dying (MAiD): Canadian nurses’ experiences. Nurs Forum. 2018;53(4):511–20.

Brown J, Goodridge D, Thorpe L, Crizzle A. What is right for me, is not necessarily right for you: the endogenous factors influencing nonparticipation in medical assistance in dying. Qual Health Res. 2021;31(10):1786–1800.

Falconer J, Couture F, Demir KK, Lang M, Shefman Z, Woo M. Perceptions and intentions toward medical assistance in dying among Canadian medical students. BMC Med Ethics. 2019;20(1):22.

Green G, Reicher S, Herman M, Raspaolo A, Spero T, Blau A. Attitudes toward euthanasia—dual view: Nursing students and nurses. Death Stud. 2022;46(1):124–31.

Hosseinzadeh K, Rafiei H. Nursing student attitudes toward euthanasia: a cross-sectional study. Nurs Ethics. 2019;26(2):496–503.

Ozcelik H, Tekir O, Samancioglu S, Fadiloglu C, Ozkara E. Nursing students’ approaches toward euthanasia. Omega (Westport). 2014;69(1):93–103.

Canning SE, Drew C. Canadian nursing students’ understanding, and comfort levels related to medical assistance in dying. Qual Adv Nurs Educ - Avancées en Formation Infirmière. 2022;8(2). https://doi.org/10.17483/2368-6669.1326 .

Edo-Gual M, Tomás-Sábado J, Bardallo-Porras D, Monforte-Royo C. The impact of death and dying on nursing students: an explanatory model. J Clin Nurs. 2014;23(23–24):3501–12.

Freeman LA, Pfaff KA, Kopchek L, Liebman J. Investigating palliative care nurse attitudes towards medical assistance in dying: an exploratory cross-sectional study. J Adv Nurs. 2020;76(2):535–45.

Brown J, Goodridge D, Thorpe L, Crizzle A. “I am okay with it, but I am not going to do it:” the exogenous factors influencing non-participation in medical assistance in dying. Qual Health Res. 2021;31(12):2274–89.

Dimoula M, Kotronoulas G, Katsaragakis S, Christou M, Sgourou S, Patiraki E. Undergraduate nursing students’ knowledge about palliative care and attitudes towards end-of-life care: A three-cohort, cross-sectional survey. Nurs Educ Today. 2019;74:7–14.

Matchim Y, Raetong P. Thai nursing students’ experiences of caring for patients at the end of life: a phenomenological study. Int J Palliative Nurs. 2018;24(5):220–9.

Canadian Institute for Health Research. Sex and gender in health research [Internet]. Ottawa: CIHR; 2021 [cited 2023 Oct 23]. Available from: https://cihr-irsc.gc.ca/e/50833.html .

Canadian Nurses’ Association. Nursing statistics. Ottawa: CNA; 2023 [cited 2023 Oct 23]. Available from: https://www.cna-aiic.ca/en/nursing/regulated-nursing-in-canada/nursing-statistics .

Krumpal I. Determinants of social desirability bias in sensitive surveys: a literature review. Qual Quant. 2013;47(4):2025–47. https://doi.org/10.1007/s11135-011-9640-9 .

Ferri P, Di Lorenzo R, Stifani S, Morotti E, Vagnini M, Jiménez Herrera MF, et al. Nursing student attitudes toward dying patient care: a European multicenter cross-sectional study. Acta Bio Medica Atenei Parmensis. 2021;92(S2): e2021018.

PubMed   PubMed Central   Google Scholar  

Beuthin R, Bruce A. Medical assistance in dying (MAiD): Ten things leaders need to know. Nurs Leadership. 2018;31(4):74–81.

Thiele T, Dunsford J. Nurse leaders’ role in medical assistance in dying: a relational ethics approach. Nurs Ethics. 2019;26(4):993–9.

Download references

Acknowledgements

We would like to acknowledge the faculty and students who generously contributed their time to this work.

JS received a student traineeship through the Principal Research Chairs program at the University of British Columbia Okanagan.

Author information

Authors and affiliations.

School of Health and Human Services, Selkirk College, Castlegar, BC, Canada

Jocelyn Schroeder & Barbara Pesut

School of Nursing, University of British Columbia Okanagan, Kelowna, BC, Canada

Barbara Pesut, Lise Olsen, Nelly D. Oelke & Helen Sharp

You can also search for this author in PubMed   Google Scholar

Contributions

JS made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and drafting and substantively revising the work. JS has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. BP made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and drafting and substantively revising the work. BP has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. LO made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and substantively revising the work. LO has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. NDO made substantial contributions to the conception of the work; data acquisition, analysis, and interpretation; and substantively revising the work. NDO has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature. HS made substantial contributions to drafting and substantively revising the work. HS has approved the submitted version and agreed to be personally accountable for the author's own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Authors’ information

JS conducted this study as part of their graduate requirements in the School of Nursing, University of British Columbia Okanagan.

Corresponding author

Correspondence to Barbara Pesut .

Ethics declarations

Ethics approval and consent to participate.

The research was approved by the Selkirk College Research Ethics Board (REB) ID # 2021–011 and the University of British Columbia Behavioral Research Ethics Board ID # H21-01181.

All participants provided written and informed consent through approved consent processes. Research was conducted in accordance with the Declaration of Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Schroeder, J., Pesut, B., Olsen, L. et al. Developing a survey to measure nursing students’ knowledge, attitudes and beliefs, influences, and willingness to be involved in Medical Assistance in Dying (MAiD): a mixed method modified e-Delphi study. BMC Nurs 23 , 326 (2024). https://doi.org/10.1186/s12912-024-01984-z

Download citation

Received : 24 October 2023

Accepted : 28 April 2024

Published : 14 May 2024

DOI : https://doi.org/10.1186/s12912-024-01984-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Medical assistance in dying (MAiD)
• End of life care
• Student nurses
• Nursing education

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

The Role and Interpretation of Pilot Studies in Clinical Research

Andrew c. leon.

1 Weill Cornell Medical College, Department of Psychiatry, New York, NY

Lori L. Davis

2 University of Alabama School of Medicine, Birmingham, AL VA Medical Center, Tuscaloosa, AL

Helena C. Kraemer

3 Stanford University, Department of Psychiatry and Behavioral Sciences, Stanford, CA

Pilot studies represent a fundamental phase of the research process. The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention.

A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot design.

A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study. Investigators should be forthright in stating these objectives of a pilot study. Grant reviewers and other stakeholders should expect no more.

INTRODUCTION

Over the past several years, research funding agencies have requested applications for pilot studies that are typically limited to a shorter duration (one to three years) and a reduced budget using, for example, the NIH R34 funding mechanism ( NIMH, 2010 ). Pilot studies play a key role in the development or refinement of new interventions, assessments, and other study procedures. Commonly, results from pilot studies are used to support more expensive and lengthier pivotal efficacy or effectiveness studies. Importantly, investigators, grant reviewers, and other stakeholders need to be aware of the essential elements, appropriate role, and exceptional strengths and limitations in the interpretation of pilot studies.

A pilot study is, “A small-scale test of the methods and procedures to be used on a larger scale …” ( Porta, 2008 ). The fundamental purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to ultimately be used in a larger scale study. This applies to all types of research studies. Here we use the randomized controlled clinical trial (RCT) for illustration. Prior to initiating a full scale RCT an investigator may choose to conduct a pilot study in order to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and/or implementation of the novel intervention. A pilot study, however, is not used for hypothesis testing. Instead it serves as an earlier-phase developmental function that will enhance the probability of success in the larger subsequent RCTs that are anticipated.

For purpose of contrast, a hypothesis testing clinical trial is designed to compare randomized treatment groups in order to draw an inference about efficacy/effectiveness and safety in the patient population , based on sample results. The primary goal in designing such a study is to minimize the bias in the estimate of the treatment effect. ( Leon et al., 2006 ; Leon & Davis, 2009 ). That is, the trial is designed to ask the question, “Is the treatment efficacious, and if so, what is the magnitude of the effect?”. Features of RCTs that help us achieve this goal are randomized group assignment, double-blinded assessments, and control or comparison groups.

This manuscript will focus on pilot studies, those used to shape some, but not all aspects of the design and implementation of hypothesis testing clinical trials. It is the feasibility results, not the efficacy or safety results, that inform subsequent trials. The objective of this manuscript is to elaborate on each of these points: efficacy, safety, and feasibility. We discuss both the design of a pilot study and the interpretation and application of pilot study results. What is discussed here applies to pilot studies, feasibility studies and proof of concept studies, terms that have been used somewhat interchangeably in the literature and henceforth are referred to here as “pilot studies”.

WHAT A PILOT STUDY CAN DO: ASSESS FEASIBILITY

Pilot study results can guide in the design and implementation of larger scale efficacy studies. There are several aspects of RCT feasibility that are informed by conducting a pilot study. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, and implementation of the novel intervention and each of these can be quantified ( Table 1 ). Study components that are deemed infeasible or unsatisfactory should be modified in the subsequent trial or removed altogether.

Aspects of Feasibility that Can be Examined with a Pilot Study

Rationale for a Control or Comparison Group in a Pilot

The inclusion of a control or comparator group in a full scale trial accounts for the passage of time, the increased attention received in the study, the expectation of a therapeutic intervention, and the psychological consequences of legitimized sick role.( Klerman, 1986 ) Nevertheless, an investigator might wonder what purpose a control group serves in a pilot if no inferential comparisons are to be conducted. Although not essential for many aspects of the study, inclusion of a control group allows for a more realistic examination of recruitment, randomization, implementation of interventions, blinded assessment procedures, and retention in blinded interventions. Each aspect of feasibility could be quite different from an uncontrolled study when intervention assignment is randomized and blinded, particularly if placebo is a distinct possibility. In an open label pilot that has no control group, participants are recruited to a known, albeit experimental, intervention with no risk of receiving placebo. Assessments are conducted in an unblinded fashion. The implementation of only one intervention can be evaluated. Retention information is based on those receiving unblinded treatment. With these issues in mind, a pilot study can better address its goals if a control group is part of the design. A control group would also be particularly illuminating for psychotherapy or psychosocial interventions, whereby the control group’s aspects and procedures are also tested for feasibility, consistency, and acceptability.

Good Clinical Practices

A pilot study provides opportunity to develop consistent practices to enhance data integrity and the protection of human subjects. These good clinical practices include the refinement of source documentation, informed consent procedures, data collection tools, regulatory reporting procedures, and monitoring/oversight procedures, especially when multiple sites and investigators are engaged in the study. A pilot study can be critical in research staff training and provide experiences that strengthen and confirm competencies and skills required for the investigation to be conducted with accuracy and precision.

SAMPLE SIZE DETERMINATION IN DESIGNING A PILOT STUDY

A pilot study is not a hypothesis testing study. Therefore, no inferential statistical tests should be proposed in a pilot study protocol. With no inferential statistical tests, a pilot study will not provide p -values. Power analyses are used to determine the sample size that is needed to provide adequate statistical power (typically 80% or 90%) to detect a clinically meaningful difference with the specified inferential statistical test. However, power analyses should not be presented in an application for a pilot study that does not propose inferential tests. A pilot sample size is instead based on the pragmatics of recruitment and the necessities for examining feasibility.

Pilot Data for a Pilot Study

Pilot studies are exploratory ventures. Pilot studies generate pilot data, their design need not be guided with the support of prior pilot data. It is quite reasonable and expected that a pilot study is proposed with no pilot or other preliminary data supporting the proposal and that its proposed sample size is based on pragmatics such as patient flow and budgetary constraints. This does not preclude the need for a theoretical rationale for the intervention or the methodology being proposed for a pilot study.

Are Pilot Data Included in the Larger Trial?

Pilot study data generally should not be combined with data from the subsequent larger scale study. This is because it is quite likely that the methods will be modified after the pilot, even if minimally. Such changes in protocol risk adding additional, perhaps unknown, source of variation. However, if a well-specified adaptive design were explicated prior to the start of a pilot study, and the risk of elevated type I error appropriately controlled, it is conceivable that data from before and after protocol changes could, in fact, be pooled. This is a rare exception.

EXCEEDING THE LIMITATIONS OF A PILOT STUDY: WHAT PILOT STUDIES CANNOT DO

Although a pilot study will undoubtedly incorporate relevant outcome measures and can serve a vital role in treatment development, it is not, and should not, be considered a preliminary test of the intervention hypothesis. There are two fundamental reasons that hypothesis testing is not used in a pilot study: the limited state of knowledge about the methods or intervention in the patient population to be studied and the smaller proposed sample size.

Tolerability and Preliminary Safety

Only in an extreme, unfortunate case, where a death occurs or repeated serious adverse events surface, do pilot studies inform the safety of testing an intervention due to the small sample size. However, pilot studies provide an opportunity to implement and examine the feasibility of the adverse event reporting system. Nevertheless, if some safety concerns are detected in a pilot study group-specific rates (with 95% confidence intervals) should be reported for adverse events, treatment emergent adverse events and serious adverse events. When event rates are reported and no adverse event is observed for a particular category, the rule of three should be applied to estimate the upper bound of the 95% CI, where the upper bound is approximately 3/n. ( Jovanovic & Levy, 1997 ; Jovanovic et al., 1997 ) For example, consider a study with N = 15 receiving medication and no suicidal ideation was reported for that group. Although the observed rate of suicidal ideation is 0%, the upper bound of the 95% confidence interval is 3/15 or 20%. This imprecision, seen in the wide confidence interval, underscores the limited value of safety data from a pilot study.

Pilot Study Effect Sizes and Sample Size Determination

There has been a venerable tradition of using pilot studies to estimate between group effect sizes that, in turn, are used to inform the design of subsequent larger scale hypothesis testing studies. Despite it widespread use, it has been argued that the tradition is ill-founded. ( Friedman, Furberg and DeMets, 1998 ; Kraemer et al., 2006 ) Pilot study results should not be used for sample size determination due to the inherent imprecision in between treatment group effect size estimates from studies with small samples. Furthermore, pilot results that are presented to grant review committees tend to be selective, overly optimistic and, at times, misrepresentational.

The adverse consequences of using a pilot study effect size for sample size estimation correspond with the two errors of inferential testing: false positive results (Type I error) and false negative results (Type II error). If a pilot study effect size is unduly large (i.e., a false positive result), subsequent trials will be designed with an inadequate number of participants to provide the statistical power needed to detect clinically meaningful effects and that would lead to negative trials. If a pilot study effect size is unduly small (i.e., a false negative result), subsequent development of the intervention could very well be terminated – even if the intervention eventually would have proven to be effective. Unfortunately, a false negative result could preclude the opportunity to further examine its latent efficacy.

An essential challenge of therapeutic development is that the true population effect size is unknown at the time a pilot study is designed. It is this gap in knowledge that motivates much research. An enthusiastic investigator may well believe that a series of cases provides evidence of efficacy, but such data are observational and uncontrolled; realistically, they are seldom replicated in RCTs -- as seen years ago with reserpine ( Kinross-Wright, 1955 ; Campden-Main, Wegielski, 1955 ; Goller 1960 ). A case series estimate tends to be steeped in optimism, particularly if an estimate of such magnitude is seldom, if ever, seen in full scale trials for psychiatric disorders. Nevertheless, it is not unusual for research grant applications and pilot study publications to convey such optimism, particularly when based on pilot data.

It is possible, but highly unlikely, that the between group effect size (d) from a pilot study sample will provide a reasonable estimate of the population effect size ( Δ ), but that cannot be known based on the pilot data. (It is the population effect size, not the sample effect size, that an RCT is designed to detect with sufficient power.) This estimation problem has to do with the precision of d and its relation to sample size. Estimates become more precise with larger sample sizes. Therefore, estimates of effect sizes should not be a specific aim of a pilot proposal. This applies to effects sizes for any type of outcome, be it a severity rating, a response status, or survival status. The reasoning for this is as follows.

Hypothetical Example

Precision is embodied in the confidence interval (CI) around d . By definition, there is a 95% probability that Δ falls within the range of the 95% CI. Consider some examples, initially a hypothetical example. For simplicity, assume that two groups (e.g., active and placebo) of equal size ( n i = n j , where the total sample size is N = 2n i ) will be compared on a normally distributed outcome measure for which the groups have a common variance. The between group effect size, Cohen’s d , is estimated as: d = X ¯ 1 − X ¯ 2 s . With equal sample sizes, the 95% CI for d is approximately: d + / − ( 4 / N ) . (Note that the 4 in the numerator of the final term is derived from 2*t (N-2, α/2) .) For example, if the sample effect size is d =. 50 (i.e., the two groups differ by one-half standard deviation unit) and there are 18 participants per group, the 95% CI is 0.50+/− 0.67: −0.17≤ Δ ≤1.17 (i.e., .50 + / − 4 / 36 ). This intervals denotes that the true effect of active relative to placebo ( Δ ) is somewhere between is slightly detrimental (−0.17) to tremendously beneficial (1.17). The corresponding estimates of sample size/group range from as many as 576 to as few as 12. Hence, with imprecision comes a vast disparity in sample size estimates and, if sample size determination for a subsequent study is based on an imprecise estimate, there is an enormous risk of underpowered or overpowered design. In other words, the efficacy data from a pilot study of this size are uninformative. Many pilot studies have far fewer than 18 participants/group and therefore even greater imprecision. We learn little if anything about the efficacy of an intervention with data from a small sample; yet, as discussed earlier, a great deal can be learned from a pilot study.

An Alternative to Using Pilot Data for Sample Size Determination

An alternative approach is to base sample size estimates for a full scale RCT on what is deemed a clinically meaningful effect . For example, the investigator must use clinical experience to describe a clinically meaningful difference on the primary outcome, in the case of MDD trials, the HAMD. How many HAMD units represent a meaningful between treatment group difference? Assume that the pre-post difference on the HAMD total has an sd = 6.0 . Then d = .20 represents 1.2 units of HAMD change, d = .40 represents 2.4 units of HAMD change, and d = .50 represents 3.0 units of HAMD change. The respective sample sizes needed per group for 80% power with two-tailed t-test (alpha=.05) are: 393, 100, and 64. (N/group ≈ 16/ d 2 ; Lehr, 1992 ) The clinical interpretation of HAMD change of 1.2 to 3.0 would drive the choice among possible sample sizes in planning a study. Ideally, a hypothesis testing study should be designed to detect the smallest difference that is generally agreed to be clinically meaningful .

The primary role of a pilot study is to examine the feasibility of a research endeavor. For instance, feasibility of recruitment, randomization, intervention implementation, blinded assessment procedures, and retention can all be examined. Investigators should be forthright in stating these objectives of a pilot study and bravely accept the limitations of a pilot study. Grant reviewers should expect no more.

The choice of the intervention for a pilot study should be based on theory, mechanism of action, a case series, or animal studies that justify a rationale for therapeutic effect. Well-conceived and implemented pilot studies will reduce the risk of several problems that are commonly faced in clinical trials. These include the inability to recruit the proposed sample size and a corresponding reduction in statistical power, excessive attrition due to intolerable procedures or interventions, and the need to modify a protocol midway through a trial. As a consequence, pilot studies can reduce the proportion of failed trials and allow research funds to be spent on projects for which feasibility has been demonstrated and quantified.

Despite the convention, pilot studies do not provide useful information regarding the population effect size because the estimates are quite crude owing to the small sample sizes. Basing a decision to proceed or terminate evaluation of a particular intervention on pilot data is perilous because there is a very good chance that the decision will be derived from false positive or false negative results. In lieu of using pilot results, sample size determination should be derived based on that required for sufficient statistical power to detect a clinically meaningful treatment effect. The definition of clinically meaningful is not entirely empirically-based, but instead requires input from clinicians who treat the patient population of interest and perhaps from patients with the disorder.

By the very nature of pilot studies, there are critical limitations to their role and interpretation. For example, a pilot study is not a hypothesis testing study and therefore safety and efficacy are not evaluated. Further, a study can only examine feasibility of the patient type included in the study. The feasibility results do not necessarily generalize beyond the inclusion and exclusion criteria of the pilot.

In summary, pilot studies are a necessary first step in exploring novel interventions and novel applications of interventions – whether in a new patient population or with a novel delivery system (e.g., transdermal patch). Pilot results inform feasibility, which in turn, is instructive in that it points to modifications needed in the planning and design of a larger efficacy trial.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

• Campden-Main BC, Wegielski Z. The control of deviant behavior in chronically disturbed psychotic patients by the oral administration of reserpine. Ann N Y Acad Sci. 1955 Apr 15; 61 (1):117–122. [ PubMed ] [ Google Scholar ]
• Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials. 3. New York: Springer; 1998. [ Google Scholar ]
• Goller ES. A controlled trial of reserpine in chronic schizophrenia. J Ment Sci Oct. 1960; 106 :1408–1412. [ PubMed ] [ Google Scholar ]
• Jovanovic BD, Levy PS. A look at the rule of three. The American Statistician. 1997; 57 :137–139. [ Google Scholar ]
• Jovanovic BD, Zalenski RJ. Safety evaluation and confidence intervals when the number of observed events is small or zero. Annals of Emergency Medicine. 1997; 30 :301–6. [ PubMed ] [ Google Scholar ]
• Kinross-Wright V. Chlorpromazine and reserpine in the treatment of psychoses. Ann N Y Acad Sci. 1955 Apr 15; 61 (1):174–182. [ PubMed ] [ Google Scholar ]
• Klerman GL. Scientific and ethical considerations in the use of placebo controls in clinical trials in psychopharmacology. Psychopharmacology Bulletin. 1986; 22 :25–29. [ PubMed ] [ Google Scholar ]
• Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavge JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Archives of General Psychiatry. 2006; 63 :484–489. [ PubMed ] [ Google Scholar ]
• Lehr R. Sixteen s-squared over d-squared: A relation for crude sample size estimates. Statistics in Medicine. 1992; 11 :1099–1102. [ PubMed ] [ Google Scholar ]
• Leon AC, Davis LL. Enhancing Clinical Trial Design of Interventions for Posttraumatic Stress Disorder. Journal of Traumatic Stress. 2009; 22 :603–611. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Leon AC, Mallinckrodt CH, Chuang-Stein C, Archibald DG, Archer GE, Chartier K. Attrition in randomized controlled clinical trials: Methodological issues in psychopharmacology. Biological Psychiatry. 2006; 59 :1001–1005. [ PubMed ] [ Google Scholar ]
• National Institute of Mental Health. Pilot Intervention and Services Research Grants (R34) [Accessed October 4, 2010]. http://grants.nih.gov/grants/guide/pa-files/PAR-09-173.html .
• Porta M. A Dictionary of Epidemiology. 5. Oxford: Oxford University Press; 2008. [ Google Scholar ]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 18 May 2024

Research on trajectory control technology for L-shaped horizontal exploration wells in coalbed methane

• Xiugang Liu 1 , 2 , 3 ,
• Zaibing Jiang 1 , 2 , 3 ,
• Yi Wang 3 ,
• Haitao Mo 3 ,
• Haozhe Li 3 &
• Jianlei Guo 3  

Scientific Reports volume  14 , Article number:  11343 ( 2024 ) Cite this article

Metrics details

• Energy science and technology
• Engineering

Horizontal wells have significant advantages in coal bed methane exploration and development blocks. However, its application in new exploration and development blocks could be challenging. Limited geological data, uncertain geological conditions, and the emergence of micro-faults in pre-drilled target coal seams make it hard to accurately control the well trajectory. The well trajectory prior to drilling needs to be optimized to ensure that the drilling trajectory is within the target coal seam and to prevent any reduction in drilling ratio (defined here as the percentage of the drilling trajectory in the entire horizontal section of the well located in the target coal seam) caused by faults. In this study, the well trajectory optimization is achieved by implementing the following process to drill pilot hole, acquire 2D resonance, and azimuthal gamma logging while drilling. The pilot hole drilling can obtain the characteristic parameters of the target coal seam and the top and bottom rock layers in advance, which can provide judgment values for the landing site design and real-time monitoring of whether the wellbore trajectory extends along the target coal seam; 2D resonance exploration can obtain the construction of set orientation before drilling and the development of small faults and formation fluctuations in the horizontal section, which can optimize the well trajectory in advance; the azimuth gamma logging while drilling technology can monitor the layers drilled by the current drill bit in real time, and can provide timely and accurate well trajectory adjustment methods.The horizontal well-Q in the Block-W of the Qinshui Basin was taken as a case study and underwent technical mechanism research and applicability analysis. The implementation of this new innovative process resulted in a successful drilling of a 711 m horizontal section, with a target coal seam drilling rate of 80%. Compared to previous L-type wells, the drilling rate increased by about 20%, and the drilling cycle shortened by 25%. The technical experience gained from this successful case provides valuable insight for low-cost exploration and development of new coalbed methane blocks.

Introduction

Coal Bed Methane (CBM) is found in many parts of the world, and is considered as a clean and abundant source of energy 1 , 2 , 3 . In general, CBM wells mainly include three types; vertical, cluster and horizontal wells. The cluster and horizontal wells belong to directional wells. Moreover, horizontal wells could be further classified into; V-, U- and L-shaped wells. Which in turn could also be divided according to their radius, and branches. Figure  1 below provide an illustration for some of these wells.

Illustration of well types; ( a ) Vertical well, ( b ) Cluster well, ( c ) Horizontal Well, and ( d ) Horizontal L-Shaped well with a vertical well forming a U-Shaped well.

In the development of CBM wells, L-shaped, U-shaped and multi-branch horizontal wells are usually used for new exploration and development blocks (defined here as new fields or area blocks in the oil and gas industry) 4 , 5 , 6 . However, complex formation structure, and small faults development have made it an extremely challenging task to achieve high output from newly developed CBM wells 7 . For instance, U-shaped wells (a well type in which a vertical well and a horizontal well are connected in the same target layer) face huge difficulties in accurate docking along the coal seam and have limited benefits in the presence of multiple faults in the horizontal Section 8 . Similarly, the applicability of multi-branch horizontal wells is poor, especially in complex stratigraphic structures and fault development of the block 9 .

On the other hand, L‑shaped horizontal wells are often adopted as the main type of wells for exploring and developing CBM in new blocks. The L-shaped horizontal wells exhibit uncomplicated drilling prerequisites, demonstrate a low probability of wellbore collapse or obstruction, and facilitate subsequent access for maintenance of the initial wellbore 10 . However, the drilling process of these wells are not free of challenges. L-shaped wells have a high requirement for wellbore trajectory control, and they are usually difficult to achieve one-time “soft landing” and ultra-long horizontal segment footage 10 . In addition, drainage equipment and method are another key restriction for the promotion and application of this type of well 11 . For example, reported completion data from several exploration wells indicated that the drilling ratio along the coal seam of the actual trajectory is less than 60%. The drilling cycle is nearly two months, and gas production is low 11 . Table 1 illustrates a tabulated analysis of the applicability and challenges associated with different well types in exploration blocks characterized by complex geological formations and the presence of micro-faults.

Various methods have been used to improve the drilling ratio, by improving the trajectory control. These methods, shown in Table 2 , include: geological guidance technology of adjacent well data, electromagnetic waves, natural gamma measurement, and three-dimensional seismic exploration technology. However, each method has its own limitations, such as high costs, difficulty in obtaining gamma values in specific directions, and signal loss when applied to drilling in complex formations 12 , 13 .

This study delves into trajectory control methods for Horizontal wells within Coalbed Methane (CBM) exploration and development blocks. The approach involves the utilization of pilot holes to determine the characteristics of the target coal seam and the surrounding upper and lower rock layers based on the magnitude of gamma values. This information serves as a predictive identification of marker layers, allowing real-time control and adjustment of the drilling trajectory within the target coal seam. This methodology enables the identification of whether the drilling trajectory is presently positioned within the target coal seam, the roof rock layer, or the floor rock layer. Additionally, a two-dimensional resonance exploration technology is employed for geological structure and fault detection prior to drilling, enabling pre-drilling trajectory optimization. Furthermore, azimuth gamma logging technology is utilized for real-time monitoring and correction of the drilling trajectory's horizontal positioning during the drilling process. Using L-shaped Short-Radius Well-Q in Block-W of the Qinshui Basin as a case study, a comprehensive assessment of the combined effectiveness of these three methods is conducted. Simultaneously, the research delves into the technical mechanisms and applicability analysis. This exploration of the technical mechanisms aims to enhance the understanding of the functions of these methods, their application conditions, and the analysis and utilization of their technical effects.

Trajectory control methodology

Pilot hole drilling, construction background and reasons.

The area formation structure and faults nature could be obtained by two-dimensional seismic data. Seismic surveys and exploratory drilling in the area could provide a good indication on the coal seam actual depth, coal seam distribution, layers, belts and interbeds. For the geological conditions of developing new blocks, such as less drilling data, less seismic exploration data, complex formation structure and micro-fault development, etc., before drilling, it is imperative to obtain the key parameters of the target coal seam, including its lithology, gas-bearing capacity, gamma value, etc., along with those of the rock layers above and below it. This will allow for the determination of the precise horizon of the coal seam and provide technical support for real-time monitoring and well trajectory control along the target coal seam. To achieve this, it is necessary to design and implement a pilot hole drilling program to obtain the characteristic parameters of the target coal seam and the surrounding strata 14 , 15 .

Pilot hole construction design

Once the goal of layer identification is achieved, the next step is to backfill and sidetrack the pilot hole to open branches and land according to the actual occurrence of the coal seam. To ensure the effectiveness of the pilot hole guidance in subsequent construction, it is advisable to minimize the distance between the coal-seem top point (the point where the drilling trajectory first drills into the target coal seam) and the landing point by increasing the well angle of inclination. Conversely, in order to enhance the construction efficiency of the pilot hole, it is preferable to keep the depth of the pilot hole to a minimum, which is indicated by a small well angle of inclination (70 degrees). Figure  2 illustrates this concept.

Optimization of pilot hole scheme.

Taking into account the underlying reasons and background for constructing a pilot hole, as well as the difficulty of side-tracking and the efficiency of construction, a comprehensive plan has been developed. The plan involves drilling the pilot hole at a steady angle of approximately 70° until the bottom of the target coal seam is reached.

• Two-dimensional resonance exploration

Resonance exploration mechanism

The seismic wave frequency resonance exploration technology is a novel geophysical exploration method that utilizes the frequency resonance principle prevalent in nature to investigate underground geological formations 16 , 17 , 18 , 19 . This technique enables the acquisition of geometric attributes of subsurface structures, such as fractures and faults. Figure  3 illustrates a typical resonance diagram of a seismic wave.

( a ) Typical resonance curve of seismic wave ( b ) self-excite resonance to vibration.

Resonance exploration technology boasts numerous advantages, including high sensitivity to density changes, exceptional vertical and horizontal resolution, and an exploration depth of up to 5000 m. Additionally, this technology can be acquired and processed passively, making it an economical and straightforward exploration method 20 .

Analysis of technical applicability

At this stage, the analysis of the existing two-dimensional seismic data in the exploration block would indicate the geological structure of the target coal seam in the block. In addition, it will reveal fault’s locations beside faults development status. The pilot hole drilling can accurately obtain the actual depth of the target coal seam and the characteristic parameter values of the target layer, as well as the roof and floor, but conventional means cannot predict structural conditions such as the development of micro faults in the horizontal section of the drilling along the designated direction. This increases the difficulty of well trajectory control and makes it challenging to ensure the coal seam drilling ratio. However, the two-dimensional resonance exploration technology can be used to infer the development of small faults in the horizontal section drilled along the specified direction by interpreting the resonance image. This enables the optimization of the well trajectory in advance to control the actual drilling trajectory and improve the drilling rate of the target coal seam.

Azimuth gamma control technology

Working principle of azimuth gamma.

The azimuth gamma logging tool is utilized to measure the width of gamma ray energy level 21 , 22 , 23 . The scintillation counter captures gamma rays from the stratum, and azimuth gamma logging while drilling offers unique advantages 24 , 25 . Firstly, it enables real-time calculation of the strata's apparent dip angle. It is convenient to calculate the apparent dip angle of the strata by utilizing the azimuth gamma data. The apparent dip angle at the current position can be obtained as long as it is required to cross an interface. The formula for calculating the apparent dip angle using the azimuth gamma 26 is as follows:

where α is the apparent strata dip; D is the well diameter; Δd is the distance between the upper and lower gamma value change points; β is the well deviation angle.

Second, measuring the natural gamma value in a specific direction. By transmitting up and down gamma data in real-time, it becomes possible to accurately determine the positions of different formation interfaces 27 , 28 . This information can then be used to ensure that the trajectory of the control well is precisely aligned with the target coal seam after drilling is complete. The specific process involved is illustrated in Fig.  4 .

Trajectory control based on azimuth-while-drilling gamma logging. ( a ) Coal seam drilled out from the roof. ( b ) Coal seam drilled out from the floor.

The drilling process in the horizontal section along the coal seam is susceptible to deviate from the target due to increased drilling pressure or the impact of the formation structure. The strata above and below the coal seam are usually mudstone or carbonaceous mudstone. When using azimuth gamma logging during drilling, the upper gamma value first increases, followed by the lower gamma value, indicating that the drilling has exited the coal seam roof at point C in Fig.  4 a. When the upper and lower gamma values become similar, it suggests that the drilling has left the layer, as shown at point D in Fig.  4 a. To correct the inclined drilling control track deviation, the trajectory correction process is initiated when drilling to point C using azimuth gamma measurement, as demonstrated at point E in Fig.  4 a. Similarly, when the lower gamma value increases first and the upper gamma value increases later, it indicates that the drilling trajectory is exiting the coal seam floor at point C1 in Fig.  4 b. When the upper and lower gamma values become similar, the drilling has left the layer, as shown at point D1 in Fig.  4 b. To correct the incremental drilling control track deviation, the trajectory correction process is initiated when drilling to point C1, as illustrated at point E1 in Fig.  4 b.

In terms of technical applicability, conventional natural single gamma logging technology cannot accurately determine the bit's position once it leaves the coal seam, making it challenging to provide precise corrective measures. This issue is particularly problematic wherever the geological structure of the target coal seam is complex, micro faults are developed, and the coal seam is thin. To ensure the penetration ratio of the target coal seam and ensure the safety of underground construction, azimuth gamma logging while drilling technology can be utilized. This technology allows for the real-time monitoring of the current drilling horizon and provides effective guidance during construction. As a result, the drill bit can efficiently drill into the coal seam, maximizing the penetration ratio of the target coal seam.

Technical applicability analysis

In the second drilling operation, if the targeted coal seam is complex due to its thinness or the presence of micro-faults, it will be very challenging to accurately determine the position of the drilling bit after it exits the coal seam. Therefore, it will be necessary to use azimuth gamma logging while drilling. This technology enables the real-time monitoring of the drilling bit's current horizon, guiding the construction process and ensuring that the bit drills to the maximum extent possible within the coal seam.

Trajectory control technology and case study

Geological setting.

In this study, the short radius, well-Q in Block-W of the Qinshui Basin is taken as an example. Based on the most recent exploration wells drilled in Block-W of Qinshui Basin, the geological horizons have been revealed. The strata in the block, from bottom to top, consist of Paleozoic Ordovician, Carboniferous, Permian, Mesozoic Triassic, Jurassic, and Cenozoic Quaternary. The stratum near Well-Q has a general inclination from northeast to northwest, and Coal Seam no.15 is the development target stratum. The coal seam is located in the lower part of the Taiyuan Formation and has a simple structure. It is a thick coal seam that is stable and easy to drill throughout the area and generally contains 0–2 layers of dirt shale. The effective thickness of the coal seam ranges from 0 to 5.30 m, with an average of 3.39 m. It is thicker in the east and thinner in the west. However, there is one exploration well in the block that did not drill into Coal Seam no.15, possibly due to fault interference resulting in the loss of the coal seam. The coal seam deposit depth ranges from 728 to 2002 m, with an average of 1479 m. The depth is shallow in the southeast of the block and gradually deepens towards the northwest. Due to the influence of the stratum tendency (Stratum dip), the depth of the coal seam reaches over 1500 m in the west 14 . The roof lithology of the coal seam mostly consists of sandy mudstone, mudstone, siltstone, and fine sandstone, while the floor is mostly sandy mudstone, mudstone, and siltstone.

Wellbore structure

Designing an optimized wellbore structure can greatly improve drilling efficiency and safety by reducing annular pressure loss and back pressure (the drilling tool back pressure phenomenon), especially for long well sections. In the case of Well-Q, the wellbore structure was designed with a three-opening sections to ensure gas production of the coal seam during subsequent fracturing development. The first section seals the formation prone to collapse and leakage in the upper part of the primary casing, creating a safe drilling environment for the second well section. The second section seals sandstone, mudstone, and sandy mudstone intervals at the upper part of the coal seam, with the second well section casing obliquely drilled to a depth of no less than 3 m from the target coal seam no.15.

The third section extends along coal seam no.15 and runs casing to form a stable gas production channel to prevent coal seam collapse in the horizontal section due to the influence of multiple factors such as fracturing in the later stage. Prior to drilling the second well section of the main borehole, pilot hole drilling was carried out to obtain relevant geological parameter information of the target coal seam and the adjacent marker bed. Specific design parameters and requirements are as follows:

In the first well section, a ø 346.1 mm drill bit was used to drill into the stable bedrock for 30 m. J55 grade steel ø 273.1 mm surface casing was then lowered and cementing cement slurry returned to the surface.

In the second well section, a ø 241.3 mm drill bit was used to drill to the roof of the target no.15 coal seam and then the drilling was stopped. The landing point was determined based on the lithology of the roof of the coal seam and the actual drilling process. N80 grade steel ø 193.7 mm technical casing was run to 3–5 m above the roof of the coal seam. Through variable density cementing process, high-density cement slurry was used to return to 300 m above the roof of Coal Seam no.15, while low-density cement slurry returned to the surface.

The third well section was drilled with a ø 171.5 mm drill bit. After entering the target coal seam no.15, the drilling followed the coal seam. Upon reaching the designed well depth, P110 grade steel ø 139.7 mm production casing was run, and the well was completed without cementing.

The pilot hole was drilled with a ø215.9 mm bit, and the inclination angle stabilizing drilling crossed the floor of the target coal seam for tens of meters. Subsequently, the bit was backfilled with pure cement slurry to the side drilling depth of the second well section. The specific wellbore structure is shown in Fig.  5 .

Well structure.

Case study: well-Q design optimization

Using Well-Q as a case study, the pilot hole trajectory design included the following: straight well section, kicking-off section, and stabilizing section. The stabilizing drilling passes through the floor of Coal Seam no.15 for approximately 30 m at an inclination angle of 70° to ensure accurate measurement of the gamma value, gas measurement value, and other characteristic parameters of the target coal seam bottom and floor using a simple gesturing instrument. The pilot hole is sealed by backfilling it with 42.5 grade Portland cement up to the well section with an inclination of about 25°, and the cement slurry has a specific gravity of 1.6–1.7 g/cm3. As the well deviation angle increases, the azimuth angle of directional and composite drilling becomes more stable, particularly when the well deviation angle exceeds 25°, resulting in a smaller azimuth drift 29 . This stability is beneficial for the subsequent inclined side-tracking in the main wellbore's second well section. The pilot hole and main borehole design trajectories are shown in Fig.  6 .

Design trajectory of pilot hole and main hole.

Significant data has been obtained through the pilot hole design and the actual drilling of Well-Q. This dataset is pivotal for precise trajectory control in Coalbed Methane (CBM) exploration. The acquisition process relies on several methods, including real-time drilling natural gamma logging for gamma values of marker layers, and downhole gas logging for coal seam gas characteristics. The examination of cuttings recorded in real-time during drilling operations further aids in the identification and differentiation of these marker layers.

The critical information gleaned encompasses the identification of the K2 marker bed, the longitudinal stratification of the target no.15 coal seam, as well as the lithological composition, gamma values, and gas-bearing attributes of the upper and lower rock layers. These specific parameters are thoughtfully presented in Fig.  7 , establishing a robust foundation for the meticulous control of trajectory and the rational design of the landing point within the target coal seam. This dataset also serves as a valuable point of reference, ensuring the seamless execution of the horizontal drilling phase within the coal seam. Consequently, these findings play a pivotal role in enhancing drilling efficiency, ultimately culminating in the realization of efficient drilling objectives.

Characteristic parameters and lithology map of the marker layer, target, top, bottom layer.

The effect of two-dimensional resonance method

The horizontal section's overall drilling azimuth in the target coal seam is 200°. To identify minor faults in the coal seam azimuth direction, measurement points are arranged every 10 m from the landing point A to the final target point B along the 200° azimuth direction. Additionally, one exploration point is set every 20 m across the azimuth line perpendicular to the landing point A and 200° azimuth direction. Furthermore, exploration points are arranged 300 m along both sides of the landing point. Figure  8 shows the specific layout of the exploration points, where Line (L1) represents the 711 m long horizontal well section of the target coal seam in the 200° azimuth direction. Meanwhile, Line (L2) represents the 600 m long vertical section between the landing point A and L1. The obtained data from these exploration points are crucial in detecting potential faults and ensuring smooth drilling of the horizontal section of the coal seam. ultimately leading to improved drilling ratios and more efficient drilling.

Two-dimensional resonance exploration layout points.

Figure  9 shows the seismic frequency resonance inversion profile. The trajectory of the designed horizontal section coincides with the ground position of L1, with the no.4700 measuring point located at the ground projection position of the A target point, and the no.4000 measuring point located at the ground projection position of the B target point. Based on the interpretation of seismic frequency resonance line L1 profile, it is observed that the burial depth of the coal seam on the horizontal well section from target A to target B of the no.15 coal seam in the direction of 200° azimuth is shallow in the northeast and deep in the southwest. The overall trend of the burial depth of the coal seam indicates a shallow-to-deep trend. Furthermore, three small faults are expected to be encountered while drilling along this azimuth direction, located at no.4700, no.4280 and no.4096 measuring points, respectively, with a fault distance of approximately 5–10 m.

Design of horizontal section trajectory resonance exploration inversion profile.

The contour map of fault points found in the horizontal section is displayed in Fig.  10 . This map serves as a useful tool in guiding the vertical depth control of the horizontal section track.

Contour map of fault points in the horizontal section.

To ensure that the drilling trajectory is within the target coal seam and to prevent any reduction in drilling ratio caused by the faults, it is necessary to optimize the well trajectory prior to drilling. Each fault point must be considered as a target point and their relative coordinate positions are presented in Table 3 .

Resonance exploration data is utilized to adjust the trajectory parameters every 10 to 20 m during the actual drilling process. This is before exploring the coal seam behind the fault following reasonable adjustment of the parameters. This method is simple and minimizes the length of the non-coal section during the coal chasing process after drilling through the fault. Based on the coordinate position of each target point, the design of the directional trajectory for the third well section is optimized, as shown in Fig.  11 .

optimized well trajectory for drilling reservoir section. ( a ) vertical section, ( b ) horizontal projection section.

The optimized design trajectory should be followed during actual drilling, ensuring that the dogleg degree ≤ 4°/30 m required by the management method for safe operations. Across the fault points F1, F2, and F3, the length of the non-coal section for coal tracking drilling was 56 m, 53 m, and 35 m, respectively. The total non-coal section for actual drilling was approximately 144 m, while achieving a drilling ratio of 80% for the target coal seam with an average thickness of 2.06 m. The entire drilling cycle takes approximately 45 days.

Azimuth gamma application

By analyzing the azimuth gamma data obtained during the drilling of the pilot hole and using the basic parameters of the pilot hole and formula ( 1 ), the apparent dip angle of the stratum near the designed landing point is determined to be α = 6.5°. The parameters of the landing point are shown in Fig.  12 , and the deviation angle of the actual main borehole trajectory of the second well section at the landing point β should be controlled at around 83.5° to ensure that the drilling ratio along the coal seam of the third well section is achieved and to reduce the frequency of directional trajectory adjustment.

Parameters of the landing site.

During the drilling of the third horizontal section of Well-Q, a combination of Two-dimensional resonance exploration results and azimuth gamma logging while drilling technology was used to guide rapid coal tracking during the drilling of three faults. The process for each fault was as follows:

F1 Fault: The logging curve in Fig.  13 indicates that the F1 fault caused the drilling track of the 1920–1976 m well section to be drilled out from the coal seam roof. Geological logging revealed that the rock debris returning out of the hole bottom contained a large amount of mudstone. Based on the Two-dimensional resonance exploration inversion (Fig.  9 ) and fault contour (Fig.  10 ), the coal seam was traced by drilling with deviation correction through the lowering of well deviation. The actual drilling track during the pursuit of coal process is shown in Fig.  14 .

Non-coal seam section azimuth gamma logging curve crossing fault F 1 .

Actual drilling trajectory of fault F 1 in pursuit coal.

F2 Fault: The logging curve in Fig.  15 shows that the F2 fault caused the drilling trajectory of the 2130–2183 m well section to be drilled out from the coal seam roof. Geological logging revealed that the rock debris returning out of the hole bottom contained a large amount of mudstone. Based on the Two-dimensional resonance exploration inversion (Fig.  9 ), the back fault block of F2 fault in the direction of drilling trajectory of F2 fault shows a tendency of coal seam incline, so directly using lowering deviation correction drilling to trace the coal seam is not feasible and increases the length of the non-coal seam section. Therefore, the coal seam was pursued by increasing well deviation and rectifying drilling. The actual drilling track during the pursuit of coal process is shown in Fig.  16 .

Non-coal seam section azimuth gamma logging curve crossing fault F 2 .

Actual drilling trajectory of fault F 2 in pursuit coal.

F3 Fault: The logging curve in Fig.  17 shows that the F3 fault caused the drilling trajectory of the 2315–2350 m well section to be drilled out from the coal seam roof floor. Geological logging revealed that the rock debris returning out of the hole bottom contained a large amount of carbonaceous mudstone. Using formula ( 1 ), the coal point well inclination angle was calculated as 96°. Based on the Two-dimensional resonance exploration inversion (Fig.  9 ) and fault contour (Fig.  10 ), the coal seam was pursued by slowly lowering the well inclination and correcting the deviation. The actual drilling track during the pursuit of coal process is shown in Fig.  18 . The well inclination angle was 91° upon returning back to the coal seam, after which drilling along the coal seam was continued normally.

Non-coal seam section azimuth gamma logging curve crossing fault F 3 .

Actual drilling trajectory of fault F 3 in pursuit coal.

In conclusion, for the exploration block of CBM, the combined use of pilot hole drilling, two-dimensional resonance exploration technology, and azimuth gamma logging technology has proven effective in controlling the drilling of short-radius horizontal sections along the seam and ensuring the coal seam drilling ratio. Two major points can be drawn from this:

The two-dimensional resonance exploration technology detected the development of micro faults in the horizontal section of the drilling, enabling trajectory optimization before drilling. The azimuth gamma logging while drilling technology monitored the current drill bit drilling horizon in real-time, ensuring timely and accurate well trajectory adjustment.

The comprehensive use of these technologies has led to a 20% improvement in the coal seam drilling ratio and a 25% reduction in drilling cycle time in tested short-radius wells in the new exploration and development block-W in Qinshui Basin. This provides technical experience for low-cost exploration and development of CBM in new blocks.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Sun, W. L., Chen, Z. Y., Chen, X., Wang, S. H. & Fu, X. Y. Geological features and resource potentials of coalbed methane basins in China. Oil Gas Geol. 26 (2), 141–146 (2005).

Google Scholar  

Qin, Y. Evaluation and production technology of coalbed methane reservoir. China University of Mining and Technology Press, (1996).

Men, X. Y., Han, Z., Gong, H. J. & Wang, X. Y. Challenges and opportunities of CBM exploration and development in China under new situations. Nat. Gas. Ind. 38 (09), 10–16 (2018).

Zhang, P. Y., Sun, J. M. & Cheng, Z. G. Application of azimuthal gamma ray imaging logging while drilling to geosteering in horizontal wells of H area, Ordos Basin. Sci. Technol. Eng. 21 (23), 9713–9724 (2021).

Dai, Y. J., Li, S. Q., Xia, L. Y., Li, J. X. & Lv, Y. A CBM development well type optimization method based on the long-run marginal cost. Nat. Gas. Ind. 38 (07), 113–119 (2018).

CAS   Google Scholar  

Liu, Y. K., Wang, F. J., Tang, H. M. & Liang, S. Well type and pattern optimization method based on fine numerical simulation in coalbed methane reservoir. Environ. Earth Sci. 73 (10), 5877–5890 (2015).

Article   ADS   CAS   Google Scholar  

Jia, H. M., Hu, Q. J., Fan, B., Mao, C. H. & Zhang, Q. Causes for low CBM production of vertical wells and efficient development technology in northern Zhengzhuang Block in Qinshui Basin. Coal Geol. Explor. 49 (2), 34–42 (2021).

Liu, C. C., Jia, H. M., Mao, S. F., Cui, X. R. & Peng, H. The development characteristics and main control factors of the open-hole multi-branch CBM horizontal wells. Coal Geol. Explor. 46 (5), 140–145 (2018).

Huang, W. et al. Construction technologies and stimulation of U-shape well for CBM development—with 2014ZX-U-05V/H well of coal 15 in SiHe mine as an example. Coal Geol. Explor. 43 (6), 133–136 (2015).

Hu, Q. J. et al. Discussion of the geological adaptability of coal-bed methane horizontal wells of high-rank coal formation in southern Qinshui Basin. J. China Coal Soc. 44 (4), 1178–1187 (2019).

Liu, C. C., Jia, H. M. & Mao, S. F. The development characteristics and main control factors of the open-hole multi-branch CBM horizontal wells. Coal Geol. Explor. 46 (5), 140–145 (2018).

Wang, L., Li, L., Sheng, L. M., Dou, X. R. & Zhang, L. C. Electromagnetic wave DREMWD system and its field test. Oil Drill Prod. Technol. 35 (02), 20–23 (2013).

Pang, Q., Feng, Q. H., Ma, Y., Zhang, Y. Y. & Peng, X. H. The application of three-dimensional geological modeling technology in horizontal well geologic steering: A case from X3–8 horizontal well development zone. Nat. Gas Geosci. 28 (3), 473–478 (2017).

Song, H. B. et al. Controlling geological factors and coalbed methane enrichment areas in Southern Wuxiang Block, Qinshui Basin. J. China Coal Soc. 46 (12), 3974–3987 (2019).

Liu, C. H., Liu, S. C., Yan, S., Liu, Y. & Su, L. Application of integrated geophysical exploration techniques to detecting shallow coal gob. Chin. J. Eng. Geophys. 8 (1), 51–54 (2021).

Zhang, Q. Key technologies for drilling and completion of No.15 coal L-shaped horizontal well in Zhengzhuang block. Qinshui Basin. Coal Eng. 53 (11), 61–66 (2021).

Xue, A. M., Li, D., Song, H. X. & Zhang, A. J. Image the earth with the frequency resonance effect of vibration noise. Geol. Rev. 65 (supplement1), 47–48 (2021).

Li, H. et al. Application of shallow seismic exploration combining mixed source surface waves and three-component frequency resonance method in fine detection of urban shallow geological structure. Prog. Geophys. 35 (3), 1149–1155 (2020).

Liu, X. G., Li, J. F., Zhang, Q. & Zhang, J. Practice of accurate control technology for multi-branch horizontal grouting well trajectory of coal seam floor limestone reinforcement in Zhaogu No.1 Mine. Saf. Coal Mines 52 (11), 100–103 (2021).

Zhu, C. C. & Li, H. Application of seismic frequency resonance technique in goaf detection of heavy-cover coal seams. Chin. J. Eng. Geophys. 18 (5), 774–779 (2021).

Du, Z. Q., Hao, Y. L., Zhang, G. L., Yang, Z. B. & Lu, D. The application of the azimuth gamma logging while drilling for the geosteering in the horizontal wells in Jidong Oil field. Mud. Logging Eng. 19 (1), 18–21 (2008).

Tang, H. Q. Image processing method of LWD azimuthal gamma data. Lithol. Reserv. 29 (1), 110–115 (2017).

Zheng, Y. T., Fang, F., Wu, J. P., Li, J. B. & Zhang, W. Development and application of near-bit gamma-ray imaging system during drilling. J. Northeast Pet. Univ. 44 (3), 70–76 (2020).

Liu, X. P., Fang, J. & Jin, Y. H. Application status and prospect of LWD data transmission technology. Well Logging Technol. 32 (3), 249–253 (2008).

Sun, D. J. & Sun, L. Application of geosteering technology in construction of CBM horizontal well. Coal Geol. Explor. 43 (02), 106–108 (2015).

Zhang, J. Q. et al. Application of comprehensive geophysical prospecting method in detecting goaf of thick overburden coal mine. Geol. Rev. 65 (supplement1), 52–54 (2021).

Wu, C. L. Application of azimuth gamma in coal bed methane horizontal wells. J. Drill. Eng. 48 (5), 69–75 (2021).

Chen, G., Wang, K. B., Jiang, B. C. & Wang, X. L. Comparison and application of LWD lithology identification method. Coal Geol. Explor. 46 (01), 165–169 (2018).

Liu, H. B., Fan, Z. X. & Gao, M. Study on decreasing the azimuth drift in the directional well. Fault-Block Oil Gas Field. 2 (10), 80–82 (2003).

Download references

Acknowledgements

The financial support by the Found of the National Key Research and Development Program and Key Special Fund Project (No.2018YFC0808202) are gratefully acknowledged.

Author information

Authors and affiliations.

China University of Mining and Technology, Beijing, 100083, China

Xiugang Liu & Zaibing Jiang

China Coal Research Institute, Beijing, 100013, China

Xi’an Research Institute Co. Ltd., China Coal Technology and Engineering Group Corp., Xi’an, 710077, China

Xiugang Liu, Zaibing Jiang, Yi Wang, Haitao Mo, Haozhe Li & Jianlei Guo

You can also search for this author in PubMed   Google Scholar

Contributions

X.L. conceived the study and, together with Z.J., Y.W., and H.M. did the literature search, selected the studies. X.L. and H.L. extracted the relevant information. X.L. synthesised the data. J.G. drawed pictures. X.L.and Z.J.wrote the first drafts of the paper.Y.W.and H.M.critically revised successive drafs of the paper. All authors approved the final drafts of the manuscript. X.L. is the guarantor of the study.

Corresponding author

Correspondence to Xiugang Liu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Liu, X., Jiang, Z., Wang, Y. et al. Research on trajectory control technology for L-shaped horizontal exploration wells in coalbed methane. Sci Rep 14 , 11343 (2024). https://doi.org/10.1038/s41598-024-60550-4

Download citation

Received : 17 January 2024

Accepted : 24 April 2024

Published : 18 May 2024

DOI : https://doi.org/10.1038/s41598-024-60550-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Coalbed methane (CBM)
• Short-radius wells
• Trajectory control
• Azimuth gamma logging while drilling (LWD)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.