writing mixed methods research questions

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

Prevent plagiarism. Run a free check.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved August 12, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, what is your plagiarism score.

Integrations

What's new?

In-Product Prompts

Participant Management

Interview Studies

Prototype Testing

Card Sorting

Tree Testing

Live Website Testing

Automated Reports

Templates Gallery

Choose from our library of pre-built mazes to copy, customize, and share with your own users

Browse all templates

Financial Services

Tech & Software

Product Designers

Product Managers

User Researchers

By use case

Concept & Idea Validation

Wireframe & Usability Test

Content & Copy Testing

Feedback & Satisfaction

Content Hub

Educational resources for product, research and design teams

Explore all resources

Question Bank

Research Maturity Model

Guides & Reports

Help Center

Future of User Research Report

The Optimal Path Podcast

User Research

Aug 15, 2024 • 13 minutes read

Mixed methods research explained: Combine data like a pro

From heatmaps to interviews, here’s how to blend qualitative and quantitative data for holistic user insights.

Ella Webber

Mixed methods research is one of the most popular and powerful UX research approaches—blending numbers with narrative to garner a holistic understanding of your product or research question.

Whether you’re in UX research and design, education, healthcare, or social sciences, mixed methods research can help you find insights and make better decisions.

Read on for a breakdown of what mixed methods are, their strengths and weaknesses, when to use them, and how to analyze the data.

UX research made easy

Explore the power of combining quantitative and qualitative research to discover new insights and test final solutions.

What is mixed methods research and when should you use it?

Mixed methods research involves collecting, analyzing, and integrating both quantitative and qualitative UX research methods within a single study. It is unique to other UX research techniques in that it combines data types, encouraging product teams to use qualitative feedback to explain the story behind quantitative numbers.

• Quantitative data can come from UX surveys , product analytics , usability testing , experiments, or statistical databases and provide broad numerical insights
• Qualitative data is gathered through user interviews , focus groups, or contextual inquiries and offers a deep, contextual understanding of the subject matter

Why use a mixed methods approach?

The power of mixed methods research is simple: it allows you to combine the best parts of both types of data—quantitative research methods, like surveys, give you broad trends, while qualitative methods, such as interviews, dig deep into personal experiences.

Anthony J. Onwuegbuzie and R. Burke Johnson, in Mixed Methods Research: A Research Paradigm Whose Time Has Come , highlight how blending these methods allows researchers to leverage the strengths of both approaches. They identify mixed methods research as one of the “three core research paradigms: qualitative, quantitative, and mixed methods.”

Like any technique, however, mixed methods research has both strengths and weaknesses to consider.

When should you use mixed methods research?

Mixed UX research methods are useful when neither qualitative nor quantitative data alone can fully answer your research question . Evaluative research further helps to assess the effectiveness of your mixed method research findings and ensure they meet user needs.

For example, use mixed methods research when:

• You need to go beyond numbers (generalizability): Quantitative methods, like surveys, provide broad trends and patterns that are relevant to a wider population. For example, a survey might show that most users enjoy a new app feature, but it won’t capture why some users might dislike it.
• The why matters (contextualization): Mixed methods allow you to put numerical findings in context, adding rich detail to your conclusions. For example, if analytics show that users are spending less time on your app (quantitative), interviews can help you understand the reasons behind this behavior, such as frustration with a recent update or a lack of engaging content (qualitative).
• Credibility is important (credibility and triangulation): When both data types converge on the same conclusion, it strengthens your findings. For example, the combined evidence is more credible if survey data indicates that most users prefer a particular software interface and focus groups echo this preference.
• You need to track changes (developmental purposes): Mixed methods are invaluable when one type of data informs the other. For example, initial qualitative research with a small group of beta testers can uncover key issues and user needs, which can then be explored quantitatively with a larger user base to see how widespread these issues are.
• Understand complex issues (complementary insights): Different data types can offer complementary insights. For example, in a study on software usability, quantitative data might show a drop in task completion rates, while qualitative data reveals specific pain points and user frustrations. This combined approach can guide more effective design improvements.

What are the types of mixed methods research design?

The type of mixed methods research design you choose depends on your research goals, the timing of data collection, and each data type. Here are some key factors to consider:

• Your research approach: Are you trying to understand existing findings (explanatory) or dig deeper into a topic (exploratory)?
• Your research questions: Do your questions need big-picture answers (like how many users are happy) and detailed explanations (like why some users are unhappy)?
• Existing data availability: Is there any existing information you can use from previous studies or a research repository (like user demographics)?
• Data you can collect yourself: What kind of in-depth information do you need to gather from users (through interviews, testing, etc.)?

Whether you're a data diver or a narrative novelist, understanding these research methods can make your studies more dynamic and insightful.

📚 A UX research repository is crucial for keeping track of research findings. You need a centralized database to store and manage all your qualitative and quantitative data. This ensures that your research is organized, accessible, and reusable for future studies.

Let’s look at the most common types of mixed methods research design:

Convergent parallel

Convergent parallel design involves collecting qualitative and quantitative data simultaneously but analyzing them separately. The primary goal is to merge the two datasets to provide a complete understanding of the research problem.

For example, let’s say you want to study user satisfaction with a new mobile app. Here’s how you might use the convergent parallel design:

• Qualitative results: Conduct in-depth user interviews with 30 participants to gather detailed insights into their experiences and perceptions of the app. Plus, analyze 200 user reviews from app stores. You might use prompts like, "What features do you find most valuable?" and "Please describe any difficulties you've experienced while using the app."
• Quantitative study: Use analytics data to measure user engagement metrics like session duration and feature usage, then distribute UX surveys to gather quantitative satisfaction scores.

Concurrent embedded design

Embedded design is a mixed methods research approach where qualitative and quantitative data are collected simultaneously, but one type of data is supplementary to the other.

The secondary data provides additional context and can help explain or clarify the primary findings. This approach is particularly beneficial when time or resources are limited, as it allows for a more comprehensive analysis without doubling the workload.

Explanatory sequential design

Explanatory sequential design is a popular mixed methods research approach introduced by John W. Creswell and Vicki L. Plano Clark. This research design involves collecting and analyzing quantitative data first, followed by qualitative data collection and analysis.

According to Creswell, this approach is particularly useful when researchers need to explain relationships found in quantitative data.

The process typically involves two phases:

• Quantitative phase: This involves collecting numerical data through methods like surveys or experiments. The goal here is to identify patterns, trends, or relationships.
• Qualitative phase: Qualitative phase: After analyzing the quantitative data, researchers collect qualitative data with qualitative approaches, like interviews or focus groups, to provide deeper insights. This phase helps explain the ‘why’ or ‘how’ behind the quantitative findings.

Creswell emphasizes that one of the strengths of this design is its simple structure, making it easy for researchers to manage and for audiences to understand the research process and findings.

Exploratory sequential design

Exploratory sequential design begins with qualitative data collection and analysis, followed by quantitative data collection. This immersive approach helps generate rich, detailed data that lays a strong foundation for the subsequent quantitative analysis.

For example, let’s say a researcher wants to understand why people don't meditate regularly. They could start with generative research techniques , like conducting workshops where participants discuss their daily routines and barriers to meditation. These qualitative insights reveal underlying themes and patterns, like time constraints and lack of motivation.

Next, the researcher analyzes these qualitative data to identify key factors impacting wellbeing habits. Based on these insights, they develop a survey to quantitatively measure how widespread these barriers are among a larger population.

So, that’s how you collect data. But how do you analyze it? Unsurprisingly, there are multiple analysis and interpretation methods commonly used in mixed methods research. Let’s look at some.

How to analyze mixed methods research data: 3 Ways to combine qualitative and quantitative data

Combining different types of research data can add credibility to your research findings. Let’s look at how to conduct mixed methods research:

Triangulation protocol

Following a thread, mixed methods matrix.

The triangulation protocol in mixed methods research is a systematic way to use multiple data sources, techniques, or perspectives to get a clear understanding of a research problem. The goal is to capitalize on the strengths of both types of data while minimizing their individual weaknesses.

Let's say you want to conduct a study aiming to evaluate the effectiveness of a new educational program on student performance, and you arrive at the following datasets:

• Quantitative finding: 80% of students improved their math scores after the program
• Qualitative finding: Students reported that interactive activities helped them understand math concepts better

When you merge these findings, the research concludes that the interactive activities (identified qualitatively) are likely a significant factor contributing to the improved scores (quantitatively).

The following a thread method allows researchers to trace a specific theme or concept across both qualitative and quantitative data sets.

Here’s how it works:

• Identify key themes: Begin by identifying key themes or variables that are central to your research questions. These themes will serve as the ‘threads’ you’ll follow through your data.
• Extracting data: Extract relevant data segments related to each theme from qualitative (e.g. interviews, focus groups) and quantitative (e.g. surveys, statistical data) sources. This involves coding qualitative data and identifying relevant quantitative measures.
• Mapping data: Create a map or matrix that links data segments from different sources according to the identified themes. This matrix helps visualize how different data points converge or diverge on the same theme.
• Comparative analysis: Compare the data segments within each theme to identify patterns, consistencies, and discrepancies. Look for how qualitative narratives support or contradict quantitative findings.
• Synthesis and interpretation: Synthesize the findings to develop an understanding of each theme. Interpret the data by integrating the qualitative insights with the quantitative results, explaining how they complement or contrast with each other.

A mixed methods matrix is a visual tool used to integrate and compare qualitative and quantitative data in mixed methods research. It helps researchers organize data according to key themes or variables, facilitating a comprehensive analysis and interpretation.

The matrix consists of several rows and columns:

• Rows represent key themes or research questions
• Columns represent different data sources or methods (e.g. interviews, surveys, observations)

By populating each cell with relevant data segments, researchers can easily identify areas of convergence, divergence, and complementarity. Let’s say you want to answer this research question: How does a new health intervention impact patient satisfaction and health outcomes?

You would populate the matrix as follows:

How to conduct mixed methods research: A mixed method research example

Let’s say you own a project management app and want to understand user satisfaction and identify areas for improvement. Here are eight steps to apply mixed methods research—using the convergent parallel technique—to discover user pain points and create a better user experience.

Step 1: Define your research objectives

In UX research , asking the right questions is crucial for identifying user needs and pain points effectively. But in order to write the right user research questions , you need to define a clear objective. What are you looking to understand?

Defining a clear UX research objective helps guide all other research decisions and acts as a lighthouse that guides your research project.

In our example , our research objective could be ‘to explore user experience and identify areas for improvement within our project management app’.

Step 2: Design your study and recruit participants

Ensure your study is designed to allow integration of both quantitative and qualitative data. There are various mixed method research designs to choose from—the right one for you depends on your research objectives and preferences.

At this stage, you should also establish a clear strategy for data integration and decide how you’ll combine the qualitative and quantitative data during the UX reporting and analysis phase. This might involve merging data sets for comparative analysis , or embedding one data set within the other to provide additional context.

The integration plan should reflect your research goals and ensure that the combined data offers a clear understanding. For our study, we’ll design a convergent parallel mixed methods study and triangulate our data during the analysis phase. This enables us to find our what and our why.

This is also when you need to recruit research participants for your study. Consider what you’re studying and identify your target test audience. You then need to create a call-out for your research study—either on socials, via email, or with In-Product Prompts .

Alternatively, you can find and filter research participants using Maze Panel , then manage your participant relationships using Maze Reach .

Step 3: Collect quantitative data

Next up, you want to start gathering your quantitative data. A good way to do this is with a survey to collect numerical data that can be statistically analyzed. For example, a user satisfaction survey that includes rating scales (1–10) for various aspects of the software.

For our research into app user satisfaction, we asked:

• Please rate your overall satisfaction with the app (1–10)
• How often do you use the app per week?
• How easy is the app to use on a scale of 1 to 10?
• How likely are you to recommend the app to a friend or colleague (1–10)?

❓ Need a quick and easy way to create and manage surveys? Maze Feedback Surveys simplify your feedback collection process so you can focus on making the changes your customers want to see. You can quickly create surveys tailored to your needs with Maze's survey templates .

Step 4: Collect qualitative data

Once you’ve got your quantitative data, it’s time to collect your qualitative data. Consider conducting user interviews or focus groups to obtain detailed, descriptive data that provides context and deep understanding.

For our study, we selected 20 users from the survey who gave varied ratings and conducted 30-minute interviews, asking:

• What do you like most about the app?
• What features do you find difficult to use?
• Can you describe a recent experience using the app?
• What improvements would you suggest?

💬 User interviews are resource-intensive and time-consuming. Speed them up with Maze’s end-to-end user interview solution: Interview Studies .

Step 5: Quantitative data analysis

Now you’ve got all your data—it’s time to dig in. For your quantitative data, this involves using statistical methodology to identify trends and patterns.

When we looked at our example data, we calculated:

• CSAT score: 75%
• Frequency of use: 70% use the app daily
• Ease of use average score: 6.8/10
• Net Promoter Score (NPS): 65

Step 6: Qualitative data analysis

Analyzing qualitative data involves coding and categorizing qualitative responses to uncover themes and patterns. Identify recurring themes in user feedback, such as ease of use, functionality, and improvement areas. If you’re using Maze Interview Studies to analyze your findings, you can automatically extract key themes and summaries to speed this process up.

When reviewing qualitative data, we found a number of interesting nuggets in our qualitative data:

• Users express dissatisfaction with the app’s usability, specifically the navigation between different functionalities
• Users wish they could access their billing details via the app, instead of solely via the web
• User find the core functionality—the project management features—to be highly valuable to their day-to-day, but also report finding the interface to be clunky and unintuitive

Step 7: Integrate data and interpret findings

Following your analysis, combine the findings from both data sets and draw conclusions. Look for correlations and insights that span both types of data.

Example integration:

• High satisfaction scores (75%) but lower ease of use (6.8/10) prove a strong product market fit but call for a more intuitive experience
• Further qualitative research agreed with this conclusion and identified specific areas for improvement, such as adding additional functionalities and improving the interface

Step 8: Report findings to stakeholders for buy-in

Present the integrated results to highlight how qualitative insights support or explain quantitative trends.

The format of your report will depend on your audience:

• Internal stakeholders (project managers, designers): Consider a concise report with clear visuals like charts, graphs, and user quotes to highlight key findings and actionable recommendations
• External stakeholders (clients, investors): Create a formal report with a clear introduction, methodology section, and comprehensive results presentation, summarizing key findings and highlighting the impact on user satisfaction and app usage

Always strive to go beyond what the data says and explain why it matters.

For example, once we’d conducted our research and drawn conclusions, we compiled this into a report that shared:

• Research methods: We used mixed methods research (surveys and interviews) to explore existing user pain points and satisfaction levels.
• Overall findings: User satisfaction is moderately high (7.5/10), indicating a generally positive reception. However, the ease of use score (6.8/10) and qualitative feedback highlight significant usability issues for new users.
• Actionable next steps based on findings: Simplify the user interface to improve the experience for new users, potentially increasing overall satisfaction and ease of use scores.

Conducting mixed methods research with Maze

Mixed methods research is one of the most effective ways to boost your UX insights, and gather a more rounded understanding of your users’ problems and perspectives. Combining research methods and types of data can uncover insights you may otherwise miss. And while there are ideal times to conduct qualitative, quantitative, or mixed methods research, ultimately it really is as simple as more research = more insights .

If you’re looking for the ideal research companion to help conduct mixed methods research, consider Maze. Maze is the user research platform that empowers all teams with the research methods they need to get game-changing insights. Whether it’s a mixed methods study or a one-off test—Maze helps you gather accurate insights, faster, for more informed decision-making.

Frequently asked questions about mixed methods research

What is the purpose of mixed methods research?

The purpose of mixed methods research is to combine quantitative and qualitative data to provide a more complete understanding of a research problem. This approach helps validate findings, explore complex issues from multiple perspectives, and produce more reliable and actionable results.

What’s the difference between qualitative and quantitative research?

• Qualitative research explores non-numerical data to understand concepts, opinions, or experiences. It uses methods like interviews, focus groups, and observations to gather in-depth insights.
• Quantitative research focuses on numerical data to quantify variables and uncover patterns. It uses methods like surveys, experiments, and statistical analysis to measure and analyze data.

What is the difference between mixed methods and multiple methods?

Mixed methods research integrates qualitative (e.g. interviews) and quantitative (e.g. surveys) data within a single study. Multiple methods research uses various research approaches, but they can be either qualitative or quantitative. For example, it might use surveys and experiments (quantitative) or interviews and focus groups (qualitative) in different parts of a study without combining the data.

How to... Use mixed methods research

In our experience, many editors are particularly pleased to receive submissions that combine qualitative and quantitative research. Find out more about this "mixed methods" approach.

On this page

Why use mixed methods research.

• Advantages & disadvantages of using mixed methods

Undertaking mixed methods research

The research question, strategies & typologies.

Back in the 1970s and 1980s, qualitative and quantitative research were seen as two opposing paradigms, each with their own supporters, most of whom would have denied the possibility of mixing them – this period is often referred to as "the paradigm wars".

During the last two decades, however, researchers in both camps have begun to see the importance of each other's data.

While researchers have been mixing quantitative and qualitative techniques for years, what is new is the deliberate attempt to provide theoretical background and formal design(s).

Undoubtedly, what has fuelled the interest has been the increasing complexity of research problems, and the desire for an understanding that is both deep and broad (afforded by qualitative and quantitative data respectively). 

The increasing importance of the link between research and policy has also played its part, as has the greater availability of data provided by the internet, which enables multiple data sets to be integrated in different ways, providing a richer and more informed picture of social phenomena.

Advantages and disadvantages of using mixed methods

Mixing data sets can give a better understanding of the problem and yield more complete evidence – the investigator gains both depth and breadth. 

Amalgamating statistics with thematic approaches can help avoid over-reliance on the former and can also capture "soft-core views and experiences" (Jogulu and Pansiri, 2011) and the subjective factors necessary to elucidate complex social situations.

It can also strengthen findings – a process known as triangulation.

On a more philosophical level, mixed methods research combines paradigms, allowing investigation from both the inductive and deductive perspectives, and consequently enabling researchers to combine theory generation and hypothesis testing within a single study (Jogulu and Pansiri, 2011). 

Having to use mixed methods also helps researchers to develop their skills, which is particularly important for those at an early stage of their career. 

Mixed methods are not without their drawbacks, however.  An obvious one being the resources and skills required – one researcher may not be skilled in both qualitative and quantitative methods and may have to call on the expertise of someone else, or another team, which will increase the cost.

While mixed methods are used widely throughout the social and behavioural sciences, there are a number of particular areas where the approach has become popular.

• Multidisciplinary research, including research which focuses on a substantive field, such as childhood or disability.
• Research on complex social issues, particularly populations which would be difficult to reach with a questionnaire, but where it is still necessary to employ quantitative methods to demonstrate scale and thereby achieve change (Brannan, 2005).
• Complex and pluralistic contexts, for example, studies involving schools, cross-national studies, and studies where it might be important to bring in a different perspective to counterbalance, say, a rather individualistic bias from informants.
• Practical and policy-related academic research. (Both sorts of research are needed for the reasons given above.)

A research methodology needs a philosophical and epistemological underpinning, i.e. what is the nature of the knowledge you are trying to uncover?

Quantitative research is based on positivism; qualitative on interpretivism or social constructivism: the distinction is between objective and subjective knowledge.

Johnson and Onwuegbuzie (2004) advocate pragmatism as the philosophical partner for mixed method research, citing the pragmatist philosophers William James and John Dewey who believed in settling metaphysical disputes by tracing the practical consequences.

Johnson and Onwuegbuzie's article provides a useful introduction to the philosophical debate, and to pragmatism. Tashakkori and Teddlie (2003) also have a chapter on pragmatism in their book. Cresswell (2009) provides a useful introduction to the different world views that accompany different research methodologies. De Loo and Lowe (2011) discuss mixed methods in the contexts of the philosophy of science. 

Most researchers would see the research question(s) and/or hypotheses as fundamental to selecting the type of research methodology.  Typically, a mixed method research design should be chosen when these are best answered by both qualitative and quantitative information.

Salfi (2011) conducted a study which looked at successful leadership practices of headteachers in Pakistan.  His research questions (RQs) were: 

• RQ1.  What type of leadership practices are employed by the headteachers of successful schools in Pakistan?
• RQ2.  Is there any similarity among leadership practices of headteachers of these schools?
• RQ3.  Is there any difference between the opinions of headteachers and their subordinates regarding their leadership practices?

The author obviously needed to obtain as wide a profile as possible so his design included a survey with demographic, Likert-scaled, and open-ended questions; however, he also needed opinions, which are subjective, and best dealt with by qualitative methods. 

Questions of frequency may best be explored by quantitative methods, and perception and opinion by qualitative. If the questions deal with both of these, then mixed methods are likely to be preferred. It may also be the case that the investigator wants to explore a subset of a population in greater depth, perhaps after a wider survey.

See Creswell (2009) for a discussion of how to frame mixed method questions and hypotheses. 

As early as 1989, Greene  et al . (cited by Johnson and Onwuegbuzie, 2004) suggested five main rationales for conducting mixed methods research:

• Triangulation  – the confirmation of results by different methods. 
• Complementarity  – results from one method are used to enhance, elaborate or clarify results from another method.
• Initiation  – where new insights are obtained which will stimulate new research questions.
• Development  – results from one method shape another method.
• Expansion  – expanding the breadth and the range of the research by using different methods for different lines of enquiry.

Murphy and Maguire (2011) used mixed methods to evaluate the costs and benefits of conducting clinical trials. They devote considerable space to the analysis of their data, which they had trouble triangulating: in the end, they decided that their data yielded complementary findings. 

Wedawatta et al. (2011) used a mixed methods approach (questionnaire and case study) to investigate the impact of extreme weather events on the construction of small to medium-sized enterprises. Their design could be described as both complementary, as results from the case studies threw further light on the findings from the survey, while it also initiated new knowledge and further research questions.

The important point to note here is that using different methods is not simply a matter of amalgamating the data and corroboration of results; rather, the investigator needs to think very carefully as to how the results are being combined, and what findings, or questions, arise.

There are a number of different types of mixed method research design.  The most common terms researchers use are:

• Concurrent  or  simultaneous  – different research methods are incorporated into the same research study and the information integrated to interpret the results.
• Sequential  – one method is used initially, followed by another method, possibly followed by a third.

The methods may have equal status, or one may be dominant.

Jogulu and Pansiri (2011) provide the following matrix, adapted from other researchers, which illustrates the typology outlined above: 

De Silva (2011) describes the adoption of mixed methods in in voluntary environmental reporting research, and her own experience of using a QUAN→qual design.

Jogulu and Pansiri (2011) analyse two management PhDs which use mixed methods research (p. 691ff). Both used a sequential strategy, using quantitative data followed by qualitative data – QUAN→qual.

Harrison and Reilly (2011) analyse the use of mixed method research in major marketing journals, categorizing their findings into exploratory (qual→QUAN), explanatory (quan→QUAL), embedded (in which one form of data play a supporting role), concurrent, and hybrid.

The type of design used and, in particular, the dominance of qualitative or quantitative will affect the analysis.

Combining quantitative and qualitative research is not simply a matter of using different methods and then adding up the results to form a set of findings. 

Mixed method research has emerged as a new research paradigm alongside qualitative and quantitative research, with its own methodology and procedures, which the investigator would do well to acquaint themselves with. 

Once they have done so, mixed methods can be an exciting and rewarding field, one that not only yields solid research, but research that can be used.

• Brannen, J. (2005) "NCRM methods review papers, NCRM/005. Mixed methods research: a discussion paper", unpublished discussion paper, available at:  http://eprints.ncrm.ac.uk/89/  [accessed 25 July 2011].
• Cresswell, J.W. (2009),  Research Design: Qualitative, Quantitative, and Mixed Methods Approaches  (3rd ed.), Sage Publications Inc.
• De Loo, I. and Lowe, A. (2011), " Mixed methods research: don't – 'just do it' ",  Qualitative Research in Accounting & Management , Vol. 8 No. 1, pp. 22-38.
• De Silva, T.A. (2011), " Mixed methods: a reflection of its adoption in environmental reporting ",  Qualitative Research in Accounting & Management , Vol. 8 No. 1, pp. 91-104.
• Greene, J.C., Caracelli, V.J. and Graham, W.F. (1989), "Toward a conceptual framework for mixed-method evaluation design",  Educational Evaluation and Policy Analysis , Vol. 11 No. 3, pp. 255-274.
• Harrison, R. and Reilly, T. (2011), " Mixed methods designs in marketing research ",  Qualitative Market Research , Vol. 14 No. 1, pp. 7-26.
• Jogulu, U.D. and Pansiri, J. (2011), " Mixed methods: a research design for management doctoral dissertations ",  Management Research Review , Vol. 34 No. 6, pp. 687-701.
• Johnson, R.B. and Onwuegbuzie, A.J. (2004), "Mixed methods research: a research paradigm whose time has come",  Educational Researcher , Vol. 33 No. 7, pp. 14-26.
• Murphy, L. and Maguire, W. (2011), " Applying mixed methods research in evaluating clinical trials ",  Qualitative Research in Accounting & Management , Vol. 8 No. 1, pp. 72-90.
• Salfi, N.A. (2011), " Successful leadership practices of head teachers for school improvement: Some evidence from Pakistan ",  Journal of Educational Administration , Vol. 49 No. 4, pp. 414-432.
• Tashakkori, A. and Teddlie, C. (Eds) (2003),  Mixed Methods in Social and Behavioural Research , Sage Publications Inc.
• Wedawatta, G., Ingirige, B., Jones, K. and Proverbs, D. (2011), " Extreme weather events and construction SMEs: Vulnerability, impacts, and responses ", Structural Survey, Vol. 29 No. 2, pp. 106-119.

 ... the use of quantitative and qualitative approaches in combination provides a better understanding of research problems than either approach alone.  

Related topics

Using ethnographic methods.

Find out how to use ethnographic research methods and participant observation in our detailed guide.

Make your research easy to find with SEO

Help your article gain attention with clever use of search engine optimisation (SEO) at the writing stage.

Proofreading

In this guide, we explain what you should look for at the proofing stage.

Mixed methods

Published November 22, 2021. Updated December 13, 2021.

Mixed methods research, also known as hybrid methods research, is an evolving research methodology that involves the methodical integration or combination of quantitative and qualitative research approaches within a single research study.

Mixed methods research incorporates the strength of both qualitative and quantitative methods, facilitating researchers in investigating diverse perspectives and discovering relationships between the complex layers of sophisticated research questions.

What is mixed methods research?

Mixed methods research involves an integration of approaches to data collection, data analysis, and elucidation of the evidence. The term ‘mixed’ signifies that data association or assimilation at an appropriate stage during the research process is essential to the mixed methods approach. Mixed methods research allows a research question to be studied comprehensively as it involves applying a precise and pre-determined research design that combines qualitative and quantitative elements to generate an integrated set of evidence addressing a single research question. Using mixed methods research allows the limitations of one approach to be compensated by the strengths of another approach. For example, the qualitative approach can provide deep insights about a trend discovered through quantitative research while the quantitative approach can validate a behavioral pattern disclosed by qualitative research. This type of research also encourages the development of conceptual models and the creation of new instruments to interpret the significance of outcomes of a study. For example:

A demographer wants to study health issues in low socioeconomic regions of a country. If mortality rates and frequency of disease outbreaks is the study focus, then a simple quantitative analysis can be performed. If instead understanding people’s perceptions about healthcare expenditure or understanding disease risk is the focus, then performing a qualitative analysis would be a more suitable option. If, however, the intent is to integrate both questions into one research study, using a mixed-methods approach would be the best choice. Using a mixed-method design in this case would allow for investigation of whether lack of healthcare expenditure is related to disease outbreaks in certain areas or why some areas have higher mortality rates than others.

Mixed methods research is usually applied in behavioral, health, and social science experiments conducted in multidisciplinary settings and complex situations. It should not be confused with multi-methods research wherein either multiple qualitative approaches or multiple quantitative approaches are combined, but no mixing of quantitative and qualitative approaches occurs.

Five characteristics of mixed methods designs

Five important characteristics of mixed methods designs are:

• Triangulation : This design allows the use of multiple methods to collect data on the same topic. It seeks convergence and validation of results from different methods.
• Complementarity : The results of one method are elaborated, enhanced, illustrated, and clarified with the results of another method.
• Initiation : Newly obtained insights help in the stimulation of new research questions.
• Development : The results of one method are used to build another method.
• Expansion : The research breadth and range are expanded using different methods for different areas of investigation.

Mixed methods designs

While designing mixed methods research, a researcher should pay attention to details such as the research questions, the research approach to be used, and the type of data to be collected (both primary and secondary). Mixed methods research designs can be of various types. Factors such as the research objective, category of each data type in the study, and the timing and sequence of data collection give rise to different mixed methods designs. Some of the most common mixed methods designs are discussed below:

• Convergent Parallel : In this design, qualitative and quantitative data collection and analyses are performed simultaneously but separately. At the end of the study, both the analyses are compared and interpreted together to draw overall conclusions.

For example, when studying health issues in low socioeconomic regions, the demographer can explore both types of data simultaneously:

• On the qualitative side, surveys and one-on-one interviews can be conducted to gain perspectives on healthcare expenditure.
• On the quantitative side, literature reviews and database searches can be used to determine frequency of disease outbreaks and mortality rates.
• After data collection and analysis, the results can be interpreted to draw meaningful conclusions.
• Explanatory Sequential : In this design, quantitative data collection and analysis are performed first and followed by qualitative data collection and analysis. Quantitative results are used to structure qualitative sampling and data collection, and the qualitative results are used to examine and explain the quantitative findings.

In the above example, while following an explanatory research design the demographer first draws primary conclusions about the mortality rates and the frequency of disease outbreaks in low socioeconomic regions. Based on these findings, interviews are conducted with the residents of those regions to qualitatively analyze reasons influencing their healthcare expenditure. These perspectives can be used to interpret mortality rates and disease outbreak frequencies in the regions.

• Exploratory Sequential : In this design, qualitative data collection and analysis are performed in the first phase and quantitative data collection and analysis are carried out in the second phase. The initial questions are first explored to develop a hypothesis, followed by the determination of quantitative research questions and variables, and finally data interpretation.  

The quantitative data is, in turn, used to check and confirm the qualitative findings. For example:  

While following the exploratory research design, in the first phase the demographer explores participants’ tendency and capability of healthcare expenditure. Then mortality rates and frequency of disease outbreaks are analyzed to confirm if participants’ perceptions about healthcare expenditure govern mortality rates and disease outbreaks in those regions.

• Embedded : In this design, qualitative and quantitative data are collected and analyzed concurrently with a secondary dataset. The collection of secondary data may occur before, during, or after the primary methods.  

This type of design can reinforce or enhance the conclusions obtained from the primary type of research design. For example:

As a part of a quantitative study, the demographer tests whether amount of healthcare expenditure is correlated to mortality rates and disease outbreaks in low socioeconomic regions. While a series of secondary interviews with relatives of the deceased could be embedded to further strengthen the demographer’s conclusions, the bulk of the research remains quantitative.

When to use mixed methods research

Mixed methods research is usually used when a research question cannot be answered solely by using either the qualitative or the quantitative research method. The common reasons for using mixed methods research are:

• Generalizability : Qualitative research lacks generalizability as it employs a small sample size. In mixed methods research, this drawback is overcome by large sample sizes and high external validity by inclusion of quantitative research.
• Contextualization : Mixed methods research helps in interpreting actions of study participants by identifying motivations and drawing conclusions that are rich in detail.
• Reliability : The rationale of triangulation makes mixed methods research reliable and credible. The convergence of qualitative and quantitative data further strengthens the validity of the study conclusions.

While formulating a research question, analyze how qualitative and quantitative studies can be combined to answer it. If the research question can be answered by either one of the studies, using mixed methods research is not a feasible option.

Data collection tools of mixed methods research

Since mixed methods research is a combination of qualitative and quantitative approaches, the data collection tools are a mixture of those used in both types of research.

• Surveys : Produce quantitative data related to the frequency of events, behavior, or perceptions through semantic differentials or multi-select questions.
• Interviews : Produce rich and rationalized qualitative data of participants’ perspectives.
• Observations : Offer natural, situational, and researcher-perspective data that are useful during big-picture analysis of a study.
• Usability studies : These studies provide quantitative data in the form of heat maps and time-to-task metrics. This type of data is often collected passively and not provided by the participants of a study.
• Co-creation work : This method produces the conventional open-ended data from interviews, focus groups, or participant-created data such as drawings or models.
• Diary : This tool can be longitudinal, producing data inclusive of rich descriptions and media such as photos and videos.

Analysis of mixed methods research

The mixed methods research can be analyzed by using the following two approaches:

• Top-down approach : In this approach, the qualitative or the open-ended analysis is improved through quantitative data obtained from surveys or reports. The quantitative data provides suggestions, pointers, or indications to look for when commencing the qualitative research. For example:  

While analyzing the customer satisfaction surveys, a researcher finds the service of a restaurant to be a recurring problem. The researcher then begins interviews to understand customers’ experiences at restaurants while focusing on restaurant services.

• Bottom-up approach : This method is the opposite of the top-down approach. This approach begins with qualitative research inclusive of open-ended data, theme generation, and grounded theory for discovering significant concepts. For example:

A travel company finds out the three most favored recreational activities from a large set of interviews. A bottom-down approach would involve checking the frequency of each of these activities from the existing customer base by using survey questions. This would help the travel company pitch new recreational trips based on the expectations and preferences of their clients.

Advantages of mixed methods

A mixed methods research design, apart from expanding and strengthening the conclusions of a study, can have other benefits such as:

• An amalgamation of different perspectives : Research questions can be studied from different perspectives using mixed methods research. It allows the combination of rich, rationalized, and subjective insights of qualitative data with standardized, generalizable, and externally valid quantitative data. The limitations of one design are compensated by the strengths of the other design. Quantitative studies are usually limited to laboratory settings and do not incorporate the real-life experiences of participants. Therefore, the addition of qualitative data provides more richness to quantitative results. Qualitative studies, on the other hand, are usually not generalizable as they are governed by the experiences of participants. The addition of quantitative data can help validate qualitative data.
• Methodological flexibility : Mixed methods are flexible in terms of the research methodologies used. They do not necessarily adhere to the established research standards or patterns and allow the adaptation of various aspects of different studies to elucidate results.
• Reflect participants’ viewpoints: Mixed methods take into consideration the perspectives of participants to ensure the study results are reflective of participants’ experiences.

Limitations of mixed methods

Apart from being challenging to implement, a mixed methods design has a few other drawbacks such as:

• Complex evaluations : Planning and conducting mixed methods research is a complex process. Integration of qualitative and quantitative data poses a major challenge during mixed methods research. Apart from this, selection of sample (for both the qualitative and quantitative phases), sequence of qualitative and quantitative components, and integration of data require careful planning.
• Requires a methodological skillset : Performing mixed methods research requires a team of multidisciplinary researchers who are comfortable working with methods that do not lie in the area of their expertise. Moreover, finding quantitative researchers who can discuss qualitative analyses and vice-versa can prove challenging.
• Increased resources : Mixed methods research is labor-intensive, time-consuming, and requires access to increased resources in comparison to a single study.

Why is mixed methods research needed?

The need for mixed methods research stems from the complexity of research problems and researchers’ desire for deep and broad understanding, both of which can be attained by combining qualitative and quantitative data. Further, this research method can produce firm and rich analyses and conclusions for various research problems. It also helps in generating deep and meaningful insights about a problem and its solutions.  Moreover, mixing methods is no more seen as a daunting or unattainable task. It allows the integration of the best features of qualitative and quantitative research while accounting for the weakness of one method by the strengths of the other. This has led to a paradigm shift from conventional research methodologies.

Key Takeaways

• Mixed methods or hybrid methods research is an evolving research methodology that involves the methodical integration of quantitative and qualitative research approaches within a single study.
• Within a mixed methods design, the qualitative approach provides deep insights about a trend discovered through quantitative research, while the quantitative approach helps validate a behavioral pattern disclosed by qualitative research.
• The five most important characteristics of mixed methods research are triangulation, complementarity, initiation, development, and expansion.
• Convergent parallel, explanatory sequential, exploratory sequential, and embedded designs are the most commonly used mixed methods research designs.
• Through mixed methods design, research is flexible and reflects participants’ viewpoints; it is labor-intensive, time-consuming, and requires a team of multidisciplinary researchers who can interpret both qualitative and quantitative analyses.

Research Basics

For more details, visit these additional research guides .

Types of Research

• What is research
• What is the scientific method
• Scientific method observation
• Inductive and deductive reasoning
• What is empirical evidence
• Research methods
• Qualitative research design
• Quantitative research design
• Qualitative vs quantitative research
• Mixed methods research

What’s included with a Chegg Writing subscription

• Unlimited number of paper scans
• Plagiarism detection: Check against billions of sources
• Expert proofreading for papers on any subject
• Grammar scans for 200+ types of common errors
• Automatically create & save citations in 7,000+ styles
• Cancel subscription anytime, no obligation

Chapter 15. Mixed Methods

Introduction.

Where deep ethnography (chapter 14) is a tradition that relies on naturalistic techniques of data collection, foregrounding the specificity of a particular culture and site, there are other times when researchers are looking for approaches that allow them to make use of some of the analytical techniques developed by statisticians and quantitative researchers to generalize the data they are collecting. Rather than push into a deeper understanding of a culture through thick interpretive descriptions, these researchers would rather abstract from a sufficiently large body of cases (or persons) to hazard predictions about a connection, relationship, or phenomenon. You may already have some experience learning basic statistical techniques for analyzing large data sets. In this chapter, we describe how some research harnesses those techniques to supplement or augment qualitative research, mixing methods for the purpose of building stronger claims and arguments. There are many ways this can be done, but perhaps the most common mixed methods research design involves the use of survey data (analyzed statistically via descriptive cross-tabs or fairly simple regression analyses of large number probability samples) plus semistructured interviews. This chapter will take a closer look at mixed methods approaches, explain why you might want to consider them (or not), and provide some guidance for successful mixed methods research designs.

What Is It? Triangulation, Multiple Methods, and Mixed Methods

First, a bit of nomenclature. Mixed methods can be understood as a path toward triangulation . Triangulation is a way of strengthening the validity of a study by employing multiple forms of data, multiple investigators, multiple theoretical perspectives, or multiple research methods. Let’s say that Anikit wants to know more about how first-year college students acclimate to college. He could talk to some college students (conduct interviews) and also observe their behavior (fieldwork). He is strengthening the validity of his study by including multiple forms of data. If both the interview and the observations indicate heavy reliance on peer networks, a reported finding about the importance of peers would be more credible than had he only interviewed students or only observed them. If he discovers that students say one thing but do another (which is pretty common, after all), then this, too, becomes an interesting finding (e.g., Why do they forget to talk about their peers when peers have so much observable influence?). In this case, we say that Anikit is employing multiple forms of data, or even that he relies on multiple methods. But he is not, strictly speaking, mixing data. Mixed methods refer specifically to the use of both quantitative and qualitative research methods. If Anikit were to supplement his interviews and/or observations with a random sample of one thousand college students, he would then be employing a mixed methods approach. Although he might not get the rich details of how friends matter in the survey, the large sample size allows statistical analyses of relationships among variables, perhaps showing which groups of students are more likely to benefit from strong peer networks. So to summarize, both multiple methods and mixed methods are forms of research triangulation, [1] but mixed methods include mixing both qualitative and quantitative research elements.

Mixed methods techniques, then, are pretty unique. Where many qualitative researchers have little interest in statistical generalizability, and many quantitative researchers undervalue the importance of rich descriptions of singular cases, the mixed methods researcher has an open mind about both approaches simultaneously. And they use the power of both approaches to build stronger results: [2]

Quantitative (mainly deductive) methods are ideal for measuring pervasiveness of “known” phenomena and central patterns of association, including inferences of causality. Qualitative (mainly inductive) methods allow for identification of previously unknown processes, explanations of why and how phenomena occur, and the range of their effects (Pasick et al. 2009). Mixed methods research, then, is more than simply collecting qualitative data from interviews, or collecting multiple forms of qualitative evidence (e.g., observations and interviews) or multiple types of quantitative evidence (e.g., surveys and diagnostic tests). It involves the intentional collection of both quantitative and qualitative data and the combination of the strengths of each to answer research questions . ( Creswell et al. 2011:5 ; emphases added)

Why Use Mixed Methods?

As with all methodological choices, the answer depends on your underlying research questions and goals. Some research questions are better answered by the strengths of the mixed methods approach. Small ( 2011 ) discusses the use of mixed methods as a confirmation or complement of one set of findings from one method by another. Creswell and Clark ( 2017:8ff .) note the following situations as being particularly aided by combining qualitative and quantitative data collection and analysis: (1) when you need to obtain both more complete (need for qualitative) and more corroborated (need for quantitative) information; (2) when you need to explain (need for qualitative) initial results (quantitative); (3) when you need to do an exploratory study (need for qualitative) before you can really create and administer a survey or other instrument (quantitative); (4) when you need to describe and compare different types of cases to get a more holistic understanding of what is going on; (5) when you need (or very much want!) to include participants in the study, adding in qualitative elements as you build a quantitative design; (6) when you need all the tools at your disposal to develop, implement, and evaluate a program.

Please note what is not included in this list: because you can . Mixed methods research is not always preferable, even if in general it makes your study “stronger.” Strength is not the only criterion for quality or value. I have met many students in my career who assume that the mixed methods approach is optimal because it includes both qualitative and quantitative research. That is the wrong way of looking at things. Mixed methods are optimal when and only when they fit the necessities of your research question (e.g., How can I corroborate this interesting finding from my interviews so that proper solutions can be fashioned?) or underlying goal (e.g., How can I make sure to include the people in this program as participants of the study?).

If you are just starting out and learning your way through designing your first study, mixed methods are not default requirements. As you will see in the next section on design, mixed methods studies often happen sequentially rather than consecutively, so I recommend you start with the study that has the most meaning to you, the one that is the most compelling. Later on, if you want to add (mix) another approach for the sake of strength or validity or “corroboration” (if you are adding quantitative) or “explanation” (if you are adding qualitative), you can always do that then, after the completion of your first study.

Segue: Historical Interlude

For those interested in a little history, one could make the case that mixed methods research in the social sciences actually predates the development of either quantitative or qualitative research methods. The very first social scientists (what we call “social science” in the West, which is itself a historical construct, as many other peoples have been exploring meaning and interpretation of the social world for centuries if not millennia) often employed a mélange of methods to address their research questions. For example, the first sociologists in the US operating out of the “Chicago School” of the early twentieth century surveyed neighborhoods, interviewing people, observing demographic subcultures, and making tallies of everything from the numbers of persons in households to what languages were being spoken. They learned many of these techniques from early statisticians and demographers in Europe—people like Charles Booth ( 1902 ), who surveyed neighborhoods in London, and Frédéric Le Play, who spent decades examining the material conditions of the working classes across Europe, famously including family “budgets” along with interviews and observations (see C. B. Silver 1982). The renowned American sociologist W. E. B. Du Bois, who was the first Black man to earn a PhD from Harvard University, also conducted one of the very first mixed methods studies in the US, The Philadelphia Negro ( 1899 ). This work mapped every Black residence, church, and business in Philadelphia’s Seventh Ward and included observations and details on family structure and occupation (similar to Booth’s earlier work on London). Continuing through the 1930s and 1940s, “community studies” were conducted by teams of researchers who basically tallied everything they could find about the particular town or city they chose to work in and performed countless interviews, months and years of fieldwork, and detailed mappings of community relationships and power relations. One of the most famous of these studies includes the “Middletown” studies conducted by Robert and Helen Lynd ( 1929 , 1937 ).

As statistical analysis progressed after World War II alongside the development of the technology that allowed for ever faster computations, quantitative research emerged as a separate field. There was a lot to learn about how to conduct statistical analyses, and there were more refinements in the creation of large survey instruments. Qualitative research—the observations and interviews at the heart of naturalistic inquiry—became a separate field for different kinds of researchers. One might even say qualitative research languished at the expense of new developments of quantitative analytical techniques until the 1970s, when feminist critiques of positivist social science emerged, casting doubt on the superiority of quantitative research methods. The rise of interdisciplinarity in recent decades combined with a lessening of the former harsh critique of quantitative research methods and the “paradigm wars” ( Small 2011 ) has allowed for an efflorescence of mixed methods research, which is where we are today.

Mixed-Methods Research Designs

Returning from our historical interlude to the list of possible uses of mixed methods, we now confront the question of research design. If we are using more than one method, how exactly do we do this, and when ? The how and the when will depend largely on why we are using mixed methods. For example, if we want to corroborate findings emerging from interviews, then we obviously begin with interviews and follow with, perhaps, a large survey. On the other hand, if we are seeking to explain findings generated from a survey, we begin with that survey and add interviews or observations or focus groups after its completion. And if we are seeking to include participants in the research design itself, we may want to work concurrently, interviewing and holding focus groups as surveys are administered. So it all depends on why we have chosen to use mixed methods.

We can think of our choices here in terms of three possibilities. The first, called sequential explanatory , begins with quantitative data (collection) and then follows with qualitative data (collection). After both are collected, interpretations are made. The second, called sequential exploratory , begins the other way around, with qualitative followed by quantitative. After both are collected, interpretations are made. The third, called concurrent triangulation , conceives of both quantitative and qualitative elements happening concurrently. In practice, one may still happen before the other, but one does not follow the other. The data then converge, and from that convergence, interpretations are made.

In sequential explanatory design (figure 15.1), we are asking ourselves, “In what ways do the qualitative findings explain the quantitative results?” ( Creswell et al. 2017 ). This design thus gives some priority to the quantitative data. The qualitative data, collected after the quantitative data, is used to provide a better understanding of the research problem and then the quantitative data alone.

Often, this means providing some context or explaining meanings and motivations behind the correlations found in the quantitative data. For example, in my research on college students ( Hurst 2019 ), I found a statistical correlation between upper-middle-class female students and study abroad. In other words, and stating this rather baldly, class*gender could be used to predict who studied abroad. But I couldn’t fully explain why, given the survey data I had collected. [3] To answer these (and other) questions that the survey results raised, I began interviewing students and holding focus groups. And it was through these qualitative forms of data collection that I found a partial answer: upper-middle-class female students had been taught to see study abroad as a final “finishing” component of their education in a way that other students simply had not. They often had mothers who had done the same. And they clearly saw connections here to the kinds of well-traveled cosmopolitan adults they wanted to become.

In sequential exploratory design (figure 15.2), we are asking ourselves, “In what ways do the quantitative findings generalize (or confirm) the qualitative results?” ( Creswell et al. 2018 ). This design thus gives some priority to the qualitative data. The quantitative data, collected after the qualitative data, is used to confirm the findings.

This approach is ideal for developing new instruments or when a researcher intends to generalize findings from a qualitative study to different groups or populations. The American Sociological Association (ASA) Task Force on First-Generation and Working-Class Persons wanted to understand how class background may have played a role in the success of sociology graduate students and faculty. Because this was a relatively new research question, the task force began by conducting several focus groups, asking general questions about how class might have affected careers in sociology. Based on several recurring findings (e.g., high debt burdens, mentorship, feelings of fit), the task force developed a survey instrument that it then administered to more than one thousand sociologists, thus generalizing the preliminary findings and providing corroboration of some of the key variables at play.

In concurrent triangulation design (figure 15.3), neither the quantitative nor the qualitative component takes precedence. Although in practice one might precede the other in time, neither is the tail that wags the dog, so to speak. They are both the dog. The general of this design is to better understand or deepen one’s understanding of the phenomenon under study. The goal is to obtain different but complementary data that strengthen (validate) the overall results.

These designs might be either nested or nonnested . In a nested design , a subsample of an original randomized sample is used for further interviews or observation. A common nested design form is where in-depth interviews are conducted with a subsample of those who filled out a survey. Nonnested designs occur when it is impractical or impossible to recruit the same individuals that took place in the survey. The research I conducted for my book Amplified Advantage ( Hurst 2019 ) is an example of this. I supplemented a large national survey of college students and recent college graduates with interviews and focus groups of similar college students and graduates who were not participants in the study (or who may have been randomly selected as participants but without my knowledge or linking their data). Nonnested designs are much more flexible than nested designs, but they eliminate the possibility of linking data across methods.

As with all research design, it is important to think about how best to address your particular research question. There are strengths and weaknesses of each design. Sequential design allows for the collection and analysis of different methods separately, which can make the process more manageable. Sequential designs are relatively easy to implement, design, and report. Sequential exploratory designs allow you to contextualize and generalize qualitative findings to larger samples, while sequential explanatory designs enable you to gain a deeper understanding of findings revealed by quantitative data analysis. All sequential design takes a lot of time, however. You are essentially doubling your research. This is why I do not recommend these approaches to undergraduate students or graduate students in master’s programs. In contrast, concurrent designs, whose dual methods may be conducted simultaneously, may be conducted more quickly. However, as a practical matter, you will probably end up focusing first on one data collection method and then the other, so the time saved might be minimal. [4] Concurrent design can also preclude following up on interesting findings that emerge from one side of the study, and the abbreviated form may prevent clarification of confusing issues that arise during analysis. If the results are contradictory or diverge, it may also be difficult to integrate the data. You might end up with more questions to pursue for further study and not much conclusive to say at the end of all your work.

Finally, there is what I will call here the recursive design model (figure 15.4), in which you combine both explanatory and exploratory sequential design.

This design is currently being used by the ASA task force mentioned above. The first stage of data collection involved several focus groups. From these focus groups, we constructed a survey that we administered to ASA members. The focus group survey could be viewed as an example of exploratory sequential design. As the surveys were being analyzed, we added a nested set of interviews with persons who had taken the survey and who indicated a willingness to participate in this later stage of data collection. These interviews then help explain some of the findings from the survey. The entire process takes several years, however, and involves multiple researchers!

Advanced: Crossover Design

Small’s ( 2011 ) review of the state of mixed methods research argues that mixed methods are being increasingly adopted in social science research. In addition to sequential and concurrent research designs, where quantitative and qualitative data work to either confirm or complement each other, he sets forth examples of innovative designs that go further toward truly blending the special techniques and strengths of both quantitative and qualitative methods. [5] Written in 2011, I have seen scant evidence so far that these blended techniques are becoming well established, but they are promising. As new software programs for data analysis emerge, along with increased computing power, there will be greater opportunities for crossover work. Perhaps you can take up the charge and attempt one of these more innovative approaches yourself.

Here is Small’s ( 2011:73ff .) list of innovative crossover research design:

• Network analyses of narrative textual data . Here, researchers use techniques of network analysis (typically quantitative) and apply them to narratives (qualitative), coding stories as separate “nodes” and then looking for connections between those nodes, as is done in network analysis.
• Sequence analyses of narrative textual data . Here, techniques of event structure analysis and optimal matching (designed for analysis of quantitative data) are applied to narratives (qualitative data). The narratives are reconceived as a series of events, and then causal pathways between these events are mapped. This allows for identification of crucial turning points as well as “nonsignificant” events that just happened.
• Quantitative analyses of semantic (meaning) elements of narrative textual data . The basic distinction between quantitative (data in the form of numbers) and qualitative (date in the form of words) gets blurred here, as words themselves and their meanings and contexts are coded numerically. I usually strongly advise beginning students to do this, as what often happens is that they begin to think quantitatively about the data, flattening it considerably. However, if done with full attention to meaning and context, the power of computing/analytical software may strengthen the coding process.
• Narrative analyses of large-n survey data. In contrast to the first three designs listed above, where quantitative techniques were applied to qualitative data, we now come to a situation where the reverse takes place. Here we have a large data set, either coded numerically or “raw” with various choice options for each question posed. Rather than read the data set as a series of factors (variables) whose relationship one explores through statistical analyses, the researcher creates a narrative from the survey responses, contextualizing the answers rather than abstracting them. [6]
• Regression-based analyses of small-n or narrative textual data. This is by far the most common crossover method and the reverse of the fourth example. Many qualitative software analysis programs now include basic quantitative analytical functions. The researcher can code interview transcripts and fieldnotes in such a way that allows for basic cross-tabulations, simple frequency statistics, or even basic regression analyses. Transcripts and fieldnotes can generate “variables” for such analyses.

Despite the promise of blending methods in this way, the possibility of doing damage to one’s study by discounting the particular values of either quantitative or qualitative approaches is a real one. Unlike mixed methods, where the two approaches work separately (even when designed to concur in time), crossover research blends or muddies the two. Small ( 2011 ) warns, “At a minimum, the application of techniques should not be fundamentally contrary to the epistemological principles from which they are derived or to the technical problems for which they were intended” ( 76 ). When employing any of these designs or blending approaches, it is very important to explain clearly and fully what one’s aims are and how the analysis has proceeded, as this allows others to evaluate the appropriateness of the design for the questions posed.

Further Readings

Cech, Erin. 2021. The Trouble with Passion: How Searching for Fulfillment at Work Fosters Inequality . Berkeley, CA: University of California Press.* Cech combines surveys with interviews to explore how people think about and talk about job searches and careers.

Cooper, Kristy S. 2014. “Eliciting Engagement in the High School Classroom: A Mixed-Methods Examination of Teaching Practices.” American Educational Research Journal 51(2):363–402. An example of using multilevel regression analyses with both interviews and observations to ascertain how best to engage students.

Creswell, John W., and J. David Creswell. 2018. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches . Thousand Oaks, CA: SAGE. Essential textbook for mixed-methods research.

Edin, Kathryn, and Maureen A. Pirog. 2014. “Special Symposium on Qualitative and Mixed-Methods for Policy Analysis.” Journal of Policy Analysis and Management 33(2):345–349. A good overview of the strengths of mixed-methods research, which, the authors argue, make it particularly well suited for public policy analysis.

Hurst, Allison L. 2019. Amplified Advantage: Going to a “Good” College in an Era of Inequality . Lanham, MD: Rowman & Littlefield: Lexington Books..* Employs a national survey of recent graduates of small liberal arts colleges combined with interviews, focus groups, and archival data to explore how class background affects college outcomes.

Johnson, R. Burke, and Anthony J. Onwuegbuzie. 2004. “Mixed Methods Research: A Research Paradigm Whose Time Has Come.” Educational Researcher 33(7):14–26. Takes a pragmatic approach and provides a framework for designing and conducting mixed-methods research.

Klinenberg, Eric. 2015. Heat Wave: A Social Autopsy of Disaster in Chicago . Chicago: University of Chicago Press.* A great read and could not be more timely. Klinenberg uses a combination of fieldwork, interviews, and archival research to investigate why some neighborhoods experience greater mortality than others.

Lynd, Robert, and Helen Lynd. 1929. Middletown: A Study in American Culture . New York: Harcourt, Brace.* This early mixed-methods study of a “typical” American city was a pioneering work in sociology. The husband-and-wife team seemingly explores every aspect of life in the city, mapping social networks, surveying attitudes and beliefs, talking to people about their expectations and lives, and observing people going about their everyday business. Although none of the techniques are very sophisticated, this remains a classic example of pragmatic research.

Lynd, Robert, and Helen Lynd. 1937. Middletown in Transition . New York: Harcourt, Brace. The follow-up to the Lynds’ original study of a small American city. More theoretical and critical than the first volume.

Markle, Gail. 2017. “Factors Influencing Achievement in Undergraduate Social Science Research Methods Courses: A Mixed Methods Analysis.” Teaching Sociology 45(2):105–115.* Examines the factors that influence student achievement using an initial survey with follow-up interviews.

Matthews, Wendy K. 2017. “‘Stand by Me’: A Mixed Methods Study of a Collegiate Marching Band Members’ Intragroup Beliefs throughout a Performance Season.” Journal of Research in Music Education 65(2):179–202.* The primary method here is focus groups, but the author also employed multivariate analysis of variance (MANOVA) to shore up the qualitative findings.

Monrad, Merete. 2013. “On a Scale of One to Five, Who Are You? Mixed Methods in Identity Research.” Acta Sociologica 56(4):347–360. A call to employ mixed methods in identity research.

Silver, Catherine Bodard. 1982. Frédéric Le Play on Family, Work and Social Change . Chicago: University of Chicago Press. For anyone interested in the historic roots of mixed-methods research, the work of Frédéric Le Play is essential. This biography is a good place to start.

Small, Mario Luis. 2011. “How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature.” Annual Review of Sociology 37:57–86. A massive review of recent mixed-methods research, distinguishing between mixed-data-collection studies, which combine two or more kinds of data, and mixed-data-analysis studies, which combine two or more analytical strategies. Essential reading for graduate students wanting to use mixed methods.

• To extend this notion of triangulation a little further: if Anikit enlisted the help of Kanchan to interpret the observations and interview transcripts, he would be strengthening the validity of the study through multiple investigators, another form of triangulation having nothing at all to do with what methods are employed. He could also bring in multiple theoretical frameworks—say, Critical Race Theory and Bourdieusian field analysis—as a form of theoretical triangulation. ↵
• If stronger is your aim, that is. For many qualitative researchers, verisimilitude, or the truthfulness of a presentation, is a more desirable aim than strength in the sense of validity. ↵
• Actually, I could do a fair amount of testing on other variables’ relationships to this finding: students who had gone far away to college (more than five hundred miles) were significantly more likely to study abroad, for example, as were students who majored in arts and humanities courses. But I still missed any way of getting at personal motivations or how individuals explained these motivations. That is the part a survey is just never going to fully get at, no matter how well or numerous the questions asked. ↵
• The big exception here is when you are relying on data that has already been collected and is ready for analysis, as in the case of large survey data sets like the General Social Survey. In that case, it is not too time consuming to design a mixed methods study that uses (nonnested) interviews to supplement your analyses of survey data. ↵
• I refer to these as blended methods rather than mixed methods because the epistemological positions and science claims, usually rather distinct from quantitative (more positivistic) and qualitative (more naturalistic), blur considerably. ↵
• I admit that trained first as a qualitative researcher, this has always been my impulse when confronting a large survey data set. ↵

A research design that employs both quantitative and qualitative methods, as in the case of a survey supplemented by interviews.

The process of strengthening a study by employing multiple methods (most often, used in combining various qualitative methods of data collection and analysis).  This is sometimes referred to as data triangulation or methodological triangulation (in contrast to investigator triangulation or theory triangulation).  Contrast mixed methods .

A mixed-methods design that conceives of both quantitative and qualitative elements happening concurrently.  In practice, one may still happen before the other, but one does not follow the other.  The data then converge and from that convergence interpretations are made.  Compare sequential exploratory design and sequential explanatory design .

A mixed-methods design that begins with quantitative data collection followed by qualitative data collection, which helps “explain” the initial quantitative findings.  Compare sequential exploratory design and concurrent triangulation .

A mixed-methods design that begins with qualitative data collection followed by quantitative data collection.  In this case, the qualitative data suggests factors and variables to include in the quantitative design.  Compare sequential explanatory design and concurrent triangulation .

A form of mixed-methods design in which a subsample of an original randomized sample is used for further interviews or observation.

Introduction to Qualitative Research Methods Copyright © 2023 by Allison Hurst is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License , except where otherwise noted.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Part 2: Conceptualizing your research project

9. Writing your research question

Chapter outline.

• Empirical vs. ethical questions (4 minute read)
• Characteristics of a good research question (4 minute read)
• Quantitative research questions (7 minute read)
• Qualitative research questions (3 minute read)
• Evaluating and updating your research questions (4 minute read)

Content warning: examples in this chapter include references to sexual violence, sexism, substance use disorders, homelessness, domestic violence, the child welfare system, cissexism and heterosexism, and truancy and school discipline.

9.1 Empirical vs. ethical questions

Learning objectives.

Learners will be able to…

• Define empirical questions and provide an example
• Define ethical questions and provide an example

Writing a good research question is an art and a science. It is a science because you have to make sure it is clear, concise, and well-developed. It is an art because often your language needs “wordsmithing” to perfect and clarify the meaning. This is an exciting part of the research process; however, it can also be one of the most stressful.

Creating a good research question begins by identifying a topic you are interested in studying. At this point, you already have a working question. You’ve been applying it to the exercises in each chapter, and after reading more about your topic in the scholarly literature, you’ve probably gone back and revised your working question a few times. We’re going to continue that process in more detail in this chapter. Keep in mind that writing research questions is an iterative process, with revisions happening week after week until you are ready to start your project.

Empirical vs. ethical questions

When it comes to research questions, social science is best equipped to answer empirical questions —those that can be answered by real experience in the real world—as opposed to  ethical questions —questions about which people have moral opinions and that may not be answerable in reference to the real world. While social workers have explicit ethical obligations (e.g., service, social justice), research projects ask empirical questions to help actualize and support the work of upholding those ethical principles.

In order to help you better understand the difference between ethical and empirical questions, let’s consider a topic about which people have moral opinions. How about SpongeBob SquarePants? [1] In early 2005, members of the conservative Christian group Focus on the Family (2005) [2] denounced this seemingly innocuous cartoon character as “morally offensive” because they perceived his character to be one that promotes a “pro-gay agenda.” Focus on the Family supported their claim that SpongeBob is immoral by citing his appearance in a children’s video designed to promote tolerance of all family forms (BBC News, 2005). [3] They also cited SpongeBob’s regular hand-holding with his male sidekick Patrick as further evidence of his immorality.

So, can we now conclude that SpongeBob SquarePants is immoral? Not so fast. While your mother or a newspaper or television reporter may provide an answer, a social science researcher cannot. Questions of morality are ethical, not empirical. Of course, this doesn’t mean that social science researchers cannot study opinions about or social meanings surrounding SpongeBob SquarePants (Carter, 2010). [4] We study humans after all, and as you will discover in the following chapters of this textbook, we are trained to utilize a variety of scientific data-collection techniques to understand patterns of human beliefs and behaviors. Using these techniques, we could find out how many people in the United States find SpongeBob morally reprehensible, but we could never learn, empirically, whether SpongeBob is in fact morally reprehensible.

Let’s consider an example from a recent MSW research class I taught. A student group wanted to research the penalties for sexual assault. Their original research question was: “How can prison sentences for sexual assault be so much lower than the penalty for drug possession?” Outside of the research context, that is a darn good question! It speaks to how the War on Drugs and the patriarchy have distorted the criminal justice system towards policing of drug crimes over gender-based violence.

Unfortunately, it is an ethical question, not an empirical one. To answer that question, you would have to draw on philosophy and morality, answering what it is about human nature and society that allows such unjust outcomes. However, you could not answer that question by gathering data about people in the real world. If I asked people that question, they would likely give me their opinions about drugs, gender-based violence, and the criminal justice system. But I wouldn’t get the real answer about why our society tolerates such an imbalance in punishment.

As the students worked on the project through the semester, they continued to focus on the topic of sexual assault in the criminal justice system. Their research question became more empirical because they read more empirical articles about their topic. One option that they considered was to evaluate intervention programs for perpetrators of sexual assault to see if they reduced the likelihood of committing sexual assault again. Another option they considered was seeing if counties or states with higher than average jail sentences for sexual assault perpetrators had lower rates of re-offense for sexual assault. These projects addressed the ethical question of punishing perpetrators of sexual violence but did so in a way that gathered and analyzed empirical real-world data. Our job as social work researchers is to gather social facts about social work issues, not to judge or determine morality.

Key Takeaways

• Empirical questions are distinct from ethical questions.
• There are usually a number of ethical questions and a number of empirical questions that could be asked about any single topic.
• While social workers may research topics about which people have moral opinions, a researcher’s job is to gather and analyze empirical data.
• Take a look at your working question. Make sure you have an empirical question, not an ethical one. To perform this check, describe how you could find an answer to your question by conducting a study, like a survey or focus group, with real people.

9.2 Characteristics of a good research question

• Identify and explain the key features of a good research question
• Explain why it is important for social workers to be focused and clear with the language they use in their research questions

Now that you’ve made sure your working question is empirical, you need to revise that working question into a formal research question. So, what makes a good research question? First, it is generally written in the form of a question. To say that your research question is “the opioid epidemic” or “animal assisted therapy” or “oppression” would not be correct. You need to frame your topic as a question, not a statement. A good research question is also one that is well-focused. A well-focused question helps you tune out irrelevant information and not try to answer everything about the world all at once. You could be the most eloquent writer in your class, or even in the world, but if the research question about which you are writing is unclear, your work will ultimately lack direction.

In addition to being written in the form of a question and being well-focused, a good research question is one that cannot be answered with a simple yes or no. For example, if your interest is in gender norms, you could ask, “Does gender affect a person’s performance of household tasks?” but you will have nothing left to say once you discover your yes or no answer. Instead, why not ask, about the relationship between gender and household tasks. Alternatively, maybe we are interested in how or to what extent gender affects a person’s contributions to housework in a marriage? By tweaking your question in this small way, you suddenly have a much more fascinating question and more to say as you attempt to answer it.

A good research question should also have more than one plausible answer. In the example above, the student who studied the relationship between gender and household tasks had a specific interest in the impact of gender, but she also knew that preferences might be impacted by other factors. For example, she knew from her own experience that her more traditional and socially conservative friends were more likely to see household tasks as part of the female domain, and were less likely to expect their male partners to contribute to those tasks. Thinking through the possible relationships between gender, culture, and household tasks led that student to realize that there were many plausible answers to her questions about how  gender affects a person’s contribution to household tasks. Because gender doesn’t exist in a vacuum, she wisely felt that she needed to consider other characteristics that work together with gender to shape people’s behaviors, likes, and dislikes. By doing this, the student considered the third feature of a good research question–she thought about relationships between several concepts. While she began with an interest in a single concept—household tasks—by asking herself what other concepts (such as gender or political orientation) might be related to her original interest, she was able to form a question that considered the relationships  among  those concepts.

This student had one final component to consider. Social work research questions must contain a target population. Her study would be very different if she were to conduct it on older adults or immigrants who just arrived in a new country. The target population is the group of people whose needs your study addresses. Maybe the student noticed issues with household tasks as part of her social work practice with first-generation immigrants, and so she made it her target population. Maybe she wants to address the needs of another community. Whatever the case, the target population should be chosen while keeping in mind social work’s responsibility to work on behalf of marginalized and oppressed groups.

In sum, a good research question generally has the following features:

• It is written in the form of a question
• It is clearly written
• It cannot be answered with “yes” or “no”
• It has more than one plausible answer
• It considers relationships among multiple variables
• It is specific and clear about the concepts it addresses
• It includes a target population
• A poorly focused research question can lead to the demise of an otherwise well-executed study.
• Research questions should be clearly worded, consider relationships between multiple variables, have more than one plausible answer, and address the needs of a target population.

Okay, it’s time to write out your first draft of a research question.

• Once you’ve done so, take a look at the checklist in this chapter and see if your research question meets the criteria to be a good one.

Brainstorm whether your research question might be better suited to quantitative or qualitative methods.

• Describe why your question fits better with quantitative or qualitative methods.
• Provide an alternative research question that fits with the other type of research method.

9.3 Quantitative research questions

• Describe how research questions for exploratory, descriptive, and explanatory quantitative questions differ and how to phrase them
• Identify the differences between and provide examples of strong and weak explanatory research questions

Quantitative descriptive questions

The type of research you are conducting will impact the research question that you ask. Probably the easiest questions to think of are quantitative descriptive questions. For example, “What is the average student debt load of MSW students?” is a descriptive question—and an important one. We aren’t trying to build a causal relationship here. We’re simply trying to describe how much debt MSW students carry. Quantitative descriptive questions like this one are helpful in social work practice as part of community scans, in which human service agencies survey the various needs of the community they serve. If the scan reveals that the community requires more services related to housing, child care, or day treatment for people with disabilities, a nonprofit office can use the community scan to create new programs that meet a defined community need.

Quantitative descriptive questions will often ask for percentage, count the number of instances of a phenomenon, or determine an average. Descriptive questions may only include one variable, such as ours about student debt load, or they may include multiple variables. Because these are descriptive questions, our purpose is not to investigate causal relationships between variables. To do that, we need to use a quantitative explanatory question.

Quantitative explanatory questions

Most studies you read in the academic literature will be quantitative and explanatory. Why is that? If you recall from Chapter 2 , explanatory research tries to build nomothetic causal relationships. They are generalizable across space and time, so they are applicable to a wide audience. The editorial board of a journal wants to make sure their content will be useful to as many people as possible, so it’s not surprising that quantitative research dominates the academic literature.

Structurally, quantitative explanatory questions must contain an independent variable and dependent variable. Questions should ask about the relationship between these variables. The standard format I was taught in graduate school for an explanatory quantitative research question is: “What is the relationship between [independent variable] and [dependent variable] for [target population]?” You should play with the wording for your research question, revising that standard format to match what you really want to know about your topic.

Let’s take a look at a few more examples of possible research questions and consider the relative strengths and weaknesses of each. Table 9.1 does just that. While reading the table, keep in mind that I have only noted what I view to be the most relevant strengths and weaknesses of each question. Certainly each question may have additional strengths and weaknesses not noted in the table. Each of these questions is drawn from student projects in my research methods classes and reflects the work of many students on their research question over many weeks.

Making it more specific

A good research question should also be specific and clear about the concepts it addresses. A student investigating gender and household tasks knows what they mean by “household tasks.” You likely also have an impression of what “household tasks” means. But are your definition and the student’s definition the same? A participant in their study may think that managing finances and performing home maintenance are household tasks, but the researcher may be interested in other tasks like childcare or cleaning. The only way to ensure your study stays focused and clear is to be specific about what you mean by a concept. The student in our example could pick a specific household task that was interesting to them or that the literature indicated was important—for example, childcare. Or, the student could have a broader view of household tasks, one that encompasses childcare, food preparation, financial management, home repair, and care for relatives. Any option is probably okay, as long as the researcher is clear on what they mean by “household tasks.” Clarifying these distinctions is important as we look ahead to specifying how your variables will be measured in Chapter 11 .

Table 9.2 contains some “watch words” that indicate you may need to be more specific about the concepts in your research question.

It can be challenging to be this specific in social work research, particularly when you are just starting out your project and still reading the literature. If you’ve only read one or two articles on your topic, it can be hard to know what you are interested in studying. Broad questions like “What are the causes of chronic homelessness, and what can be done to prevent it?” are common at the beginning stages of a research project as working questions. However, moving from working questions to research questions in your research proposal requires that you examine the literature on the topic and refine your question over time to be more specific and clear. Perhaps you want to study the effect of a specific anti-homelessness program that you found in the literature. Maybe there is a particular model to fighting homelessness, like Housing First or transitional housing, that you want to investigate further. You may want to focus on a potential cause of homelessness such as LGBTQ+ discrimination that you find interesting or relevant to your practice. As you can see, the possibilities for making your question more specific are almost infinite.

Quantitative exploratory questions

In exploratory research, the researcher doesn’t quite know the lay of the land yet. If someone is proposing to conduct an exploratory quantitative project, the watch words highlighted in Table 9.2 are not problematic at all. In fact, questions such as “What factors influence the removal of children in child welfare cases?” are good because they will explore a variety of factors or causes. In this question, the independent variable is less clearly written, but the dependent variable, family preservation outcomes, is quite clearly written. The inverse can also be true. If we were to ask, “What outcomes are associated with family preservation services in child welfare?”, we would have a clear independent variable, family preservation services, but an unclear dependent variable, outcomes. Because we are only conducting exploratory research on a topic, we may not have an idea of what concepts may comprise our “outcomes” or “factors.” Only after interacting with our participants will we be able to understand which concepts are important.

Remember that exploratory research is appropriate only when the researcher does not know much about topic because there is very little scholarly research. In our examples above, there is extensive literature on the outcomes in family reunification programs and risk factors for child removal in child welfare. Make sure you’ve done a thorough literature review to ensure there is little relevant research to guide you towards a more explanatory question.

• Descriptive quantitative research questions are helpful for community scans but cannot investigate causal relationships between variables.
• Explanatory quantitative research questions must include an independent and dependent variable.
• Exploratory quantitative research questions should only be considered when there is very little previous research on your topic.
• Identify the type of research you are engaged in (descriptive, explanatory, or exploratory).
• Create a quantitative research question for your project that matches with the type of research you are engaged in.

Preferably, you should be creating an explanatory research question for quantitative research.

9.4 Qualitative research questions

• List the key terms associated with qualitative research questions
• Distinguish between qualitative and quantitative research questions

Qualitative research questions differ from quantitative research questions. Because qualitative research questions seek to explore or describe phenomena, not provide a neat nomothetic explanation, they are often more general and openly worded. They may include only one concept, though many include more than one. Instead of asking how one variable causes changes in another, we are instead trying to understand the experiences ,  understandings , and  meanings that people have about the concepts in our research question. These keywords often make an appearance in qualitative research questions.

Let’s work through an example from our last section. In Table 9.1, a student asked, “What is the relationship between sexual orientation or gender identity and homelessness for late adolescents in foster care?” In this question, it is pretty clear that the student believes that adolescents in foster care who identify as LGBTQ+ may be at greater risk for homelessness. This is a nomothetic causal relationship—LGBTQ+ status causes changes in homelessness.

However, what if the student were less interested in  predicting  homelessness based on LGBTQ+ status and more interested in  understanding  the stories of foster care youth who identify as LGBTQ+ and may be at risk for homelessness? In that case, the researcher would be building an idiographic causal explanation . The youths whom the researcher interviews may share stories of how their foster families, caseworkers, and others treated them. They may share stories about how they thought of their own sexuality or gender identity and how it changed over time. They may have different ideas about what it means to transition out of foster care.

Because qualitative questions usually center on idiographic causal relationships, they look different than quantitative questions. Table 9.3 below takes the final research questions from Table 9.1 and adapts them for qualitative research. The guidelines for research questions previously described in this chapter still apply, but there are some new elements to qualitative research questions that are not present in quantitative questions.

• Qualitative research questions often ask about lived experience, personal experience, understanding, meaning, and stories.
• Qualitative research questions may be more general and less specific.
• Qualitative research questions may also contain only one variable, rather than asking about relationships between multiple variables.

Qualitative research questions have one final feature that distinguishes them from quantitative research questions: they can change over the course of a study. Qualitative research is a reflexive process, one in which the researcher adapts their approach based on what participants say and do. The researcher must constantly evaluate whether their question is important and relevant to the participants. As the researcher gains information from participants, it is normal for the focus of the inquiry to shift.

For example, a qualitative researcher may want to study how a new truancy rule impacts youth at risk of expulsion. However, after interviewing some of the youth in their community, a researcher might find that the rule is actually irrelevant to their behavior and thoughts. Instead, their participants will direct the discussion to their frustration with the school administrators or the lack of job opportunities in the area. This is a natural part of qualitative research, and it is normal for research questions and hypothesis to evolve based on information gleaned from participants.

However, this reflexivity and openness unacceptable in quantitative research for good reasons. Researchers using quantitative methods are testing a hypothesis, and if they could revise that hypothesis to match what they found, they could never be wrong! Indeed, an important component of open science and reproducability is the preregistration of a researcher’s hypotheses and data analysis plan in a central repository that can be verified and replicated by reviewers and other researchers. This interactive graphic from 538 shows how an unscrupulous research could come up with a hypothesis and theoretical explanation  after collecting data by hunting for a combination of factors that results in a statistically significant relationship. This is an excellent example of how the positivist assumptions behind quantitative research and intepretivist assumptions behind qualitative research result in different approaches to social science.

• Qualitative research questions often contain words or phrases like “lived experience,” “personal experience,” “understanding,” “meaning,” and “stories.”
• Qualitative research questions can change and evolve over the course of the study.
• Using the guidance in this chapter, write a qualitative research question. You may want to use some of the keywords mentioned above.

9.5 Evaluating and updating your research questions

• Evaluate the feasibility and importance of your research questions
• Begin to match your research questions to specific designs that determine what the participants in your study will do

Feasibility and importance

As you are getting ready to finalize your research question and move into designing your research study, it is important to check whether your research question is feasible for you to answer and what importance your results will have in the community, among your participants, and in the scientific literature

Key questions to consider when evaluating your question’s feasibility include:

• Do you have access to the data you need?
• Will you be able to get consent from stakeholders, gatekeepers, and others?
• Does your project pose risk to individuals through direct harm, dual relationships, or breaches in confidentiality? (see Chapter 6 for more ethical considerations)
• Are you competent enough to complete the study?
• Do you have the resources and time needed to carry out the project?

Key questions to consider when evaluating the importance of your question include:

• Can we answer your research question simply by looking at the literature on your topic?
• How does your question add something new to the scholarly literature? (raises a new issue, addresses a controversy, studies a new population, etc.)
• How will your target population benefit, once you answer your research question?
• How will the community, social work practice, and the broader social world benefit, once you answer your research question?
• Using the questions above, check whether you think your project is feasible for you to complete, given the constrains that student projects face.
• Realistically, explore the potential impact of your project on the community and in the scientific literature. Make sure your question cannot be answered by simply reading more about your topic.

Matching your research question and study design

This chapter described how to create a good quantitative and qualitative research question. In Parts 3 and 4 of this textbook, we will detail some of the basic designs like surveys and interviews that social scientists use to answer their research questions. But which design should you choose?

As with most things, it all depends on your research question. If your research question involves, for example, testing a new intervention, you will likely want to use an experimental design. On the other hand, if you want to know the lived experience of people in a public housing building, you probably want to use an interview or focus group design.

We will learn more about each one of these designs in the remainder of this textbook. We will also learn about using data that already exists, studying an individual client inside clinical practice, and evaluating programs, which are other examples of designs. Below is a list of designs we will cover in this textbook:

• Surveys: online, phone, mail, in-person
• Experiments: classic, pre-experiments, quasi-experiments
• Interviews: in-person or via phone or videoconference
• Focus groups: in-person or via videoconference
• Content analysis of existing data
• Secondary data analysis of another researcher’s data
• Program evaluation

The design of your research study determines what you and your participants will do. In an experiment, for example, the researcher will introduce a stimulus or treatment to participants and measure their responses. In contrast, a content analysis may not have participants at all, and the researcher may simply read the marketing materials for a corporation or look at a politician’s speeches to conduct the data analysis for the study.

I imagine that a content analysis probably seems easier to accomplish than an experiment. However, as a researcher, you have to choose a research design that makes sense for your question and that is feasible to complete with the resources you have. All research projects require some resources to accomplish. Make sure your design is one you can carry out with the resources (time, money, staff, etc.) that you have.

There are so many different designs that exist in the social science literature that it would be impossible to include them all in this textbook. The purpose of the subsequent chapters is to help you understand the basic designs upon which these more advanced designs are built. As you learn more about research design, you will likely find yourself revising your research question to make sure it fits with the design. At the same time, your research question as it exists now should influence the design you end up choosing. There is no set order in which these should happen. Instead, your research project should be guided by whether you can feasibly carry it out and contribute new and important knowledge to the world.

• Research questions must be feasible and important.
• Research questions must match study design.
• Based on what you know about designs like surveys, experiments, and interviews, describe how you might use one of them to answer your research question.
• You may want to refer back to Chapter 2 which discusses how to get raw data about your topic and the common designs used in student research projects.

Media Attributions

• patrick-starfish-2062906_1920 © Inspired Images is licensed under a CC0 (Creative Commons Zero) license
• financial-2860753_1920 © David Schwarzenberg is licensed under a CC0 (Creative Commons Zero) license
• target-group-3460039_1920 © Gerd Altmann is licensed under a CC0 (Creative Commons Zero) license
• Not familiar with SpongeBob SquarePants? You can learn more about him on Nickelodeon’s site dedicated to all things SpongeBob:  http://www.nick.com/spongebob-squarepants/ ↵
• Focus on the Family. (2005, January 26). Focus on SpongeBob.  Christianity Today . Retrieved from  http://www.christianitytoday.com/ct/2005/januaryweb-only/34.0c.html ↵
• BBC News. (2005, January 20). US right attacks SpongeBob video. Retrieved from:  http://news.bbc.co.uk/2/hi/americas/4190699.stm ↵
• In fact, an MA thesis examines representations of gender and relationships in the cartoon: Carter, A. C. (2010).  Constructing gender and   relationships in “SpongeBob SquarePants”: Who lives in a pineapple under the sea . MA thesis, Department of Communication, University of South Alabama, Mobile, AL. ↵

research questions that can be answered by systematically observing the real world

unsuitable research questions which are not answerable by systematic observation of the real world but instead rely on moral or philosophical opinions

the group of people whose needs your study addresses

attempts to explain or describe your phenomenon exhaustively, based on the subjective understandings of your participants

"Assuming that the null hypothesis is true and the study is repeated an infinite number times by drawing random samples from the same populations(s), less than 5% of these results will be more extreme than the current result" (Cassidy et al., 2019, p. 233).

whether you can practically and ethically complete the research project you propose

the impact your study will have on participants, communities, scientific knowledge, and social justice

Graduate research methods in social work Copyright © 2021 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Academic Writing & Research

An online resource for students and researchers

Using Mixed Methods for Research

In the ever-evolving domain of research methodologies, a powerful and innovative approach has been gaining momentum: mixed methods research. This methodology combines both quantitative and qualitative research techniques, offering researchers a versatile toolkit to delve deeper into complex phenomena. In this post, we explore the rising use of mixed methods in research, its benefits, and how it bridges the gap between quantitative and qualitative inquiry.

Understanding Mixed Methods Research

Mixed methods research is a dynamic and flexible approach that integrates the strengths of both quantitative and qualitative methods. It allows researchers to gather, analyze, and interpret data from multiple perspectives, providing a more comprehensive understanding of a research question or phenomenon.

The Surge in Popularity

In recent years, the popularity of mixed methods research has surged across various disciplines for several compelling reasons:

1. Holistic Understanding: Mixed methods research acknowledges that not all research questions can be adequately answered using just quantitative or qualitative methods alone. Combining both approaches allows researchers to paint a more complete picture of complex phenomena.

2. Validation and Triangulation: By utilizing multiple data sources and methods, researchers can cross-validate findings. This enhances the credibility and reliability of research outcomes.

3. Contextualization: Qualitative data can provide essential context to quantitative findings. For example, survey results may indicate a correlation between variables, but qualitative data can help explain why this correlation exists and what it means in real-world terms.

4. Depth and Richness: Qualitative data, with its open-ended and exploratory nature, adds depth and richness to research. It can uncover nuances, attitudes, and experiences that quantitative data might overlook.

5. Practical Application: Mixed methods research often leads to findings that are not only academically valuable but also practically relevant. Policymakers, organizations, and practitioners can use mixed methods research to inform decision-making.

Benefits of Mixed Methods Research

1. Enhanced Validity: Combining both quantitative and qualitative data can enhance the validity of research findings. Triangulation, the process of comparing and contrasting data from different sources, helps researchers build a stronger case for their conclusions.

2. Versatility: Mixed methods research is adaptable and suitable for a wide range of research questions and topics. Researchers can tailor their approach to the specific needs of their study.

3. Complementarity: Quantitative and qualitative data often complement each other. Qualitative data can help explain or contextualize quantitative results, and quantitative data can provide empirical support for qualitative findings.

4. Rich Insights: The qualitative component of mixed methods research allows researchers to explore underlying meanings, motivations, and complexities that quantitative data alone cannot capture. This leads to richer, more nuanced insights.

5. Robustness: By using multiple data sources and methods, mixed methods research can yield robust and comprehensive findings that are less vulnerable to bias or limitations associated with a single-method approach.

Mixed methods research represents a powerful fusion of quantitative and qualitative methodologies, offering researchers a versatile toolkit to address complex research questions. Its rising popularity is a testament to its effectiveness in providing a more holistic, credible, and practical understanding of phenomena. As research continues to evolve, mixed methods research is likely to play an increasingly vital role in bridging the gap between quantitative and qualitative inquiry, unlocking new insights, and contributing to our collective knowledge

Recommended reading

Creswell, J.W. (2021) A Concise Introduction to Mixed Methods Research 2nd Edition Sage (Click to view on Amazon #Ad)

For students and researchers new to mixed methods,  A Concise Introduction to Mixed Methods Research  2e by renowned author John W. Creswell provides a brief and practical introduction to mixed methods. Many graduate students and researchers in the social, behavioural and health sciences may not have the time or resources to read long treatises or stacks of journal articles on mixed methods research. 

Glenn Stevens

Glenn is an academic writing and research specialist with 15 years experience writing, editing, PhD and Masters supervision and journal editing. He is also a qualified English teacher. His prior career was in magazine publishing. He is now editor of this blog. Contact Glenn

Share this:

Qualitative, Quantitative, and Mixed Methods Research Sampling Strategies

Sampling is a critical, often overlooked aspect of the research process. The importance of sampling extends to the ability to draw accurate inferences, and it is an integral part of qualitative guidelines across research methods. Sampling considerations are important in quantitative and qualitative research when considering a target population and when drawing a sample that will either allow us to generalize (i.e., quantitatively) or go into sufficient depth (i.e., qualitatively). While quantitative research is generally concerned with probability-based approaches, qualitative research typically uses nonprobability purposeful sampling approaches. Scholars generally focus on two major sampling topics: sampling strategies and sample sizes. Or simply, researchers should think about who to include and how many; both of these concerns are key. Mixed methods studies have both qualitative and quantitative sampling considerations. However, mixed methods studies also have unique considerations based on the relationship of quantitative and qualitative research within the study.

• Related Documents

Linking Research Questions to Mixed Methods Data Analysis Procedures 1

The purpose of this paper is to discuss the development of research questions in mixed methods studies. First, we discuss the ways that the goal of the study, the research objective(s), and the research purpose shape the formation of research questions. Second, we compare and contrast quantitative research questions and qualitative research questions. Third, we describe how to write mixed methods research questions, which we define as questions that embed quantitative and qualitative research questions. Finally, we provide a framework for linking research questions to mixed methods data analysis techniques. A major goal of our framework is to illustrate that the development of research questions and data analysis procedures in mixed method studies should occur logically and sequentially.

Mixed methods research in pedagogy: Characteristics, advantages and difficulties in application

The mixed methods research, as a new type of research, is discussed in the context of re-examining the relation between the two approaches, i.e. the possibility of not opposing, but connecting, combining and integrating them within research. The possibility to use such grounding in research as well could be quite important for pedagogy since the nature of the examined phenomena in this field is such that the majority has both quantitative and qualitative aspects. In order to explain the essence of a mixed methods research, the paper analyses its characteristics, first of all, what elements are combined, what is the nature of the relation between the combined elements and why they are combined, as well as its advantages and difficulties in application. The essential thing in a mixed methods research is the fact that combining refers to the research process as a whole, including the ontological and epistemological assumptions it is based on, which implies that the elements that are combined are understood rather broadly. The advantages and difficulties in application are considered in the context of the discussion of the possibility to combine all research elements, neutralise the limitations of quantitative and qualitative research methodology, implement the complex combining procedures etc.

How Marketers Conduct Mixed Methods Research

The complimentary nature of qualitative and quantitative research methods are examined with respect to a study assessing the market's view of a training and development institute in the Middle East. The qualitative portion consisted of focus groups conducted with seven distinct market segments served by the institute. The results proved insightful with respect to uncovering and understanding differences of opinion among the seven groups; however, taken alone, the qualitative research would have been very misleading with respect to the institute's standing in the Middle East.

Using Mixed-methods Research in Health & Education in Nepal

In the areas of health promotion and health education, mixed-methods research approach has become widely used. In mixed-methods research, also called multi-methods research, the researchers combine quantitative and qualitative research designs in a single study. This paper introduces the mixed-methods approach for use in research in health education. To illustrate this pragmatic research approach we are including an example of mixed-methods research as applied in Nepalese research.Journal of Health Promotion Vol.6 2008, p.45-48

Research Methods

<p class="MsoNormal" style="text-justify: inter-ideograph; text-align: justify; margin: 0in 34.2pt 0pt 0.5in;"><span style="font-size: 10pt;"><span style="font-family: Times New Roman;">This paper discusses three common research approaches, qualitative, quantitative, and mixed methods, along with the various research designs commonly used when conducting research within the framework of each approach. Creswell (2002) noted that quantitative research is the process of collecting, analyzing, interpreting, and writing the results of a study, while qualitative research is the approach to data collection, analysis, and report writing differing from the traditional, quantitative approaches. This paper provides a further distinction between quantitative and qualitative research methods. This paper also presents a summary of the different research methods to conduct research in quantitative, qualitative, and mixed methods studies.</span></span></p>

Distinguishing Between Quantitative and Qualitative Research: A Response to Morgan

This is a response to Morgan’s article ( Journal of Mixed Methods Research, 12(3), 268-279) on the qualitative/quantitative distinction. I argue that Morgan has mischaracterized my views on this distinction, and on the value of design typologies in mixed methods research, and that the qualitative/quantitative distinction is more productively framed on a different basis than the one he proposed.

Analysis of Community Preparedness Facing Erosion Disaster in Sambas Regency

This research was conducted to analyze the community's preparedness in the face of erosion in Sambas Regency, which is erosion caused by filing by river water. This research is a mixed methods research<em>.</em> This mixed research combines qualitative research and quantitative research, with a sample of people living in riverbank areas. The head of the local household was also asked for information on preparedness to face erosion disasters. The results of the research on community preparedness for erosion in Sambas District were based on the researchers' view that the community was not ready, people who had filled out the new research questionnaire were about 15% out of 100% who were ready. The results of interviews with residents indicated that there was no counselling, training on this preparedness was also one of the causes of the low level of community preparedness.

Kesejahteraan Anak Adopsi Usia Prasekolah (3-5 Tahun)

Child welfare is the responsibility of the family because the child is part of the family. However, in reality there are still many who neglect their children so that the children's welfare is threatened. Abandoned children need protection to ensure their survival. One of the efforts made in dealing with the problem of neglected children is through an institution-based child service program through child social service institutions. However, institution-based child services have not been optimal in realizing children's welfare. Thus, children who are in institution-based care need to be transferred to family-based care so that the child's welfare can be better. One of the permanent efforts to care for children is through adoption. The method used in this research is mixed methods research method. The design chosen in this study is Explanatory Sequential Mixed Methods, the researcher will measure the level of children's welfare with quantitative research first followed by qualitative research. The results of quantitative research regarding the welfare of preschool adopted children show that basically the welfare of adopted children is in the good category. The results of the qualitative research found that the background and reasons or motivation of adoptive parents to adopt an effect on the care of the adopted child so that the child's welfare can be better. Most adoptive parents do not yet have biological children, so the presence of adopted children is a complement to their long-awaited family. The opportunity they get for adoption makes them try to care for, nurture, and treat their adopted child very well. They always pay attention to children's physical development, children's psychological development, children's social development and children's cognitive development so that children's welfare can be achieved.

Change, Challenges, and Mixed Methods

This chapter discusses three ongoing issues related to the evaluation of qualitative research. First, the chapter considers whether a set of evaluation criteria is either determinative or changeable. Due to the evolving nature of qualitative research, it is likely that the way in which qualitative research is evaluated can change—not all at once, but gradually. Second, qualitative research has been criticized by newly resurrected positivists whose definitions of scientific research and evaluation criteria are narrow. “Politics of evidence” and a recent big-tent evaluation strategy are examined. Last, this chapter analyzes how validity criteria of qualitative research are incorporated into the evaluation of mixed methods research. The elements of qualitative research seem to be fairly represented but are largely treated as trivial. A criterion, the fit of research questions to design, is identified as distinctive in the review guide of the Journal of Mixed Methods Research.

Mixed Methods Research in Developing Country Contexts: Lessons From Field Research in Six Countries Across Africa and the Caribbean

Mixed methods research in developing countries has been increasing since the turn of the century. Given this, there is need to consolidate insights for future researchers. This article contributes to the methodological literature by exploring how cultural factors and logistical challenges in developing contexts interplay with mixed methods research design and implementation. Insights are based on the author’s research experience of using mixed methods in six projects across three African and three Caribbean countries. Three lessons are provided to aid researchers using mixed methods working in developing countries. First, cultural factors call for more reflexivity. Second, adopting a pragmatic research paradigm is necessary. And third, the research process should be iterative and adaptive.

Export Citation Format

Share document.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Nepal J Epidemiol
• v.12(1); 2022 Mar

The Growing Importance of Mixed-Methods Research in Health

Sharada prasad wasti.

1,2 School of Human and Health Sciences, University of Huddersfield, United Kingdom

Padam Simkhada

3 Centre for Midwifery, Maternal and Perinatal Health, Bournemouth University, Bournemouth, United Kingdom

Edwin R. van Teijlingen

Brijesh sathian.

4 Geriatrics and long term care Department, Rumailah Hospital, Hamad Medical Corporation, Doha, Qatar

Indrajit Banerjee

5 Sir Seewoosagur Ramgoolam Medical College, Belle Rive, Mauritius

All authors have made substantial contributions to all of the following: (1) the conception and design of the study (2) drafting the article or revising it critically for important intellectual content, (3) final approval of the version to be submitted

There is no conflict of interest for any author of this manuscript.

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sector.

This paper illustrates the growing importance of mixed-methods research to many health disciplines ranging from nursing to epidemiology. Mixed-methods approaches requires not only the skills of the individual quantitative and qualitative methods but also a skill set to bring two methods/datasets/findings together in the most appropriate way. Health researchers need to pay careful attention to the ‘best’ approach to designing, implementing, analysing, integrating both quantitative (number) and qualitative (word) information and writing this up in a way offers greater insights and enhances its applicability. This paper highlights the strengths and weaknesses of mixed-methods approaches as well as some of the common mistakes made by researchers applying mixed-methods for the first time.

Quantitative and qualitative research methods each address different types of questions, collect different kinds of data and deliver different kinds of answers. Each set of methods has its own inherent strengths and weaknesses, and each offers a particular approach to address specific types of research questions (and agendas). Health disciplines such as dentistry, nursing, speech and language therapy, and physiotherapy often use either quantitative or qualitative research methods on their own. However, there is a steadily growing literature showing the advantages of mixed-methods research is used in the health care and health service field [ 1-2 ]. Although we have advocated the use of mixed-methods in this journal eight years ago [ 3 ], there is still not enough mixed-methods research training in the health research field, particularly for health care practitioners, such as nurses, physiotherapists, midwives, and doctors, wanting to do research. Mixed-methods research has been popular in the social sciences since the twentieth century [ 4 ], and it has been growing in popularity among healthcare professionals [ 5 ], although it is still underdeveloped in disciplines such nursing and midwifery [ 6 , 7 ].

Underpinning philosophies

To help understand that mixed-methods research is not simply employing two different methods in the same study, one needs to consider their underpinning research philosophies (also called paradigms). First, quantitative research is usually underpinned by positivism. This includes most epidemiological studies; such research is typically based on the assumption that there is one single real world out there that can be measured. For example, quantitative research would address the question “What proportion of the population of India drinks coffee?” Secondly, qualitative research is more likely to be based on interpretivism. This includes research based on interviews and focus groups, research which us is typically based on the assumption that we all experience the world differently. Since we all live in a slightly different world in our heads the task of qualitative research is to analyse the interpretations of the people in the sample. For example, qualitative research would address the question “How do people experience drinking coffee in India?”, and “What does drinking coffee mean to them?”

Mixed-methods research brings together questions from two different philosophies in what is being referred to as the third path [ 8 ], third research paradigm [ 9 , 10 ], the third methodology movement [ 11 , 12 ] and pragmatism [ 5 ]. The two paradigms differ in key underlying assumptions that ultimately lead to choices in research methodology and methods and often give a breadth by answering more complicated research questions [ 4 ]. The roles of mixed-methods are clear in an understanding of the situation (the what), meaning, norms, values (the why or how) within a single research question which combine the strength of two different method and offer multiple ways of looking at the research question [ 13 ]. Epidemiology sits strongly in the quantitative research corner, with a strong emphasis on large data sets and sophisticated statistical analysis. Although the use of mixed methods in health research has been discussed widely researchers raised concerns about the explanation of why and how mixed methods are used in a single research question [ 5 ].

The relevance of mixed-methods in health research

The overall goal of the mixed-methods research design is to provide a better and deeper understanding, by providing a fuller picture that can enhance description and understanding of the phenomena [ 4 ]. Mixed-methods research has become popular because it uses quantitative and qualitative data in one single study which provides stronger inference than using either approach on its own [ 4 ]. In other words, a mixed-methods paper helps to understand the holistic picture from meanings obtained from interviews or observation to the prevalence of traits in a population obtained from surveys, which add depth and breadth to the study. For example, a survey questionnaire will include a limited number of structured questions, adding qualitative methods can capture other unanticipated facets of the topic that may be relevant to the research problem and help in the interpretation of the quantitative data. A good example of a mixed-methods study, it one conducted in Australia to understand the nursing care in public hospitals and also explore what factors influence adherence to nursing care [ 14 ]. Another example is a mixed-methods study that explores the relationship between nursing care practices and patient satisfaction. This study started with a quantitative survey to understand the general nursing services followed by qualitative interviews. A logistic regression analysis was performed to quantify the associations between general nursing practice variables supplemented with a thematic analysis of the interviews [ 15 ]. These research questions could not be answered if the researchers had used either qualitative or quantitative alone. Overall, this fits well with the development of evidence-based practice.

Despite the strengths of mixed-methods research but there is not much of it in nursing and other fields [ 7 ]. A recent review paper shows that the prevalence of mixed-methods studies in nursing was only 1.9% [ 7 ]. Similarly, a systematic review synthesised a total of 20 papers [ 16 ], and 16 papers [ 17 ] on nursing-related research paper among these only one mixed-methods paper was identified. Worse, a further two mixed-methods review recently revealed that out of 48 [ 18 , 19 ] synthesised nursing research papers, not one single mixed-methods paper was identified. This clearly depicts that mixed-methods research is still in its infancy stage in nursing but we can say there is huge scope to implement it to understand research questions on both sides of coin [ 4 ]. Therefore, there is a great need for mixed-methods training to enhance the evidence-based decision making in health and nursing practices.

Strengths and weaknesses of mixed-methods

There are several challenges in identifying expertise of both methods and in working with a multidisciplinary, interdisciplinary, or transdisciplinary team [ 20 ]. It increases costs and resources, takes longer to complete as mixed-methods design often involves multiple stages of data collection and separate data analysis [ 4 , 5 ]. Moreover, conducting mixed-methods research does not necessarily guarantee an improvement in the quality of health research. Therefore, mixed-methods research is only appropriate when there are appropriate research questions [ 4 , 6 ].

Identifying an appropriate mixed-methods journal can also be challenging when writing mixed-methods papers [ 21 ]. Mixed-methods papers need considerably more words than single-methods papers as well as sympathetic editors who understand the underlying philosophy of a mixed-methods approach. Such papers, simply require more words. The mixed-methods researcher must be reporting two separate methods with their own characteristics, different samples, and ways of analysing, therefore needs more words to describe both methods as well as both sets of findings. Researcher needs to find a journal that accepts longer articles to help broaden existing evidence-based practice and promote its applicability in the nursing field [ 22 ].

Common mistakes in applying mixed-methods

Not all applied researchers have insight into the underlying philosophy and/or the skills to apply each set of methods appropriately. Younas and colleagues’ review identified that around one-third (29%) of mixed-methods studies did not provide an explicit label of the study design and 95% of studies did not identify the research paradigm [ 7 ]. Whilst several mixed-methods publications did not provide clear research questions covering both quantitative and qualitative approaches. Another common issue is how to collect data either concurrent or sequential and the priority is given to each approach within the study where equal or dominant which are not clearly stated in writing which is important to mention while writing in the methods section. Similarly, a commonly overlooked aspect is how to integrate both findings in a paper. The responsibility lies with the researcher to ensure that findings are sufficiently plausible and credible [ 4 ]. Therefore, intensive mixed-methods research training is required for nursing and other health practitioners to ensure its appropriate.

The way forward

Despite the recognised strengths and benefits of doing mixed-methods research, there is still only a limited number of nursing and related-health research publications using such this approach. Researchers need training in how to design, conduct, analyse, synthesise and disseminate mixed-methods research. Most importantly, they need to consider appropriate research questions that can be addressed using a mixed methods approach to add to our knowledge in evidence-based practice. In short, we need more training on mixed-methods research for a range of health researchers and health professionals.

Acknowledgement

• Open access
• Published: 16 August 2024

Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat

• Joseph J. Gallo 1 ,
• Sarah M. Murray 1 ,
• John W. Creswell 2 ,
• Charles Deutsch 3 &
• Timothy C. Guetterman 2  

BMC Medical Education volume  24 , Article number:  882 ( 2024 ) Cite this article

Metrics details

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. The focused goal was to evaluate online training by comparing the self-rated skills of scholars who experienced an in-person retreat to scholars in an online retreat in specific domains of mixed methods research for the health sciences from 2015–2023.

The authors administered a scholar Mixed Methods Skills Self-Assessment instrument based on an educational competency scale that included domains on: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination” to participants of the Mixed Methods Research Training Program for the Health Sciences (MMRTP). Self-ratings on confidence on domains were compared before and after retreat participation within cohorts who attended in person ( n  = 73) or online ( n  = 57) as well as comparing across in-person to online cohorts. Responses to open-ended questions about experiences with the retreat were analyzed.

Scholars in an interactive program to improve mixed methods skills reported significantly increased confidence in ability to define or explain concepts and in ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars in the online retreat had self-rated skill improvements as good or better than scholars who participated in person. With the possible exception of networking, scholars found the online format was associated with advantages such as accessibility and reduced burden of travel and finding childcare. No differences in difficulty of learning concepts was described.

Conclusions

Keeping in mind that the retreat is only one component of the MMRTP, this study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Peer Review reports

Introduction

The coronavirus pandemic accelerated interest in distance or remote learning. While the acute nature of the pandemic has abated, changes in the way people work have largely remained, with hybrid conferences and trainings more commonly implemented now than during the pre-pandemic period. Studies of health-related online teaching have focused on medical students [ 1 , 2 , 3 ], health professionals [ 4 , 5 ], and medical conferences [ 6 , 7 , 8 ] and have touted the advantages of virtual training and conferences in health education, but few studies have assessed relative growth in skills and competencies in health research methods for synchronous online vs. in-person training.

The National Institutes of Health (NIH)-funded Mixed Methods Research Training Program (MMRTP) for the Health Sciences provided training to faculty-level investigators across health disciplines from 2015–2023. The NIH is a major funder of health-related research in the United States. Its institutes span diseases and conditions (e.g., mental health, environmental health) in addition to focus areas (e.g., minority health and health disparities, nursing) and developing research capacity. Scholars in the MMRTP seek to develop skills in mixed methods research through participation in a summer retreat followed by ongoing mentorship for one year from a mixed methods expert matched to the scholar to support their development of a research proposal. Webinars leading up to the retreat include didactic sessions taught by the same faculty each year, and the retreat itself contains multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal. Due to pandemic restrictions on gatherings and travel, in 2020 the MMRTP retained all components of the program but transitioned the in-person retreat to a synchronous online retreat.

The number of NIH agencies funding mixed methods research increased from 23 in 1997–2008 to 36 in 2009–2014 [ 9 ]. The usefulness of mixed methods research aligns with several Institutes’ strategic priories, including improving health equity, enhancing feasibility, acceptability, and sustainability of interventions, and addressing patient-centeredness. However, there is a tension between growing interest in mixed methods for health sciences research and a lack of training for investigators to acquire mixed methods research skills. Mixed methods research is not routinely taught in doctoral programs, institutional grant-writing programs, nor research training that academic physicians receive. The relative lack of researchers trained in mixed methods research necessitates ongoing research capacity building and mentorship [ 10 ]. Online teaching has the potential to meet growing demand for training and mentoring in mixed methods, as evidenced by the growth of online offerings by the Mixed Methods International Research Association [ 11 ]. Yet, the nature of skills and attitudes required for doing mixed methods research, such as integration of quantitative and qualitative data collection, analysis, and epistemologies, may make this type of training difficult to adapt to an online format without compromising its effectiveness.

Few studies have attempted to evaluate mixed methods training [ 12 , 13 , 14 , 15 ] and none appear to have evaluated online trainings in mixed methods research. Our goal was to evaluate our online MMRTP by comparing the self-rated skills of scholars who experienced an in-person retreat to an online retreat across specific domains. While the MMRTP retreat is only one component of the program, assessment before and after the retreat among persons who experienced the synchronous retreat online compared to in-person provides an indication of the effectiveness of online instruction in mixed methods for specific domains critical to the design of research in health services. We hypothesized that scholars who attended the retreat online would exhibit improvements in self-rated skills comparable to scholars who attended in person.

Participants

Five cohorts with a total of 73 scholars participated in the MMRTP in person (2015–2019), while four cohorts with a total of 57 scholars participated online (2020–2023). Scholars are faculty-level researchers in the health sciences in the United States. The scholars are from a variety of disciplines in the health sciences; namely, pediatrics, psychiatry, general medicine, oncology, nursing, human development, music therapy, nutrition, psychology, and social work.

The mixed methods research training program

Formal program activities include two webinars leading up to a retreat followed by ongoing mentorship support. The mixed methods content taught in webinars and the retreat is informed by a widely used textbook by Creswell and Plano Clark [ 18 ] in addition to readings on methodological topics and the practice of mixed methods. The webinars introduce mixed methods research and integration concepts, with the goal of imparting foundational knowledge and ensuring a common language. Specifically, the first webinar introduces mixed methods concepts, research designs, scientific rigor, and becoming a resource at one’s institution, while the second focuses on strategies for the integration of qualitative and quantitative research. Retreats provide an active workshop blending lectures, one-on-one meetings, and interactive faculty-led small workgroups. In addition to scholars, core program faculty who serve as investigators and mentors for the MMRTP, supplemented with consultants and former scholars, lead the retreat. The retreat has covered the state-of-the-art topics within the context of mixed methods research: rationale for use of mixed methods, procedural diagrams, study aims, use of theory, integration strategies, sampling strategies, implementation science, randomized trials, ethics, manuscript and proposal writing, and becoming a resource at one’s home institution. In addition to lectures, the retreat includes multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal and is expected to make revisions based on feedback and lectures.

Scholars are matched for one year with a mentor based on the Scholar’s needs, career level, and area of health research from a national list of affiliated experienced mixed methods investigators with demonstrated success in obtaining independent funding for research related to the health sciences and a track record and commitment to mentoring. The purpose of this arrangement is to provide different perspectives on mixed methods design while also providing specific feedback on the scholar's research proposal, reviewing new ideas, and together developing a strategy and timeline for submission.

From 2015–2019 (in-person cohorts) the retreat was held over 3 days at the Johns Hopkins University Bloomberg School of Public Health (in 2016 Harvard Catalyst, the Harvard Clinical and Translational Science Center, hosted the retreat at Harvard Medical School). Due to pandemic restrictions, from 2020–2023 the retreat activities were conducted via Zoom with the same number of lecture sessions (over 3 days in 2020 and 4 days thereafter). We made adaptations for the online retreat based on continuous feedback from attendees. We had to rapidly transition to online in 2020 with the same structure as in person, but feedback from scholars led us to extend the retreat to 4 days online from 2021–2023. The extra day allowed for more breaks from Zoom sessions with time for scholars to consider feedback from small groups and to have one-on-one meetings with mentors. Discussion during interactive presentations was encouraged and facilitated by using breakout rooms at breaks mid-presentation. Online resources were available to participants through CoursePlus, the teaching and learning platform used for courses at the Johns Hopkins Bloomberg School of Public Health, hosting publications, presentation materials, recordings of lectures, sharing proposals, email, and discussion boards that scholars have access to before, during, and after the retreat.

Measurement strategy

Before and after the retreat in each year, we distributed a self-administered scholar Mixed Methods Skills Self-Assessment instrument (Supplement 1) to all participating scholars [ 15 ]; we have reported results from this pre-post assessment for the first two cohorts [ 14 ]. The Mixed Methods Skills Self-Assessment instrument has been previously used and has established reliability for the total items (α = 0.95) and evidence of criterion-related validity between experiences and ability ratings [ 15 ]. In each year, the pre-assessment is completed upon entry to the program, approximately four months prior to the retreat, and the post-assessment is administered two weeks after the retreat. The instrument consists of three sections: 1) professional experiences with mixed methods, including background, software, and resource familiarity; 2) a quantitative, qualitative, and mixed methods skills self-assessment; and 3) open-ended questions focused on learning goals for the MMRTP. The skills assessment contains items for each of the following domains: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination.” Each skill was assessed via three items drawn from an educational competency ratings scale that ask scholars to rate: [ 16 ] “My ability to define/explain,” “My ability to apply to practical problems,” and “Extent to which I need to improve my skill.” Response options were on a five-point Likert-type scale that ranged from “Not at all” (coded ‘1’) to “To a great extent” (coded ‘5’), including a mid-point [ 17 ]. We took the mean of the scholar’s item ratings over all component items within each domain (namely, “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination”).

Open-ended questions

The baseline survey included two open-ended prompts: 1) What skills and goals are most important to you?, and 2) What would you like to learn? The post-assessment survey also included two additional open-ended questions about the retreat: 1) What aspects of the retreat were helpful?, and 2) What would you like to change about the retreat? In addition, for the online cohorts (2020–2023), we wanted to understand reactions to the online training and added three questions for this purpose: (1) In general, what did you think of the online format for the MMRTP retreat?, 2) What mixed methods concepts are easier or harder to learn virtually?, and 3) What do you think was missing from having the retreat online rather than in person?

Data analysis

Our evaluation employed a convergent mixed methods design [ 18 ], integrating an analysis of ratings pre- and post-retreat with analysis of open-ended responses provided by scholars after the retreat. Our quantitative analysis proceeded in 3 steps. First, we analyzed item-by-item baseline ratings of the extent to which scholars thought they “need to improve skills,” stratified into two groups (5 cohorts who attended in-person and 4 cohorts who attended online). The purpose of comparing the two groups at baseline on learning needs was to assess how similar the scholars in the in-person or online groups were in self-assessment of learning needs before attending the program. Second, to examine the change in scholar ratings of ability to “define or explain a concept” and in their ability to “apply to practical problems,” from before to after the retreat, we conducted paired t-tests. The goal was to compare the ratings before and after the retreat among scholars who attended the program in person to scholars who attended online. Third, we compared post-retreat ratings among in-person cohorts to online cohorts to gauge the effectiveness of the online training. We set statistical significance at α  < 0.05 as a guide to inference. We calculated Cohen’s d as a guide to the magnitude of differences [ 19 ]. SPSS Version 28 was employed for all analyses.

We analyzed qualitative data using a thematic analysis approach that consisted of reviewing all open-ended responses, conducting open coding based on the data, developing and refining a codebook, and identifying major themes [ 20 ]. We then compared the qualitative results for the in-person versus online cohorts to understand any thematic differences concerning retreat experiences and reactions.

Background and experiences of scholars

Scholars in the in-person ( n  = 59, 81%) and online ( n  = 52, 91%) cohorts reported their primary training was quantitative rather than qualitative or mixed methods, and scholars across cohorts commonly reported at least some exposure to mixed methods research (Table  1 ). However, most scholars did not have previous mixed methods training with 17 (23%) and 16 (28%) of the in-person and online cohorts, respectively, having previously completed a mixed methods course. While experiences were similar across in-person vs. online cohorts, there were two areas in which the scholars reported a statistically significant difference: a larger portion of the online cohorts reported writing a mixed methods application that received funding ( n  = 35, 48% in person; n  = 46, 81% online), and a smaller proportion of the online cohorts had given a local or institutional mixed methods presentation ( n  = 32, 44% in person; n  = 15, 26% online).

Self-identified need to improve skills in mixed methods

At baseline, scholars rated the extent to which they needed to improve specific mixed methods skills (Table  2 ). Overall, scholars endorsed a strong need to improve all mixed methods skills. The ratings between the in-person and online cohorts were not statistically significant for any item.

Change in self-ratings of skills after the retreat

Within cohorts.

For all domains, the differences in pre-post assessment scores were statistically significant for both the in-person and online cohorts in ability to define or explain concepts and to apply concepts to practical problems (left side of Table  3 ). In other words, on average scholars improved in both in-person and online cohorts.

Across cohorts

Online cohorts had significantly better self-ratings after the retreat than did in-person cohorts in ability to define or explain concepts and to apply concepts to practical problems (in sampling, data collection, analysis, and dissemination) but no significant differences in research questions and design / approach (rightmost column of Table  3 ).

Scholar reflections about online and in-person retreats

Goals of training.

In comparing in-person to online cohorts, discussions of the skills that scholars wanted to improve had no discernable differences. Scholars mentioned wanting to develop skills in the foundations of mixed methods research, how to write competitive proposals for funding, the use of the terminology of mixed methods research, and integrative analysis. In addition, some scholars expressed wanting to become a resource at their own institutions and providing training and mentoring to others.

Small group sessions

Scholars consistently reported appreciating being able to talk through their project and gaining feedback from experts in small group sessions. Some scholars expressed a preference for afternoon small group sessions, “The small group sessions felt the most helpful, but only because we can apply what we were learning from the morning lecture sessions” (online cohort 9). How participants discussed the benefits of the small group sessions or how they used the sessions did not depend on whether they had experienced the session in person or online.

Online participants described a tradeoff between the accessibility of a virtual retreat versus advantages of in-person training. One participant explained, “I liked the online format, as I do not have reliable childcare” (online cohort 8). Many of the scholars felt that there was an aspect of networking missing when the retreat was held fully online. As one scholar described, when learning online they, “miss getting to know the other fellows and forming lasting connections” (online cohort 9). However, an equal number of others reported that having a virtual retreat meant less hassle; for instance, they were able to join from their preferred location and did not have to travel. Some individuals specifically described the tradeoff of fewer networking opportunities for ease of attendance. One scholar wrote, being online “certainly loses some of the perks of in person connection building but made it equitable to attend” (online cohort 8).

Learning online

No clear difference in ease of learning concepts was described. A scholar explained: “Learning most concepts is essentially the same virtually versus in person” (online cohort 8). However, scholars described some concepts as easier to learn in one modality versus the other, for example, simpler concepts being more suited to learning virtually while complex concepts were better suited to in-person learning. There was notable variation though in the topics which scholars considered to be simple versus complex. For instance, one scholar noted that “I suppose developing the joint displays were a bit tougher virtually since you were not literally elbow to elbow” (online cohort 7) while another explained, “joint displays lend themselves to the zoom format” (online cohort 8).

Integrating survey responses and scholar reflections

In-person and online cohorts were comparable in professional experiences and ratings of the need to improve skills before attending the retreat, sharpening the focus on differences in self-rated skills associated with attendance online compared to in person. If anything, online attendees rated skills as good or better than in-person attendees. Open-ended questions revealed that, for the most part, scholar reflections on learning were similar across in-person and online cohorts. Whether learning the concept of “mixed methods integration” was more difficult online was a source of disagreement. Online attendance was associated with numerous advantages, and small group sessions were valued, regardless of format. Taken together, the evidence from nine cohorts shows that the online retreat was acceptable and as effective in improving self-rated skills as meeting in person.

Mixed methods have become indispensable to health services research from intervention development and testing [ 21 ] to implementation science [ 22 , 23 , 24 ]. We found that scholars participating in an interactive program to improve mixed methods skills reported significantly increased confidence in their ability to define or explain concepts and in their ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars who participated in the online retreat had self-rated skill improvements as good or better than scholars who participated in person, and these improvements were relatively large as indicated by the Cohen’s d estimates. The online retreat appeared to be effective in increasing confidence in the use of mixed methods research in the health sciences and was acceptable to scholars. Our study deserves attention because the national need is so great for investigators with training in mixed methods to address complex behavioral health problems, community- and patient-centered research, and implementation research. No program has been evaluated as we have done here.

Aside from having written a funded mixed methods proposal, the online compared to earlier in person cohorts were comparable in experiences and need to improve specific skills. Within each cohort, scholars reported significant gains in self-rated skills on their ability to “define or explain” a concept and on their ability to “apply to practical problems” in domains essential to mixed methods research. However, consistent with our hypothesis that online training would be as effective as in person we found that online scholars reported better improvement in self-ratings in ability to define or explain concepts and to apply concepts to practical problems in sampling, data collection, analysis, and dissemination but no significant differences in research questions and design / approach. Better ratings in online cohorts could reflect differences in experience with mixed methods, secular changes in knowledge and availability of resources in mixed methods, and maturation of the program facilitated by continued modifications based on feedback from scholars and participating faculty [ 13 , 14 , 15 ].

Ratings related to the “analysis” domain, which includes the central concept of mixed methods integration, deserve notice since scholars rated this skill well below other domains at baseline. While both in-person and online cohorts improved after the retreat, and online cohorts improved substantially more than in-person cohorts, ratings for analysis after the retreat remained lower than for other domains. Scholars consistently have mentioned integration as a difficult concept, and our analysis here is limited to the retreat alone. Continued mentoring one year after the retreat and work on their proposal is built in to the MMRTP to enhance understanding of integration.

Several reviews point out the advantages of online training including savings in time, money, and greenhouse emissions [ 1 , 7 , 8 ]. Online conferences may increase the reach of training to international audiences, improve the diversity of speakers and attendees, facilitate attendance of persons with disabilities, and ease the burden of finding childcare [ 1 , 8 , 25 ]. Online training in health also appears to be effective [ 2 , 4 , 5 , 25 ], though studies are limited because often no skills were evaluated, no comparison groups were used, the response rate was low, or the sample size was small [ 1 , 6 ]. With the possible exception of networking, scholars found the online format was associated with advantages, including saving travel, maintaining work-family balance, and learning effectively. As scholars did discuss perceived increase in difficulty networking, deliberate effort needs to be directed at enhancing collaborations and mentorship [ 8 ]. The MMRTP was designed with components to facilitate networking during and beyond the retreat (e.g., small group sessions, one-on-one meetings, working with a consultant on a specific proposal).

Limitations of our study should be considered. First, the retreat was only one of several components of a mentoring program for faculty in the health sciences. Second, in-person and online cohorts represent different time periods spanning 9 years during which mixed methods applications to NIH and other funders have been increasing [ 9 ]. Third, the pre- and post-evaluations of ability to explain or define concepts, or to apply the concepts to practical problems, were based on self-report. Nevertheless, the pre-post retreat survey on self-rated skills uses a skills self-assessment form we developed [ 15 ], drawing from educational theory related to the epistemology of knowledge [ 26 , 27 ].

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. Our study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Mixed Methods Research Training Program

Wilcha RJ. Effectiveness of Virtual Medical Teaching During the COVID-19 Crisis: Systematic Review. JMIR Med Educ. 2020;6(2):e20963.

Article   Google Scholar  

Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Medical Education Online. 2019;24(1) https://doi.org/10.1080/10872981.2019.1666538

Barche A, Nayak V, Pandey A, Bhandarkar A, Nayak K. Student perceptions towards online learning in medical education during the COVID-19 pandemic: a mixed-methods study. F1000Res. 2022;11:979. https://doi.org/10.12688/f1000research.123582.1 .

Ebner C, Gegenfurtner A. Learning and Satisfaction in Webinar, Online, and Face-to-Face Instruction: A Meta-Analysis. Frontiers in Education. 2019;4(92) https://doi.org/10.3389/feduc.2019.00092

Randazzo M, Preifer R, Khamis-Dakwar R. Project-Based Learning and Traditional Online Teaching of Research Methods During COVID-19: An Investigation of Research Self-Efficacy and Student Satisfaction. Frontiers in Education. 2021;6(662850) https://doi.org/10.3389/feduc.2021.662850

Chan A, Cao A, Kim L, et al. Comparison of perceived educational value of an in-person versus virtual medical conference. Can Med Educ J. 2021;12(4):65–9. https://doi.org/10.36834/cmej.71975 .

Rubinger L, Gazendam A, Ekhtiari S, et al. Maximizing virtual meetings and conferences: a review of best practices. Int Orthop. 2020;44(8):1461–6. https://doi.org/10.1007/s00264-020-04615-9 .

Sarabipour S. Virtual conferences raise standards for accessibility and interactions. Elife. Nov 4 2020;9 https://doi.org/10.7554/eLife.62668

Coyle CE, Schulman-Green D, Feder S, et al. Federal funding for mixed methods research in the health sciences in the United States: Recent trends. J Mixed Methods Res. 2018;12(3):1–20.

Poth C, Munce SEP. Commentary – preparing today’s researchers for a yet unknown tomorrow: promising practices for a synergistic and sustainable mentoring approach to mixed methods research learning. Int J Multiple Res Approaches. 2020;12(1):56–64.

Creswell JW. Reflections on the MMIRA The Future of Mixed Methods Task Force Report. J Mixed Methods Res. 2016;10(3):215–9. https://doi.org/10.1177/1558689816650298 .

Hou S. A Mixed Methods Process Evaluation of an Integrated Course Design on Teaching Mixed Methods Research. Int J Sch Teach Learn. 2021;15(2):Article 8. https://doi.org/10.20429/ijsotl.2021.150208 .

Guetterman TC, Creswell J, Deutsch C, Gallo JJ. Process Evaluation of a Retreat for Scholars in the First Cohort: The NIH Mixed Methods Research Training Program for the Health Sciences. J Mix Methods Res. 2019;13(1):52–68. https://doi.org/10.1177/1558689816674564 .

Guetterman T, Creswell JW, Deutsch C, Gallo JJ. Skills Development and Academic Productivity of Scholars in the NIH Mixed Methods Research Training Program for the Health Sciences (invited publication). Int J Multiple Res Approach. 2018;10(1):1–17.

Guetterman T, Creswell JW, Wittink MN, et al. Development of a Self-Rated Mixed Methods Skills Assessment: The NIH Mixed Methods Research Training Program for the Health Sciences. J Contin Educ Health Prof. 2017;37(2):76–82.

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. AACE; 2007:3053–3055.

DeVellis RF. Scale development: Theory and applications. 3rd ed. Sage; 2012.

Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Sage Publications; 2017.

Cohen J. Statistical power analysis for the behavioral sciences. 3rd ed. Academic Press; 1988.

Boeije H. A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Qual Quant. 2002;36:391–409.

Aschbrenner KA, Kruse G, Gallo JJ, Plano Clark VL. Applying mixed methods to pilot feasibility studies to inform intervention trials. Pilot Feasibility Stud. 2022;8(1):217–24. https://doi.org/10.1186/s40814-022-01178-x .

Palinkas LA. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

Albright K, Gechter K, Kempe A. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad Pediatr Sep-Oct. 2013;13(5):400–7. https://doi.org/10.1016/j.acap.2013.06.010 .

Palinkas L, Aarons G, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed methods designs in implementation research. Adm Policy Ment Health. 2011;38:44–53.

Ni AY. Comparing the Effectiveness of Classroom and Online Learning: Teaching Research Methods. J Public Affairs Educ. 2013;19(2):199–215. https://doi.org/10.1080/15236803.2013.12001730 .

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. presented at: Society for Information Technology & Teacher Education International Conference; March 26, 2007 2007; San Antonio, Texas.

Guetterman TC. What distinguishes a novice from an expert mixed methods researcher? Qual Quantity. 2017;51:377–98.

Download references

Acknowledgements

The Mixed Methods Research Training Program is supported by the Office of Behavioral and Social Sciences Research under Grant R25MH104660. Participating institutes are the National Institute of Mental Health, National Heart, Lung, and Blood Institute, National Institute of Nursing Research, and the National Institute on Aging.

Author information

Authors and affiliations.

Johns Hopkins University, Baltimore, MD, USA

Joseph J. Gallo & Sarah M. Murray

University of Michigan, Ann Arbor, MI, USA

John W. Creswell & Timothy C. Guetterman

Harvard University, Boston, MA, USA

Charles Deutsch

You can also search for this author in PubMed   Google Scholar

Contributions

All authors conceptualized the design of this study. TG analyzed the scholar data in evaluation of the program. TG and JG interpreted results and were major contributors in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Timothy C. Guetterman .

Ethics declarations

Ethics approval and consent to participate.

The program was reviewed by the Johns Hopkins Institutional Review Board and was deemed exempt as educational research under United States 45 CFR 46.101(b), Category (2). Data were collected through an anonymous survey. Consent to participate was waived.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gallo, J.J., Murray, S.M., Creswell, J.W. et al. Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat. BMC Med Educ 24 , 882 (2024). https://doi.org/10.1186/s12909-024-05877-2

Download citation

Received : 15 January 2024

Accepted : 08 August 2024

Published : 16 August 2024

DOI : https://doi.org/10.1186/s12909-024-05877-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research training
• Mixed methods research
• Research capacity building
• Online education
• Teaching methods

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A review on multi-objective mixed-integer non-linear optimization programming methods.

1. Introduction

1.1. problem formulation, 1.2. applications, 1.3. state-of-the-art of review articles, 1.4. contribution, 1.5. procedure and approach, 1.6. paper organization, 2. exact methods, 2.1. scalarization methods, 2.1.1. the weighted sum method, 2.1.2. the ϵ -constraint method, 2.1.3. pascoletti–serafini scalarization, 2.1.4. normal boundary intersection (nbi), 2.2. multi-criteria branch and bound, 2.2.1. branching, 2.2.2. bounding and fathoming, 2.2.3. discussion, 2.3. drawbacks and conclusions, 3. approximate methods, 3.1. metaheuristics overview.

• Dominance-based metaheuristics: These metaheuristics employ the concept of Pareto dominance as the primary selection criterion to guide the multi-objective search process.
• Decomposition-based metaheuristics: These metaheuristics decompose the problem into multiple scalar sub-problems and optimize them simultaneously.
• Indicator-based metaheuristics: These metaheuristics prefer to use indicators to guide their search process.

Click here to enlarge figure

3.2. Drawbacks and Discussion

3.2.1. best metaheuristic, 3.2.2. metaheuristics suitability for mo-minlp problems, 4. hybrid methods, 4.1. hybrids classifications, 4.1.1. metaheuristics with metaheuristics.

• Hybridization to enhance search aggressiveness: The objective here is to improve the exploration capability of the approximation procedure. For instance, a common approach is to combine a multi-objective evolutionary algorithm (EA) with a neighbor search algorithm. This combination aims to refine promising solutions obtained from evolutionary operators and maximize their quality.
• Hybridization for guided search: In this category, the goal is to incorporate a guiding mechanism along the non-dominated frontier. Population-based metaheuristics like multi-objective genetic algorithms (GAs) are utilized to extract global information about the current approximation. This information then guides local search processes, ensuring a thorough coverage of the non-dominated frontier.
• Hybridization for leveraging complementary strengths: Here, the focus is on harnessing the complementary strengths of different metaheuristics. For example, a GA algorithm may be employed initially to generate a diverse set of high-quality solutions, followed by the application of an aggressive search method (e.g., tabu search) to further refine the solutions.

4.1.2. Metaheuristics with Exact Methods

• Metaheuristic-based upper bound generation: This approach involves executing a multi-objective metaheuristic to obtain an approximation of the Pareto set. The obtained approximation is then used to initialize a multi-objective exact algorithm. By employing a branch and bound algorithm, a significant number of nodes in the search tree can be pruned, improving the overall efficiency of the hybrid approach.
• Exact algorithm for solving subproblems: In this hybrid strategy, a multi-objective exact algorithm is utilized to solve subproblems that are generated by the multi-objective metaheuristic. Leveraging the strengths of the exact algorithm, these subproblems can be solved optimally or near-optimally.

4.2. Combining B&B with Metaheuristics

• Metaheuristic for generating an upper bound.
• Exact algorithm for exploring very large neighborhoods.
• Exact algorithm for solving subproblems.
• Exact algorithm for achieving one objective improvement.
• Exact algorithm for exploring the infinite region ( ∞ Region).

4.3. Combining MCBB with Heuristics

4.4. discussion and conclusions, 5. conclusions, author contributions, conflicts of interest.

• Talbi, E.G. Metaheuristics: From Design to Implementation ; Wiley: Hoboken, NJ, USA, 2009. [ Google Scholar ]
• Breitkopf, P.; Coelho, R.F. Multidisciplinary Design Optimization in Computational Mechanics ; Wiley-ISTE: Hoboken, NJ, USA, 2013. [ Google Scholar ]
• Bensemlali, M.; Hatimi, B.; Sanad, A.; El Gaini, L.; Joudi, M.; Labjar, N.; Nasrellah, H.; Aarfane, A.; Bakasse, M. Novel Synthesis of Nanocalcite from Phosphogypsum and Cesium Carbonate: Control and Optimization of Particle Size. Eng 2024 , 5 , 932–943. [ Google Scholar ] [ CrossRef ]
• Bikas, H.; Manitaras, D.; Souflas, T.; Stavropoulos, P. Process-Driven Layout Optimization of a Portable Hybrid Manufacturing Robotic Cell Structure. Eng 2024 , 5 , 918–931. [ Google Scholar ] [ CrossRef ]
• Socha, K. Ant Colony Optimisation for Continuous and Mixed-Variable Domains ; VDM Verlag: Saarbrücken, Germany, 2009. [ Google Scholar ]
• Socha, K.; Dorigo, M. Ant colony optimization for continuous domains. Eur. J. Oper. Res. 2008 , 185 , 1155–1173. [ Google Scholar ] [ CrossRef ]
• Srinivas, N.; Deb, K. Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evol. Comput. 1994 , 2 , 221–248. [ Google Scholar ] [ CrossRef ]
• Jaber, A.; Lafon, P.; Younes, R. An Application of BnB-NSGAII: Initializing NSGAII to Solve 3 Stage Reducer Problem. In Proceedings of the Optimization and Learning, Sicilia, Italy, 21–23 June 2021; Communications in Computer and Information Science. pp. 337–349. [ Google Scholar ] [ CrossRef ]
• Han, L.; Liu, G.; Yang, X.; Han, B. Dimensional and Layout Optimization Design of Multistage Gear Drives Using Genetic Algorithms. Math. Probl. Eng. 2020 , 2020 , 3197395. [ Google Scholar ] [ CrossRef ]
• Jaber, A.; Lafon, P.; Younes, R. A branch-and-bound algorithm based on NSGAII for multi-objective mixed integer nonlinear optimization problems. Eng. Optim. 2021 , 54 , 1004–1022. [ Google Scholar ] [ CrossRef ]
• El Samrout, A. Hybridization of Multicriteria Metaheuristic Optimization Methods for Mechanical Problems. Ph.D. Thesis, Université de technologie de Troyes, Troyes, France, 2019. [ Google Scholar ]
• Canelas, E.; Pinto-Varela, T.; Sawik, B. Electricity portfolio optimization for large consumers: Iberian electricity market case study. Energies 2020 , 13 , 2249. [ Google Scholar ] [ CrossRef ]
• Syauqi, A.; Purwanto, W.W. Mixed-integer non-linear programming (MINLP) multi-period multi-objective optimization of advanced power plant through gasification of municipal solid waste (MSW). Chem. Prod. Process Model. 2020 , 15 , 20190126. [ Google Scholar ] [ CrossRef ]
• Ghaseminejad, A.; Kazemipoor, H.; Fallah, M. Modeling the robust facility layout problem for unequal space considering health and environmental safety criteria under uncertain parameters. Decis. Mak. Appl. Manag. Eng. 2023 , 6 , 426–460. [ Google Scholar ] [ CrossRef ]
• Monsiváis-Alonso, R.; Mansouri, S.; Román-Martínez, A. Life cycle assessment of intensified processes towards circular economy: Omega-3 production from waste fish oil. Chem. Eng. Process. Process Intensif. 2020 , 158 . [ Google Scholar ] [ CrossRef ]
• Ernst, P.; Zimmermann, K.; Fieg, G. Multi-objective Optimization-Tool for the Universal Application in Chemical Process Design. Chem. Eng. Technol. 2017 , 40 , 1867–1875. [ Google Scholar ] [ CrossRef ]
• Zimmermann, K.; Fieg, G. Development of a Diversity-Preserving Strategy for the Pareto Optimization in Chemical Process Design. Chemie-Ingenieur-Technik 2017 , 89 , 1297–1305. [ Google Scholar ] [ CrossRef ]
• Gargalo, C.; Carvalho, A.; Gernaey, K.; Sin, G. Optimal Design and Planning of Glycerol-Based Biorefinery Supply Chains under Uncertainty. Ind. Eng. Chem. Res. 2017 , 56 , 11870–11893. [ Google Scholar ] [ CrossRef ]
• Brunet, R.; Guillén-Gosálbez, G.; Jiménez, L. Combined simulation–optimization methodology to reduce the environmental impact of pharmaceutical processes: Application to the production of Penicillin V. J. Clean. Prod. 2014 , 76 , 55–63. [ Google Scholar ] [ CrossRef ]
• Zhang, S.; Zhuang, Y.; Tao, R.; Liu, L.; Zhang, L.; Du, J. Multi-objective optimization for the deployment of carbon capture utilization and storage supply chain considering economic and environmental performance. J. Clean. Prod. 2020 , 270 , 122481. [ Google Scholar ] [ CrossRef ]
• Bonnin, M.; Azzaro-Pantel, C.; Domenech, S. Optimization of natural resource management: Application to French copper cycle. J. Clean. Prod. 2019 , 223 , 252–269. [ Google Scholar ] [ CrossRef ]
• Rabbani, M.; Akbarpour, M.; Hosseini, M.; Farrokhi-Asl, H. A multi-depot vehicle routing problem with time windows and load balancing: A real world application. Int. J. Supply Oper. Manag. 2021 , 8 , 347–369. [ Google Scholar ] [ CrossRef ]
• Khodashenas, M.; Najafi, S.E.; Kazemipoor, H.; Sobhani, M. Providing an integrated multi-depot vehicle routing problem model with simultaneous pickup and delivery and package layout under uncertainty with fuzzy-robust box optimization method. Decis. Mak. Appl. Manag. Eng. 2023 , 6 , 372–403. [ Google Scholar ] [ CrossRef ]
• Moheb-Alizadeh, H.; Handfield, R.; Warsing, D. Efficient and sustainable closed-loop supply chain network design: A two-stage stochastic formulation with a hybrid solution methodology. J. Clean. Prod. 2021 , 308 . [ Google Scholar ] [ CrossRef ]
• Mohammadi, S.; Al-e Hashem, S.; Rekik, Y. An integrated production scheduling and delivery route planning with multi-purpose machines: A case study from a furniture manufacturing company. Int. J. Prod. Econ. 2020 , 219 , 347–359. [ Google Scholar ] [ CrossRef ]
• Mzili, T.; Mzili, I.; Riffi, M.; Pamucar, D.; Simic, V.; Abualigah, L. Hybrid Genetic and Penguin Search Optimization Algorithm (GA-PSEOA) for Efficient Flow Shop Scheduling Solutions. Facta Univ. Ser. Mech. Eng. 2023 , 22 . [ Google Scholar ] [ CrossRef ]
• Dhiman, G.; Singh, K.K.; Soni, M.; Nagar, A.; Dehghani, M.; Slowik, A.; Kaur, A.; Sharma, A.; Houssein, E.H.; Cengiz, K. MOSOA: A new multi-objective seagull optimization algorithm. Expert Syst. Appl. 2021 , 167 , 114150. [ Google Scholar ] [ CrossRef ]
• Benli, A.; Akgün, b. A Multi-Objective Mathematical Programming Model for Transit Network Design and Frequency Setting Problem. Mathematics 2023 , 11 , 4488. [ Google Scholar ] [ CrossRef ]
• Montoya, O.D.; Gil-González, W.; Grisales-Noreña, L.F. On the mathematical modeling for optimal selecting of calibers of conductors in DC radial distribution networks: An MINLP approach. Electr. Power Syst. Res. 2021 , 194 , 107072. [ Google Scholar ] [ CrossRef ]
• Juwari; Renanto; Arifin, R.; Anugraha, R.P.; Tamimi, F.Q.; Roostewen, K. Multi-objective optimization hydrogen network in refinery expansion with improved transport constraint. Int. J. Hydrogen Energy 2024 , 64 , 368–380. [ Google Scholar ] [ CrossRef ]
• Li, J.; Zhao, H. Multi-Objective Optimization and Performance Assessments of an Integrated Energy System Based on Fuel, Wind and Solar Energies. Entropy 2021 , 23 , 431. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Puchinger, J.; Raidl, G.R. Combining Metaheuristics and Exact Algorithms in Combinatorial Optimization: A Survey and Classification. In Proceedings of the Artificial Intelligence and Knowledge Engineering Applications: A Bioinspired Approach ; Mira, J., Álvarez, J.R., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2005; pp. 41–53. [ Google Scholar ]
• Rao, R.V.; Savsani, V.J. Mechanical Design Optimization Using Advanced Optimization Techniques ; Springer Series in Advanced Manufacturing; Springer: London, UK, 2012. [ Google Scholar ] [ CrossRef ]
• Chinchuluun, A.; Pardalos, P.M. A survey of recent developments in multiobjective optimization. Ann. Oper. Res. 2007 , 154 , 29–50. [ Google Scholar ] [ CrossRef ]
• Gunantara, N. A review of multi-objective optimization: Methods and its applications. Cogent Eng. 2018 , 5 , 1502242. [ Google Scholar ] [ CrossRef ]
• Belotti, P.; Kirches, C.; Leyffer, S.; Linderoth, J.; Luedtke, J.; Mahajan, A. Mixed-integer nonlinear optimization. Acta Numer. 2013 , 22 , 1–131. [ Google Scholar ] [ CrossRef ]
• Boukouvala, F.; Misener, R.; Floudas, C.A. Global optimization advances in Mixed-Integer Nonlinear Programming, MINLP, and Constrained Derivative-Free Optimization, CDFO. Eur. J. Oper. Res. 2016 , 252 , 701–727. [ Google Scholar ] [ CrossRef ]
• Harish Kumar, P.; Mageshvaran, R. Methods and solvers used for solving mixed integer linear programming and mixed nonlinear programming problems: A review. Int. J. Sci. Technol. Res. 2020 , 9 , 1872–1882. [ Google Scholar ]
• Blum, C.; Puchinger, J.; Raidl, G.R.; Roli, A. Hybrid metaheuristics in combinatorial optimization: A survey. Appl. Soft Comput. 2011 , 11 , 4135–4151. [ Google Scholar ] [ CrossRef ]
• Peres, F.; Castelli, M. Combinatorial Optimization Problems and Metaheuristics: Review, Challenges, Design, and Development. Appl. Sci. 2021 , 11 , 6449. [ Google Scholar ] [ CrossRef ]
• Liu, Q.; Li, X.; Liu, H.; Guo, Z. Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art. Appl. Soft Comput. 2020 , 93 , 106382. [ Google Scholar ] [ CrossRef ]
• Talbi, E.G. Hybrid Metaheuristics for Multi-Objective Optimization. J. Algorithms Comput. Technol. 2015 , 9 , 41–63. [ Google Scholar ] [ CrossRef ]
• Ehrgott, M.; Gandibleux, X. Hybrid Metaheuristics for Multi-objective Combinatorial Optimization. In Hybrid Metaheuristics: An Emerging Approach to Optimization ; Blum, C., Aguilera, M.J.B., Roli, A., Sampels, M., Eds.; Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2008; pp. 221–259. [ Google Scholar ] [ CrossRef ]
• Ehrgott, M.; Gandibleux, X. A survey and annotated bibliography of multiobjective combinatorial optimization. Spektrum 2000 , 22 , 425–460. [ Google Scholar ] [ CrossRef ]
• Trespalacios, F.; Grossmann, I.E. Review of Mixed-Integer Nonlinear and Generalized Disjunctive Programming Methods. Chem. Ing. Tech. 2014 , 86 , 991–1012. [ Google Scholar ] [ CrossRef ]
• Przybylski, A.; Gandibleux, X. Multi-objective branch and bound. Eur. J. Oper. Res. 2017 , 260 , 856–872. [ Google Scholar ] [ CrossRef ]
• Ehrgott, M.; Gandibleux, X.; Hillier, F.S. (Eds.) Multiple Criteria Optimization: State of the Art Annotated Bibliographic Surveys. In International Series in Operations Research & Management Science ; Springer: Boston, MA, USA, 2002; Volume 52. [ Google Scholar ] [ CrossRef ]
• Ehrgott, M. A discussion of scalarization techniques for multiple objective integer programming. Ann. Oper. Res. 2006 , 147 , 343–360. [ Google Scholar ] [ CrossRef ]
• Burachik, R.S.; Kaya, C.Y.; Rizvi, M.M. Algorithms for generating Pareto fronts of multi-objective integer and mixed-integer programming problems. Eng. Optim. 2022 , 54 , 1413–1425. [ Google Scholar ] [ CrossRef ]
• Pascoletti, A.; Serafini, P. Scalarizing vector optimization problems. J. Optim. Theory Appl. 1984 , 42 , 499–524. [ Google Scholar ] [ CrossRef ]
• Das, I.; Dennis, J.E. Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems. SIAM J. Optim. 1998 , 8 , 631–657. [ Google Scholar ] [ CrossRef ]
• Ahmadi, A.; Kaymanesh, A.; Siano, P.; Janghorbani, M.; Nezhad, A.E.; Sarno, D. Evaluating the effectiveness of normal boundary intersection method for short-term environmental/economic hydrothermal self-scheduling. Electr. Power Syst. Res. 2015 , 123 , 192–204. [ Google Scholar ] [ CrossRef ]
• Simab, M.; Javadi, M.S.; Nezhad, A.E. Multi-objective programming of pumped-hydro-thermal scheduling problem using normal boundary intersection and VIKOR. Energy 2018 , 143 , 854–866. [ Google Scholar ] [ CrossRef ]
• Zhu, Z.; Wang, X.; Jiang, C.; Wang, L.; Gong, K. Multi-objective optimal operation of pumped-hydro-solar hybrid system considering effective load carrying capability using improved NBI method. Int. J. Electr. Power Energy Syst. 2021 , 129 , 106802. [ Google Scholar ] [ CrossRef ]
• Land, A.H.; Doig, A.G. An Automatic Method of Solving Discrete Programming Problems. Econometrica 1960 , 28 , 497–520. [ Google Scholar ] [ CrossRef ]
• Kiziltan, G.; Yucaoğlu, E. An Algorithm for Multiobjective Zero-One Linear Programming. Manag. Sci. 1983 , 29 , 1444–1453. [ Google Scholar ] [ CrossRef ]
• Mavrotas, G.; Diakoulaki, D. A branch and bound algorithm for mixed zero-one multiple objective linear programming. Eur. J. Oper. Res. 1998 , 107 , 530–541. [ Google Scholar ] [ CrossRef ]
• Cacchiani, V.; D’Ambrosio, C. A branch-and-bound based heuristic algorithm for convex multi-objective MINLPs. Eur. J. Oper. Res. 2017 , 260 , 920–933. [ Google Scholar ] [ CrossRef ]
• Boix, M.; Montastruc, L.; Pibouleau, L.; Azzaro-Pantel, C.; Domenech, S. Multiobjective optimization of industrial water networks with contaminants. Comput. Aided Chem. Eng. 2010 , 28 , 859–864. [ Google Scholar ] [ CrossRef ]
• De Santis, M.; Eichfelder, G.; Niebling, J.; Rocktäschel, S. Solving Multiobjective Mixed Integer Convex Optimization Problems. SIAM J. Optim. 2020 , 30 , 3122–3145. [ Google Scholar ] [ CrossRef ]
• Morrison, D.R.; Jacobson, S.H.; Sauppe, J.J.; Sewell, E.C. Branch-and-bound algorithms: A survey of recent advances in searching, branching, and pruning. Discret. Optim. 2016 , 19 , 79–102. [ Google Scholar ] [ CrossRef ]
• Belotti, P.; Lee, J.; Liberti, L.; Margot, F.; Wächter, A. Branching and bounds tighteningtechniques for non-convex MINLP. Optim. Methods Softw. 2009 , 24 , 597–634. [ Google Scholar ] [ CrossRef ]
• Kronqvist, J.; Bernal, D.E.; Lundell, A.; Grossmann, I.E. A review and comparison of solvers for convex MINLP. Optim Eng 2019 , 20 , 397–455. [ Google Scholar ] [ CrossRef ]
• Burer, S.; Letchford, A.N. Non-convex mixed-integer nonlinear programming: A survey. Surv. Oper. Res. Manag. Sci. 2012 , 17 , 97–106. [ Google Scholar ] [ CrossRef ]
• el Sheshtawy, H.; el Moctar, O.; Natarajan, S. Multi-Point Shape Optimization of a Horizontal Axis Tidal Stream Turbine. Eng 2021 , 2 , 340–355. [ Google Scholar ] [ CrossRef ]
• Bade, S.O.; Meenakshisundaram, A.; Tomomewo, O.S. Current Status, Sizing Methodologies, Optimization Techniques, and Energy Management and Control Strategies for Co-Located Utility-Scale Wind–Solar-Based Hybrid Power Plants: A Review. Eng 2024 , 5 , 677–719. [ Google Scholar ] [ CrossRef ]
• Deb, K.; Pratap, A.; Agarwal, S.; Meyarivan, T. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 2002 , 6 , 182–197. [ Google Scholar ] [ CrossRef ]
• Verma, S.; Pant, M.; Snasel, V. A Comprehensive Review on NSGA-II for Multi-Objective Combinatorial Optimization Problems. IEEE Access 2021 , 9 , 57757–57791. [ Google Scholar ] [ CrossRef ]
• Glover, F. Tabu Search—Part II. ORSA J. Comput. 1990 , 2 , 4–32. [ Google Scholar ] [ CrossRef ]
• Dréo, J.; Candan, C. Different Classifications of Metaheuristics. 2007. Available online: https://commons.wikimedia.org/w/index.php?curid=16252087 (accessed on 20 July 2021).
• Yildiz, A.R.; Abderazek, H.; Mirjalili, S. A Comparative Study of Recent Non-traditional Methods for Mechanical Design Optimization. Arch. Comput. Methods Eng. 2020 , 27 , 1031–1048. [ Google Scholar ] [ CrossRef ]
• Abderazek, H.; Yildiz, A.R.; Sait, S.M. Mechanical engineering design optimisation using novel adaptive differential evolution algorithm. Int. J. Veh. Des. 2019 , 80 , 285. [ Google Scholar ] [ CrossRef ]
• Dhiman, G.; Singh, K.K.; Slowik, A.; Chang, V.; Yildiz, A.R.; Kaur, A.; Garg, M. EMoSOA: A new evolutionary multi-objective seagull optimization algorithm for global optimization. Int. J. Mach. Learn. Cybern. 2021 , 12 , 571–596. [ Google Scholar ] [ CrossRef ]
• Assiri, A.S. On the performance improvement of Butterfly Optimization approaches for global optimization and Feature Selection. PLoS ONE 2021 , 16 , e0242612. [ Google Scholar ] [ CrossRef ]
• Mzili, T.; Mzili, I.; Riffi, M.; Pamucar, D.; Simic, V.; Kurdi, M. A Novel Discrete Rat Swarm Optimization Algorithm for the Quadratic Assignment Problem. Facta Univ. Ser. Mech. Eng. 2023 , 21 , 529–552. [ Google Scholar ] [ CrossRef ]
• Kaur, H.; Rai, A.; Bhatia, S.S.; Dhiman, G. MOEPO: A novel Multi-objective Emperor Penguin Optimizer for global optimization: Special application in ranking of cloud service providers. Eng. Appl. Artif. Intell. 2020 , 96 , 104008. [ Google Scholar ] [ CrossRef ]
• Chakraborty, S.; Kumar Saha, A.; Sharma, S.; Mirjalili, S.; Chakraborty, R. A novel enhanced whale optimization algorithm for global optimization. Comput. Ind. Eng. 2021 , 153 , 107086. [ Google Scholar ] [ CrossRef ]
• Sattar, D.; Salim, R. A smart metaheuristic algorithm for solving engineering problems. Eng. Comput. 2021 , 37 , 2389–2417. [ Google Scholar ] [ CrossRef ]
• Giagkiozis, I.; Purshouse, R.C.; Fleming, P.J. An overview of population-based algorithms for multi-objective optimisation. Int. J. Syst. Sci. 2015 , 46 , 1572–1599. [ Google Scholar ] [ CrossRef ]
• Jourdan, L.; Basseur, M.; Talbi, E.G. Hybridizing exact methods and metaheuristics: A taxonomy. Eur. J. Oper. Res. 2009 , 199 , 620–629. [ Google Scholar ] [ CrossRef ]
• Blum, C.; Cotta, C.; Fernández, A.J.; Gallardo, J.E.; Mastrolilli, M. Hybridizations of Metaheuristics with Branch & Bound Derivates. In Hybrid Metaheuristics: An Emerging Approach to Optimization ; Blum, C., Aguilera, M.J.B., Roli, A., Sampels, M., Eds.; Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2008; pp. 85–116. [ Google Scholar ] [ CrossRef ]
• Nagar, A.; Heragu, S.S.; Haddock, J. A combined branch-and-bound and genetic algorithm based approach for a flowshop scheduling problem. Ann. Oper. Res. 1996 , 63 , 397–414. [ Google Scholar ] [ CrossRef ]
• Cotta, C.; Troya, J.M. Embedding Branch and Bound within Evolutionary Algorithms. Appl. Intell. 2003 , 18 , 137–153. [ Google Scholar ] [ CrossRef ]
• Woodruff, D.L. A Chunking Based Selection Strategy for Integrating Meta-Heuristics with Branch and Bound. In Meta-Heuristics: Advances and Trends in Local Search Paradigms for Optimization ; Voß, S., Martello, S., Osman, I.H., Roucairol, C., Eds.; Springer: Boston, MA, USA, 1999; pp. 499–511. [ Google Scholar ] [ CrossRef ]
• Jozefowiez, N.; Semet, F.; Talbi, E.G. The bi-objective covering tour problem. Comput. Oper. Res. 2007 , 34 , 1929–1942. [ Google Scholar ] [ CrossRef ]
• Puchinger, J.; Raidl, G.R.; Koller, G. Solving a Real-World Glass Cutting Problem. In Proceedings of the Evolutionary Computation in Combinatorial Optimization ; Gottlieb, J., Raidl, G.R., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2004; pp. 165–176. [ Google Scholar ] [ CrossRef ]
• Backer, B.D.; Furnon, V.; Shaw, P.; Kilby, P.; Prosser, P. Solving Vehicle Routing Problems Using Constraint Programming and Metaheuristics. J. Heuristics 2000 , 6 , 501–523. [ Google Scholar ] [ CrossRef ]
• Florios, K.; Mavrotas, G.; Diakoulaki, D. Solving multiobjective, multiconstraint knapsack problems using mathematical programming and evolutionary algorithms. Eur. J. Oper. Res. 2010 , 203 , 14–21. [ Google Scholar ] [ CrossRef ]
• Wang, S.; Liu, M. A heuristic method for two-stage hybrid flow shop with dedicated machines. Comput. Oper. Res. 2013 , 40 , 438–450. [ Google Scholar ] [ CrossRef ]

Share and Cite

Jaber, A.; Younes, R.; Lafon, P.; Khoder, J. A Review on Multi-Objective Mixed-Integer Non-Linear Optimization Programming Methods. Eng 2024 , 5 , 1961-1979. https://doi.org/10.3390/eng5030104

Jaber A, Younes R, Lafon P, Khoder J. A Review on Multi-Objective Mixed-Integer Non-Linear Optimization Programming Methods. Eng . 2024; 5(3):1961-1979. https://doi.org/10.3390/eng5030104

Jaber, Ahmed, Rafic Younes, Pascal Lafon, and Jihan Khoder. 2024. "A Review on Multi-Objective Mixed-Integer Non-Linear Optimization Programming Methods" Eng 5, no. 3: 1961-1979. https://doi.org/10.3390/eng5030104

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals