<!DOCTYPE html>

<html data-programme="Research Integrity" data-course="Data Selection, Analysis and Presentation">

	<head>
		<title>Application</title>
		<meta charset="UTF-8">
		<meta http-equiv="X-UA-Compatible" content="IE=edge">
		<meta name="viewport" content="width=device-width, initial-scale=1">

		<link rel="stylesheet" type="text/css" href="../css/project.min.css">
		<link id="basic-print" rel="stylesheet" type="text/css" href="../css/print.min.css" disabled>
		<link id="summary-print" rel="stylesheet" type="text/css" href="../css/print_interactive.min.css">
	<script type="text/javascript" src="../scripts_scorm_12/SCORM_12_APIWrapper.js"></script>
<script type="text/javascript" src="../scripts_scorm_12/courseFunctions.js"></script>
</head>

	<body onLoad="javascript:Initialize();" onUnload="javascript:Terminate();">

		<div id="app" v-cloak>
			<base-container
				screen-id="dap_1_80"
				current-module="Data Selection, Analysis and Presentation"
				page-title="Application"
				screen-time="5"
				screen-time-period="mins">

				<page-container>

					<page-row>
						<page-content-sidebar>
							
							<!-- OPTIONAL COURSE EVALUATION START -->
							<pod variant="d" has-modal="true">
								<pod-heading>Module evaluation</pod-heading> <!-- pod title -->
								<pod-content></pod-content>
								<!-- pop-up pod content -->
								<pod-modal-heading>Module evaluation (optional)</pod-modal-heading>
								<pod-modal-content>
									<p><a href="https://www.epigeum.com/downloads/hosted_content/RI_INT/survey/ri_uk_survey.html" target="_blank">Click here</a> to help develop and improve this course by providing some brief feedback (anonymously, of course). Please note that completing this survey is entirely optional. However, your input would be very welcome.</p>
								</pod-modal-content>
							</pod>
							<!-- OPTIONAL COURSE EVALUATION END -->
							
						</page-content-sidebar>
						
						<page-content-main>

							<p>This module is essentially about honest versus dishonest creativity in research. A
								responsible researcher is expected to carefully select, analyse and present
								data/information. This is the challenge of research, and finding answers to questions that
								others have not been able to answer is the reward.</p>
							<p>When researchers fail to meet their responsibilities in this area, it is often because they
								have not received good advice, training and/or supervision. To round out the module, the
								following activity illustrates how this can happen.</p>

							<instructional-text>
								<p>Work through the scenario below, which outlines a conversation between a supervisor,
									whose conduct is questionable, and their supervisee. After reading each exchange, click
									"Next" to access a question about it and type in your answer. Once you have answered the
									question, click "View feedback" to read our thoughts. You should then click "Next" to
									proceed to the following part of the conversation.</p>
							</instructional-text>
							<instructional-text-accessible><p>Work through the following scenario, which outlines a conversation between a supervisor, whose conduct is questionable, and their supervisee. After working through each exchange, continue on to access a question about it and take a moment to consider your answer. Once you have answered the question, continue on for our feedback. You should then proceed to the following part of the conversation.</p></instructional-text-accessible>

							<!-- ACTIVITY START -->
							<process show-in-accessible-version="false" summary="true" variant="mcq-notemaking" match-height="true">
								<process-page next-btn-always-active="true">
									<p><strong>Supervisor:</strong> "Thank you for adjusting your plans so we can meet. To
										start off, let me apologise for the delay; I've been busy with grant applications.
										Rather than waste time reviewing your results, can you give me a brief update on how
										your research is going? If I understand correctly, you are getting close to
										publication?"</p>
									<p><strong>Supervisee:</strong> "There have been one or two questions that have come up
										over the past couple of months that might have benefited from your advice, but I am
										confident we made the right decisions and you will be pleased with the progress we
										have made."</p>
								</process-page>
								<process-page>
									<note-making>
										<note-question>
											<p>What are the problems with this approach to supervision/advising in relation
												to data selection, analysis and reporting?</p>
										</note-question>
										<note-input></note-input>
										<note-feedback>
											<p>To judge whether the supervisee has made good decisions, the supervisor will
												at some stage have to review the evidence (data) that will be used in the
												publication. Relying on the account of the supervisee is not sufficient.
												This meeting may be the best way to start the conversation but it cannot be
												the only step taken before publication.</p>
										</note-feedback>
									</note-making>
								</process-page>
								<process-page next-btn-always-active="true">
									<p><strong>Supervisee:</strong> "To start with, the interviewing stage went very well.
										We were hoping to conduct 100 interviews. We had more volunteers than we expected
										and ultimately conducted 122 interviews. This was pretty lucky, since some of the
										responses showed that the participants didn't understand the questions. We dropped
										those interviews and still have the sample size we wanted. Our results are much more
										robust as a result."</p>
									<p><strong>Supervisor:</strong> "That's great. We usually struggle to get enough
										volunteers. This should improve your chances of getting this work published."</p>
								</process-page>
								<process-page>
									<note-making>
										<note-question>
											<p>What other questions should the supervisor have asked at this stage?</p>
										</note-question>
										<note-input></note-input>
										<note-feedback>
											<p>The supervisor should have asked for further information. How did the
												supervisee know that the participants did not understand the questions? Is
												it possible that the participants did understand, but this was not
												appreciated since the responses given weren't what the researcher
												expected/wanted to hear? At the very least, more information is needed on
												how the data was selected. Simply dropping some interviews because they did
												not provide the information you were looking for is not responsible.</p>
											<p>As an aside, the supervisee might not have had permission to interview more than 100 participants; if this were the case, they may have violated the
												conditions of the ethics committee approval.</p>
										</note-feedback>
									</note-making>
								</process-page>
								<process-page next-btn-always-active="true">
									<p><strong>Supervisee:</strong> "Once we coded the responses and entered our variables,
										we ran the planned statistical tests. For the most part, the predictions we set out
										in the grant application were confirmed and the results fit in nicely with prior
										work in this area. The only problem was that there was only marginal significance in
										a few cases. We were getting <em>p</em>-values above 0.05 for a couple of the
										crucial measures. However, it turned out we were not properly weighting our
										variables. As soon as we changed the way we were weighting one of the key variables,
										the <em>p</em>-values dropped well below .05. This is what we need to publish."</p>
									<p><strong>Supervisor:</strong> "That's good news, but you need to be careful here. You
										can't 'fish' for results &ndash; that is, run your tests in different ways until you
										get the results you want."</p>
									<p><strong>Supervisee:</strong> "That's a good point, but the initial results were very
										close to significance. I'm confident that the changes are making the results
										clearer, not misleading. We can test this with later work if we want to."</p>
									<p><strong>Supervisor:</strong> "Okay, I see your point. Go ahead, but keep the need to
										check this later in mind."</p>
								</process-page>
								<process-page>
									<mcq>
										<mcq-question><strong>Has the team come up with a responsible way to deal with a
												marginal <em>p</em>-value?</strong></mcq-question>
										<mcq-options>
											<mcq-option neutral>
												<mcq-answer>Yes</mcq-answer>
												<mcq-feedback>
													<p>The response to this question depends on what the supervisee
														ultimately publishes, so there is no definitively right or wrong
														answer. If the publication contains a full and honest description of
														how the analysis was undertaken, accompanied by access to the raw
														data, then other researchers can decide for themselves whether the
														analysis was responsible. However, if the details of the analysis
														are not explained, then the supervisee may be carrying out a form of
														<em>p</em>-hacking (i.e. analysing the data in different ways until
														a statistically significant result is achieved). If your aim in
														processing data is simply to optimise the statistical significance of your work, it is unlikely to be objective.</p>
												</mcq-feedback>
											</mcq-option>
											<mcq-option neutral>
												<mcq-answer>No</mcq-answer>
												<mcq-feedback>
													<p>The response to this question depends on what the supervisee
														ultimately publishes, so there is no definitively right or wrong
														answer. If the publication contains a full and honest description of
														how the analysis was undertaken, accompanied by access to the raw
														data, then other researchers can decide for themselves whether the
														analysis was responsible. However, if the details of the analysis
														are not explained, then the supervisee may be carrying out a form of
														<em>p</em>-hacking (i.e. analysing the data in different ways until
														a statistically significant result is achieved). If your aim in
														processing data is simply to optimise the statistical significance
														your work, it is unlikely to be objective.</p>
												</mcq-feedback>
											</mcq-option>
										</mcq-options>
									</mcq>
								</process-page>
								<process-page next-btn-always-active="true">
									<p><strong>Supervisee:</strong> "We are now grappling with the problem of how to present
										this information in a publication. When we plotted all the factors that influence
										social change on the graph, we found the overall picture was fairly confusing, since
										some of the factors were quite weak; some may even constitute a negative influence.
										We've decided to focus on two major factors, feature them in our graphs and tables,
										and emphasise them in the title of the paper. We feel we will have a better chance
										of getting this work published if we focus on what we can demonstrate."</p>
									<p><strong>Supervisor:</strong> "As before, you should be careful. You must mention that
										you looked at other factors and add a line or two about this under 'limitations'. You
										don't have to go into too much detail, and there is nothing wrong with emphasising
										the positive. After all, if you don't do a good job of selling your research, who
										will?"</p>
									<p><strong>Supervisee:</strong> "That's more or less what we thought. We will wrap this
										up in the next few days and send it off. Do you want to see the final version before
										we do?"</p>
									<p><strong>Supervisor:</strong> "This is your publication. If you are comfortable with
										the work, go ahead and send it."</p>
								</process-page>
								<process-page>
									<mcq>
										<mcq-question><strong>If you were in the supervisee's situation, would you ask your
												supervisor to review the paper again or go straight ahead and submit it
												based on the advice you have received?</strong></mcq-question>
										<mcq-options>
											<mcq-option neutral>
												<mcq-answer>Submit the publication</mcq-answer>
												<mcq-feedback>
													<p>The correct answer is to "ask the supervisor to give the paper a
														second look". The supervisor in this case is not taking a hands-on
														approach. While the supervisee may be well intentioned, some of the
														decisions that have been made could seriously undermine the
														reliability of the final publication. If you find yourself in this
														situation and feel you need more advice, do not hesitate to ask
														other experienced colleagues.</p>
													<p>In this case, the researcher could find their publication receiving
														the following review comments: "The data selection is flawed and
														might even violate the ethics approval received for the project;
														there is no doubt that they are engaged in <em>p</em>-hacking [i.e. selecting
														data and using different analytical approaches with the goal of
														increasing significance]; and the graphs and title hype what is otherwise a less&ndash;than&ndash;critical analysis of an important problem."</p>
												</mcq-feedback>
											</mcq-option>
											<mcq-option neutral>
												<mcq-answer>Ask the supervisor to give the paper a second look</mcq-answer>
												<mcq-feedback>
													<p>The correct answer is to "ask the supervisor to give the paper a
														second look". The supervisor in this case is not taking a hands-on
														approach. While the supervisee may be well intentioned, some of the
														decisions that have been made could seriously undermine the
														reliability of the final publication. If you find yourself in this
														situation and feel you need more advice, do not hesitate to ask
														other experienced colleagues.</p>
													<p>In this case, the researcher could find their publication receiving
														the following review comments: "The data selection is flawed and
														might even violate the ethics approval received for the project;
														there is no doubt that they are engaged in <em>p</em>-hacking [i.e. selecting
														data and using different analytical approaches with the goal of
														increasing significance]; and the graphs and title hype what is otherwise a less&ndash;than&ndash;critical analysis of an important problem."</p>
												</mcq-feedback>
											</mcq-option>
										</mcq-options>
									</mcq>
								</process-page>
							</process>
							<!-- ACCESSIBLE START -->
							<accessible-block>
								<h4>Part 1 of 4:</h4>
								<p><strong>Supervisor:</strong> "Thank you for adjusting your plans so we can meet. To start off, let me apologise for the delay; I've been busy with grant applications. Rather than waste time reviewing your results, can you give me a brief update on how your research is going? If I understand correctly, you are getting close to publication?"</p>
								<p><strong>Supervisee:</strong> "There have been one or two questions that have come up over the past couple of months that might have benefited from your advice, but I am confident we made the right decisions and you will be pleased with the progress we have made."</p>
								<hr>
								<p><strong>Question 1 of 4:</strong> What are the problems with this approach to supervision/advising in relation to data selection, analysis and reporting?</p>
								<p>Take a moment to consider your thoughts before moving on to our feedback.</p>
								<p><strong>Feedback:</strong> To judge whether the supervisee has made good decisions, the supervisor will, at some stage, have to review the evidence (data) that will be used in the publication. Relying on the account of the supervisee is not sufficient. This meeting may be the best way to start the conversation, but it cannot be the only step taken before publication.</p>
								<hr>
								<h4>Part 2 of 4:</h4>
								<p><strong>Supervisee:</strong> "To start with, the interviewing stage went very well. We were hoping to conduct 100 interviews. We had more volunteers than we expected and ultimately conducted 122 interviews. This was pretty lucky, since some of the responses showed that the participants didn't understand the questions. We dropped those interviews and still have the sample size we wanted. Our results are much more robust as a result."</p>
								<p><strong>Supervisor:</strong> "That's great. We usually struggle to get enough volunteers. This should improve your chances of getting this work published."</p>
								<hr>
								<p><strong>Question 2 of 4:</strong> What other questions should the supervisor have asked at this stage?</p>
								<p>Take a moment to consider your answer before moving on to our feedback.</p>
								<p><strong>Feedback:</strong> The supervisor should have asked for further information. How did the supervisee know that the participants did not understand the questions? Is it possible that the participants did understand, but this was not appreciated since the responses given weren't what the researcher expected/wanted to hear? At the very least, more information is needed on how the data was selected. Simply dropping some interviews because they did not provide the information you were looking for is not responsible.</p>
								<p>As an aside, the supervisee might not have had permission to interview more than 100 participants; if this were the case, they may have violated the conditions of the ethics committee approval.</p>
								<hr>
								<h4>Part 3 of 4:</h4>
								<p><strong>Supervisee:</strong> "Once we coded the responses and entered our variables, we ran the planned statistical tests. For the most part, the predictions we set out in the grant application were confirmed and the results fit in nicely with prior work in this area. The only problem was that there was only marginal significance in a few cases. We were getting p-values above 0.05 for a couple of the crucial measures. However, it turned out we were not properly weighting our variables. As soon as we changed the way we were weighting one of the key variables, the p-values dropped well below .05. This is what we need to publish."</p>
								<p><strong>Supervisor:</strong> "That's good news, but you need to be careful here. You can't 'fish' for results &ndash; that is, run your tests in different ways until you get the results you want."</p>
								<p><strong>Supervisee:</strong> "That's a good point, but the initial results were very close to significance. I'm confident that the changes are making the results clearer, not misleading. We can test this with later work if we want to."</p>
								<p><strong>Supervisor:</strong> "Okay, I see your point. Go ahead, but keep the need to check this later in mind."</p>
								<hr>
								<p><strong>Question 3 of 4:</strong> Has the team come up with a responsible way to deal with a marginal p-value?</p>
								<ul>
									<li>Yes</li>
									<li>No</li>
								</ul>
								<p><strong>Feedback:</strong> The response to this question depends on what the supervisee ultimately publishes, so there is no definitively right or wrong answer. If the publication contains a full and honest description of how the analysis was undertaken, accompanied by access to the raw data, then other researchers can decide for themselves whether the analysis was responsible. However, if the details of the analysis are not explained, then the supervisee may be carrying out a form of p-hacking (i.e. analysing the data in different ways until a statistically significant result is achieved). If your aim in processing data is simply to optimise the statistical significance of your work, it is unlikely to be objective.</p>
								<hr>
								<h4>Part 4 of 4:</h4>
								<p><strong>Supervisee:</strong> "We are now grappling with the problem of how to present this information in a publication. When we plotted all the factors that influence social change on the graph, we found the overall picture was fairly confusing, since some of the factors were fairly weak; some may even constitute a negative influence. We've decided to focus on two major factors, feature them in our graphs and tables, and emphasise them in the title of the paper. We feel we will have a better chance of getting this work published if we focus on what we can demonstrate."</p>
								<p><strong>Supervisor:</strong> "As before, you should be careful. You must mention that you looked at other factors and add a line or two about this under 'limitations'. You don't have to go into too much detail, and there is nothing wrong with emphasising the positive. After all, if you don't do a good job of selling your research, who will?"</p>
								<p><strong>Supervisee:</strong> "That's more or less what we thought. We will wrap this up in the next few days and send it off. Do you want to see the final version before we do?"</p>
								<p><strong>Supervisor:</strong> "This is your publication. If you are comfortable with the work, go ahead and send it."</p>
								<hr>
								<p><strong>Question 4 of 4:</strong> If you were in the supervisee's situation, would you ask your supervisor to review the paper again or go straight ahead and submit it based on the advice you have received?</p>
								<ul>
									<li>Submit the publication</li>
									<li>Ask the supervisor to give the paper a second look</li>
								</ul>
								<p><strong>Feedback:</strong> The correct answer is to "ask the supervisor to give the paper a second look". The supervisor in this case is not taking a hands-on approach. While the supervisee may be well intentioned, some of the decisions that have been made could seriously undermine the reliability of the final publication. If you find yourself in this situation and feel you need more advice, do not hesitate to ask other experienced colleagues.</p>
								<p>In this case, the researcher could find their publication receiving the following review comments: "The data selection is flawed and might even violate the ethics approval received for the project; there is no doubt that they are engaged in p-hacking [i.e. selecting data and using different analytical approaches with the goal of increasing significance]; and the graphs and title hype what is otherwise a less-than-critical analysis of an important problem."</p>
							</accessible-block>
							<!-- ACCESSIBLE END -->
							<!-- ACTIVITY END -->

						</page-content-main>
					</page-row>

				</page-container>
			</base-container>
		</div> <!-- END content wrapper -->

		<!-- scripts -->
		<script src="../js/vendor.min.js"></script>
		<script src="../js/vue-components.min.js"></script>
		<script>
			new Vue({
				el: "#app",
				data: {},
				mounted: function () {
					var core = document.createElement('script');
					core.src = "../js/core.min.js";
					document.getElementsByTagName('head')[0].appendChild(core);

					var project = document.createElement('script');
					project.src = "../js/project.min.js";
					document.getElementsByTagName('head')[0].appendChild(project);
				}
			});
		</script>

	</body>

</html>