RRP
making science self-correcting through...
transparency, verification and collaboration
choosing what to study in the research code of conduct falls under...
research integrity: dont waste rescources
why have a course on research ethics? (4)
• standards change over time and people will have to relearn/ keep up-to-date
• mentor/leadership to transfer the rules
• reflection now to deal with dilemmas later
• make values explicit
what is the goal of having codes, courses on ethics, mentorship etc?
responsible scholarship
what is responsible scholarship?
conducting work with integrity and meeting the needs for better quality and efficiency in science.
why is responsible scholarship important? (6)
• safeguard quality
• enable trust within scientific community
• safeguard reputation of science
• equality and equity in opportunities
• prevent waste
• robust cumulative science
5 steps of rogers diffusion model
1. infrastructure
2. user experience
3. communities
4. incentives
5. policy
4 levels of distorted scientific integrity
• scientific misconduct
• questionable research practice (QRP)
• poor research practice (competence)
• honest errors (fallibility)
replication
same method, different data
same method, different data
replication
robustness
same data, different analysis
same data, different analysis
robustness
reproducibility
same data, same analysis
same data, same analysis
reproducibility
what is the opposite of robustness
fragility
3 correlates of effective replicability
• theory maturity
o old/ well-established theories replicate better than new ones with (still) abstract variables
• features of the original study
• features of the replication
what features of the original study might make replication less likely
small sample, poorly controlled design, false positive, low statistical power, low transparency
what features of the replication study might make replication less likely
small sample, poorly controlled design, false negative, not adhering to the original design
a good hypothesis:
- specifeis the population
- is quantifiable
- is testable
(- mentions the direction)
why does HARKing mean using the data twice
1. to generate a hypothesis
2. to test the hypothesis
why is HARKing a big problem
uses the data twice:
- This inflates type I error (false positive), as you would likely find the hypothesis you came up with to be true.
- also makes it less likely the findings will replicate
7 deadly sins
- sin of bias
- sin of hidden flexibility
- sin of unreliability
- sin of data hoarding
- sin of corruptibility
- sin of internment
- sin of bean counting
sin of bias
favouring studies that confirm your hypothesis
favouring studies that confirm your hypothesis
sin of bias
sin of hidden flexibility
torturing data/ QRP to achieve sign. results
torturing data/ QRP to achieve sign. results
sin of hidden flexibility
sin of unreliability
low statistical power which leads to false positives
low statistical power which leads to false positives
sin of unreliability
sin of data hoarding
not sharing raw data
not sharing raw data
sin of data hoarding
sin of corruptability
allowing professional incentives to encourage fraud or ethical breaches
allowing professional incentives to encourage fraud or ethical breaches
sin of corruptability
sin of internment
publishing behind a paywal
publishing behind a paywal
sin of internment
sin of bean counting
obsessive reliance on specific metrics over research quality
obsessive reliance on specific metrics over research quality
sin of bean counting
10 top guidelines
1. data citation
2. reporting guidelines
3. data transparency
4. replication
5. analysis transparency
6. badges
7. publication bias
8. material transparency
9. preregistration of analysis
10. preregistration of study
what are TOP guidelines
a framework for journals and institutions to improve the transparency and replicability of research
COG statement
constraints on generalization - who the study can or cannot extend to.
β (beta)
the strength and direction of the relationship between a predictor and the outcome
the strength and direction of the relationship between a predictor and the outcome
β (beta)
two types of variation that most statistical methods assume
1. random sampling cases from a population
2. random allocation of treatment
2 assumptions (most) statistical models rely on
- observations are independent of one another
- observations are identically distributed
WEIRD
• Western
• Educated
• Industrialized
• Rich
• Democratic
4 variables of a power analysis
- sample size
- effect size
- significance level
- power
type I error
false positive
Type II error
false negative
what does a significance level of 0.05 mean?
You accept a 5% chance of finding a false positive
what is power?
The probability of finding an effect when it is really there
what does it mean if power is 0.8?
meaning you're willing to miss a real effect 20% of the time. (have a false negative)
what is the effect size?
The expected magnitude of the phenomenon being studied
what is the problem with small sample size?
small sample sizes might give a distorted view of the population
- sample might not be random
- leads to smaller power (might not detect it)
- which leads to more Type II errors
- so, low replicability
what does sample size depend on? (7)
- smallest effect of interest
- desired power and significance level
- distribution of observables
- statistical tests
- one or two-tailed
- anticipated drop-out
- precision of measurements
jsutifications for not doing a power analysis (6)
1. have an (almost) entire population
2. resource constraints
3. power analysis
4. planning for desired accuracy
5. using heuristics
6. acknowledging there is no justification
what is something you can do if you data has an atypical distribution
bootstrapping
bootstrapping
a resampling technique. instead of assuming a normal distribution this method (by resampling many different times) lets you estimate power and significance level from the data itself.
what can be a problem if you guess the effect size wrong
your study is either wastefully large or underpowered.
what can you do if the effect size is unkown
Sequential analysis (no set sample size, no waste of rescources based on a wrong power analysis)
Sequential analysis
• Collect data in stages
• Analyse results at intermediate points
• Stop when you have enough evidence
what is meant with the results paradox?
results are beyond the control of the scientist, but are what is most important (for publication).
two types of preregistration
1. unreviewed
2. reviewed
IPA
in priciple acceptance
What can be preregistered?
- hypotheses
- methods
o design
o planned sample
o exclusion criteria
o procedure
- analysis plan
o confirmatory analyses
o contingencies and assumptions
benefits of reviewed preregistration (10)
• prioritizing theory and method, rather than just results
• distinguishes confirmatory and exploratory research
• transparency
• reduces positive results bias/ publication bias
• reduces reporting bias
• more thoroughness, review, and input
• less dependence on chance
• more opportunity to show skill
• faster dissemination
• help with research
o more collaboration (adversarial collaboration)
potential drawbacks of pre-registration (5)
• more work?
• too restrictive? loose flexibility?
• null literature?
• idea theft?
• it does not stop fraud
common issues with preregistration (3)
• difficult to prespecify full analysis plan
• difficult to avoid all unambiguity
• difficult to know what effect size to set the power for
preregistration might not be suitable for:
- fully exploratory
- studies seeking to capture the effects of unpredictable events
- students working with deadlines for their thesis
for confirmatory analysis is Type I error or Type II error a bigger problem?
Type I, as you want to confirm your hypothesis and make changes based on it and a false positive might lead to wrong changes being made
for exploratory analysis is Type I error or Type II error a bigger problem?
Type II as you are trying to generate new hypotheses and you will not find anything if you have a false negative
preventing type II error
higher power
preventing type I error
stricter significance level (0.01)
how to move forward with RRs?
- transparency (publish both stages)
- standardization
- efficiency
reliability
Consistent and repeatable over time
Consistent and repeatable over time
reliability
precision is an aspect of
reliability
validity
Measures what you want it to
Measures what you want it to
validity
accuracy is an aspect of...
validity
... is a necessary but not sufficient condition for ...
reliability and validity
test-retest reliability
within the same researcher
reliability within the same researcher
test-retest reliability
internal consistency
consistency across items
reliability, consistency across items
internal consistency
interrater reliability
between researchers
reliability between researchers
interrater reliability
Internal validity
results are genuinely caused by the independent variable rather than confounding factors.
external validity
determines if results can be generalized.
Questionable measurement practices (4)
- lack of transparency
- ignorance
- negligence
- misrepresentation of evidence
give an example of a lack of transparency regarding your measurement
not mentioning why you chose that measure
give an example of a ignorance regarding your measurement
not researching a validated measure, just making one up
give an example of negligance regarding your measurement
using the measure for an unintended population
give an example of misrepresentation of evidence regarding your measurement
claiming a measure is valid when its not
what should you ask yourself to have good measurement practice (5)
- what is my construct?
- how do I operationalise my construct?
- why did I select my measure?
- did I modify my measure?
- did I create my measure (on a whim)?
what is an outcome neutral tests
When you test a theory, you're not just testing the theory. You're also relying on assumptions:
• That your manipulation actually manipulates what you think it does
• That your measure actually measures what you think it does
Outcome-neutral tests check those assumptions independently.
what assumptions do you assume when you use a manipulation
• That your manipulation actually manipulates what you think it does
• That your measure actually measures what you think it does
what does a positve control do?
Something that should produce an effect if your procedure is working correctly.
It verifies that your measurement or manipulation is sensitive enough to detect a real effect.
If your positive control doesn't work, you know your procedure is broken — not that your hypothesis is wrong.
what is a negative control
Something that should not produce an effect if your procedure is working correctly.
It verifies that your measurement isn't picking up noise, confounds, or non-specific responses.
If your negative control shows an effect, something other than your intended manipulation is driving your results.
what is experimenter bias
a scientist's expectations, beliefs, or preferences unintentionally influence the results of a study
three ways to reduce experimenter bias
o blinding
o maintaining a lab notebook with all choices (and publishing this aswell)
o reflexivity – embracing and acknowledging the researchers influence
reflexivity
embracing and acknowledging the researchers influence