PY1501.3 ~ Theory of Measurement & Experiment Design
what is psychometris
study of how to measure psychological attriobutes; [ai: measuring the mind + behaviour through standardized tools, ensuring they're reliable, valid + useful]
from the gree 'psyche [mind/soul] + metron [measurement]
aims of psychometry
to develop and refine a theoretical understanding of the measurement of psychological attributes
to construct instruments and procedures for measuring psychological attributes
William Thompson (Lord Kelvin) (1883)
"When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be”
measurement error
diff between the true value of a quantity and the observed or measured value [errors can be systematic: consistent + predictable or random: varying + unpredictable
classical test theory [ctt]
observed score = true score + error
grounded in the premise thag any observed test score consists of 2 components (true score + error)
true score
the actual score a person would obtain if there were no erros in measurement
error score
random error or variance in the observed score that arises due to various factors [eg test-taking conditions + individual diff etc]
mean measurement
if you measure a piece of string several times you get a range of diff measurements [some largerr some smaller than actual value] get an avg of these to cancel out errors
benefits of mean/repeated measurement
under ctt errors are normally distributed + random
uses items that measure the same latent variable
latent variable
variable that is not direvtly observed but is inferred from other observed [measured] variables, used to represent constructs that cannot be directly measured but are believed to influence observed data
how can the amount of measurement error be estimated
from standard deviation of measurements
measurement error contributes to the variability or spread of observed scores in a test or measurement process
errors follow normal distribution
how accurate is your mean estimate
the more times you measure the string, the more accurate your mean estimate will be , cause each time you take a new measurement, the mean changes slightly
what is the distribution of mean estimate
normally distributed
how sample means behave. when repeated sampling is done
standard deviation of mean estimate
standard error of measurement [S.E.]
provides estimate of how much an individual's observed score deviates from thei true score due to random measurement errors [quantifies precision of test score]
confidence intervals formula
95% CI = mean estimate +/- 2 x S.E.
theres a 95% chance than the 'true' measure lies within 2 standard errors of the mean estimate
the trade-off between accuracy and time
more times you measure, the more accurate your estimate will be but each additional measurement adds less and less accuracy (1/ √ N)
what are measurement scales
systems used to classify + quantify variables in research, defining how numbers or labels are assigned to observations
4 primary types are nominal + ordinal + interval + ratio
3 things about measurement scales
are arbitrary + need to be defined + do not affect the amount of error in your measurement
how many levels of measurement are there and what are they
SS Stevens (1946) postulated that there are 4 levels
nominal + ordinal + interval + ratio
nominal scale
a set of mutually exclusive categories, which have no particular ordering
only rule is, the same label must not be applies to diff categories [eg species + numbers of footbal shirts]
ordinal scale
measures a variable along an ordered scale [positions in a winning tace + Mohs scale of hardness]
interval scale
measures a variable along an ordered scale, equal diff between points on the scale are equal + there is no natural zero point [ temp measured in °C + years measured in A.D.]
ratio scale
measures a variable along an ordered scale + equal ratios are equal + natural zero point [most physical measurements: time, mass, length]
what level of measurement do psychologists use
some osychological attributes can be measured at interval / ratio level [physiological + behavioural measures]
but most psychologival measures are ordinal at best [self report]
purpose + dangers of an experiment
to establish cause and effect relationships
dangers: cofounding + inappropriate variables
cofounding variables
factors that are not of primary interest in a study but can influence both the IV and DV potentially leading to a false or misleading conclusion. They create a spurious association between the variables being studied, making it difficult to determine the true cause-and-effect relationship.
inappropriate variables
irrelevant or not suitable for the research question or analysis being conducted. These variables can distort results, introduce bias, or lead to invalid conclusions if included in a study.
what do we desire
clearly identifiable IV and DV
closely matched control conditions
no confounding variables
appropriate counterbalancing
easy to analyse
difference between experiment design and procedure
they refer to diff aspects of setting up and conducting an experiment
ed-overall plan or structure of experiment
p-step-by-step instructions that are followed during the experiment to implement the ed
flow of order of designing experiment from hypothesis
hypothesis + experiment design + procedure [manipulation + measure + task + stimulus selection] + hypothesis testing + analysis [statistics]
steps for designing an experiment
determine hypothesis + identify dependent and independent variables + identify potential confounding variables + devise control for confounding variables + determine appropriate experiment design + plan statistical analysis and check that design can be analysed + devise counterbalancing strategy for things that cannot be controlled + devise randomisation strategy for things that cannot be controlledor counterbalanced + define procedure (manipulations, measures, task etc)
potential cofounds in experiments
order effects: tiredness + time of day + habituation + training effects
group differences
how to remove effects of cofounds
vary order of conditions
counterbalancing / randomisation
match participants
include baseline measurements
measure relative perfomance
control group
3 types of participants
diff participants - independent samples + between subjects
same participants - paired samples + within subjects
multiple groups of participants [but each do several conditions] - mixed design + split-plot
what is the type of ANOVA decided by [in terms of number]
the number of variables / factors
single - one way
what are levels
number of different values / groups / conditions for each factor
difference between factors and levels
f - distinct IVs [ each variable can take multiple values]
l - values that you plan to test
factors
variable that is being manipulated or categorized in an experiment to observe its effect on the outcome. the condition that you're interested in testing
older adults will have a poor short term memory compared to younger adults - IV + DV?
DV - memory ability
IV - age
older adults shorter term memory - potential cofounding variables
slowed reaction times in older adults
tiredness
education level
IQ
senile dementia
older adults short term memory - control for cofounding variables
measure baseline RT / use per-cent correct measure
use general population as control group match for Ed level
match IQ and screen for demntia
baseline measurements
initial set of data collected before any intervention or treatment is applied in a study, serves as a reference point to compare against future measurements or outcomes after intervention is applied
purpose of baseline measurements
help establish the starting conditions of the participamnts, you can track changes or effects that occur due to the intervention over time [ act as a control / point of comparison for evaluating impact]
older adults short term memory - appropriate experiment design
independent samples, 1 factor, 2 groups
diff participants - 75+ / 20-30 year olds [independent samples + between subjects]
single factor [memory recall] - one way ANOVA
[education level, IQ and snility are control variables not IVs
older adults short term memory - stats analysis check
independent samples t-test or Mann-Whitney test
how can you plan a statistical analysis and check that the data can be analysed
define research question and hypothesis
identify the type of data
determine the statistical test
check assumptions for the chosen test
prepare the data
assess sample size and power
exploratory data analysis
perform statistical analysis
interpret results
draw conclusions and report findings
older adults short term memory - plan statistical analysis
mixed deisgn ANOVA [check that design can be analysed]
older adults short term memory - cofounding variables + counterbalancing & randomisation strategy
cv - tiredness
cs - blocked design
rs - interleaved design [for things that cannot be controlled / counterbalanced]
general aim of counterbalancing
each participant is presented the levels of the independent variable in a different order, where possible, all orders are used. [in a between participant design the orders used are the same for each group]
full counterbalance with 2 / 3 / levels
each condition appears in each position with all possible orders included [minimum 2 participants per group] [ for 3 levels - 6 participants] [for 4 levels - 24 participants in each group]
blocked conditions
participant undertakes a batch or block of trials belonging to the same condition followed by a batch for another condition
reduced uncertainty
increases habituation
good for spitting conditions across sessions but be careful of 'time of day' effects
interleaved conditions
participants see all conditions each session in either a counterbalanced or randomised order
reduces habituation
increases uncertainty
can still be split into multiple sessions
counterbalancing between participants
each participant undertakes conditiond in a diff order all orders presented but not all participants see all orders
counterbalancing within participants
in some experiments - especially those involving imagine and EEG the presentation order for individual trials / stimuli matters
brains recation to one stimulus may affect its response ti a subsequent stimulus
counterbalance the order of stimuli within a test sesion [interleaved presentation]
aim is to show all possible sequential orderings to ALL the participants
alternative to counterbalancing
you can present conditions in a random order
especially useful when there are a lot of conditions
works best with interleaved conditions such that [potentially] each consecutive trial belongs to a diff condition
can also be applied to blocked conditions such that blocks appear in a diff raandom order for each person ]usually doe such that all conditions appr=ear equally often
summary of the 'design' element
separate from its procedure
refers to number of factors to be tested + levels of each factor whteher all participants take part in all conditions
ensures hypothesus is tested with adequate control
determines how to analyse the results
controls help with cofounding variables
counterbalancing + randomisation to avoid cofounds [order]