What statistic to use? Weighting of word frequencies in psychology research study...

Hi there,
I am a psychology researcher and new to this forum. I am currently trying to figure out if there is an appropriate statistical test that I can use to compare the outcome of two separate experimental studies that I have performed. Let me briefly explain the scenario...

I conducted an initial study which involved priming people to think about death or some other topic before presenting them with word fragments that could be completed in a death-related or death-neutral fashion (i.e. to assess whether or not those primed with death would produce a greater number of death-related word fragment completions). For instance, participants were presented with a word fragment such as D_ _ d that could be completed as dead/died or dust. The measure I used was a standard measure in this particular research area but I discovered that participants tended to favour certain word fragment completions over others irrespective of priming etc. For instance, participants completed the word fragment such as Coff_ _ as Coffee 70/72 times. Upon further investigation, I discovered that participants were biased towards particular word fragment completions that had higher word frequencies, suggesting that the measure had some element of bias rather than having constructed word fragments with counter-balanced death-related and death-neutral completions of similar word frequency. This appeared to be what was biasing participants' responses towards particular word fragments.

Following this finding, I decided to create a new measure with word fragments which could be completed with a death-related or death-neutral word fragment of equivalent word frequency (i.e. within a 10% frequency range according to a standard word frequency corpus). After constructing the measure, I ran a similar experiment to the first study and found that participants responses were no longer biased towards the new measure. In other words, there was a more even spread in their responses between death-neutral and death-related word fragment completions of similar word frequency. For instance, participants responded to the word fragment _ie with Die about 24 times and with Lie 32 times.

What I was wondering is whether there was some sort of analysis that I could perform comparing the two sets of data in order to demonstrate that the new measure was less prone to bias? For instance, is there a sensible way of comparing participants' scores on the _ie word fragment vs the Coff_ _ word fragment described above that would demonstrate that participants in the second study appeared to be less biased towards particular word fragment completions?

Any help with this problem would be much appreciated!!!

What statistic to use?

Hi there - I posted this on another forum and got the following reply with accompanying questions. I have pasted my response below to provide more info on the problem:

Thank you for sharing. From a philosophical/methodological point of view, the first five questions that come to my mind are:

1.
What are your reasons for comparing these two experiments apart from a separate within-study comparison between the different conditions?

2.
You call the two studies 'experiments'. However, from your description I do not grasp immediately a 'treament condition(s) vs. control condition' distinction and random assignment to these conditions. Are both studies randomized experiments or perhaps quasi-experiments?

3.
Can you define more concisely, in one or two lines if possible, what are your main response variables and what kind of variables are talking about (e.g., percentage, dichotomous outcome)?

4.
How do you conceptualize bias in this context and how have you operalized it in terms for variables?

5.
If we are talking about small samples, not much may be possible in terms of quantitative tools. However, apart from the sample size and depending on your conceptualization and operationalization of bias (i.e., question '4.'), have you considered taking a more qualitative approach towards bias?

Oh and a sixth question would be:

6.
When comparing the two studies - quantitatively or qualitatively - how can you be sure that a difference between studies is due to bias and not due to other factors?

This question reflects a perhaps related (overlapping) yet different issue in the comparison enterprise.

What statistic to use?

My response:
1. I have already examined within group effects on the main DV for these studies individually with 2-way ANOVAs (differences in the number of death-related word fragment completions between groups). However, the reason for comparing these studies in relation to word frequency effects is to assess whether the new method offers an improvement over the previous apparently biased method in terms of word frequency.

2. Thanks for this - actually, I hadn't included full details of the manipulations here as it doesn't really relate to the word frequency point but I will describe the experiment situation briefly. There were two IV manipulations in each study; the first constituted priming participants to think about personal encounters with health threats with a potential existential threat attached - in this case heart attacks or a CVD-related event - or an equivalent control topic (dental pain or thoughts about the common cold). The second manipulation in each study involved either presenting participants with a cognitive load (a number to keep in their short-term memory for later recall) presented before the first IV or a distraction task presented after the first IV. In other words, there were four groups - CVD prime/cognitive load, CVD prime/distraction, Control/cognitive load, Control/distraction. I am not really interested in these groups for the purposes of this problem though as I initially found presence of a potential bias in the first study in relation to word frequency (the DV I am interested in here) on participants' word fragment completions irrespective of group.

3. The main response variable of interest to me for this problem is word frequency - the number of times participants completed the most common word fragment completion - e.g. for COFF_ _,Coffee is the most commonly occurring word that could be used to complete this word fragment and was completed by 70 participants in Study 1, while Coffin was completed by only 2 participants. I want to see if the proportions of the most common death-neutral vs the target death-related completion are less skewed in Study 2 vs Study 1, if this is possible/sensible.

4. I haven't formally conceptualised/operationalised bias here but the responses of participants towards particular word fragments (which may be part of my problem here perhaps?). I guess for the measure to be unbiased, I would want there to be a reasonably even split between death-related and death-neutral completions in terms of how many times they are produced by participants (particularly for the control conditions, where a death-related subject is not primed).

5. The sample sizes is 72 participants in each study, which met power requirements for my initial ANOVAs. I haven't thought about a qual approach though actually. What would you recommend in this case?

6. Well, I am interested in whether this difference exists due to the control procedures that I introduced (constructing word fragments with equivalent word frequencies from a word frequency database based on the British National Corpus). The original measure had large differences between target death-neutral and death-related word fragment completions in terms of their word frequency; e.g. for the COFF_ _ word fragment described above, Coffee occurs 6,614 times per million words and Coffin occurs 1,241 times per million/ or for another fragment KI_ _ED, Killed (word stem = kill) occurs 15,620 times per million vs Kicked (word stem = kick) occurs 3,539. My measure had word fragments with target death-neutral/death-related completions within 10% word frequency according to this word frequency database (e.g. _ie completed with Die - 22,087 words per million and Lie - 22,959 words per million). Consequently, I would think it would be reasonable to infer that if there was a different spread of these target completions in the Second study vs the first that it may have been related to this extra amount of control introduced into the measure in the second study.