Constant sum question results analysis - please help

[issta]

I ran a study with a constant sum question: participants had to divide 100 units (I called the units USD) between 5 possibilities. In other words, they had to bet on five possibilities: 1 object out of 5 has a property vs. 2 objects out of 5 have a property vs. 3 objects out of five have a property vs. 4 objects out of five have a property vs. 5 objects out of 5 have a property.

Next, I ran a paired samples t-test to compare the answers (or 'bets') on two possibilities, namely that 4 objects out of five have a property vs. that 5 objects out of 5 have a. property. A reviewer pointed out that I cannot use a paired-samples t-test because the answers are not independent of one another: your next bets are dependent on the previous ones as it all must sum to 100 (the bet on 5 is dependent on how much you already bet on 4). Which significance test should I use then? Please help!

 

[vclaire]

I’ve been thinking about it for days, but I’m not sure if the argument articulates exactly where the problem may be (if there is one)

The paired t-test allows a correlation between the two samples, and it is very common that the second variable is dependant on the first one (in some degree) A negative correlation is expected in your case. If only two events could be bet on, the correlation would be r = -1. (you can search in google for paired t-test with negative / strong correlation)

The question that arises is whether you randomized the questions with the probabilities, i.e., was there a case where you asked 1/5 first, and was there an other case where you asked 3/5 first, or 5/5 first?
If not, it may be problematic not at the level of test selection but at the level of research methodology. Since then you would not be able to separate the effect of seriality from the effect probability - and I don't think it's possible with any test.

Of course, hypotheses rarely stays with a single t-test. For example if you examined the difference between 4/5 and 5/5 in different groups, you are actually interested in the interaction of the group and probability effect, and in this case your results (especially the interaction but not the main effect of probability) can be interpreted (with caution) despite the presence of seriality.

 

[vclaire]

Would you mind describing in details how you conducted your experiment? E.g.
1. how many participants you had?
2. How many trials each participants had (e.g. how many times did they have to divide the 100 USD?
3. In each trial - where the events displayed simultaneously or serially? E.g. Did they see the option of 1/5; the option of 2/5 and 3/5 and 4/5 and 5/5 at the same time or did they see only one at a time?
4. Was the display of options randomized? E.g. Did they see 1/5 first in each trial or did it vary which option came sooner?
5. Was it fully randomized or semirandomized? E.g. did the randomization vary for each participant?
Example for semirandomized:
For each participant it was the same:
1. trial: 1/5 then 2/5 then 4/5 then 3/5 then 5/5
2. trial: 2/5 5/5 1/5 4/5 3/5
etc.
Example for fully randomized:
First participant:
1. trial: 1/5 then 2/5 then 4/5 then 3/5 then 5/5
2. trial: 2/5 5/5 1/5 4/5 3/5
etc.
Second participant:
1. trial: 5/5 then 4/5 then 2/5 then 1/5 then 3/5
2. trial: 4/5 3/5 1/5 5/5 2/5
etc.

And would you mind describing your hypotheses?