Bayes factors
Suppose a rational agent makes an observation, which changes the subjective probability she assigns to a hypothesis H. In this case, the new probability of H is usually sensitive to both the observation and the prior probability. Can we factor our the prior probability to get a measure of how the experience bears on the probability of H, independently of the prior probability?
A common answer, going back to Alan Turing and I.J.Good, is to use Bayes factors. The Bayes factor B(H) for H is the ratio (P'(H)/P'(not-H))/(P(H)/P(not-H)) of new odds on H to old odds. Thus the new odds on H are the old odds multiplied by the Bayes factor. For example, if the prior credence in H was 0.25 and the posterior is 0.5, then the odds on H changed from 1:3 to 1:1, and so the Bayes factor of the update is 3. The same Bayes factor would characterise an update from probability 0.01 to about 0.03 (odds 1:99 to 1:33) or from 0.9 to about 0.96 (odds 9:1 to 27:1).
Bayes factors satisfy the minimal conditions for an answer to our question:
(1) The new probability P'(H) is determined (in a simple, fixed way) by the old probabilities P and the Bayes factor B(H).
(2) A statement of the form B(H)=x entails nothing concrete about prior probabilities, except that P(H) is not 1.
Bayes factors also satisfy an attractive commutativity condition.
(3) Let E and F be two observations changing P(H) first to P'(H) and then to P''(H). In this case, the final result P''(H) is determined by the initial probability P(H) together with the Bayes factors for the two updates, irrespective of which Bayes factor belongs to E and which to F.
(Proof: consider two sequences of updates, one leading from P(H)=x via P'(H)=y1 to P''(H)=z1, the other from P(H)=x via P*(H)=y2 to P**(H)=z2. Let a, b, c, d be the Bayes factors for H in these four updates (so a = (P'(H)/P'(not-H))/(P(H)/P(not-H)), and so on). Since new odds result from previous odds by multiplication with Bayes factors, this means that z1/(1-z1) = b a x/(1-x) and thus z1 = b a k / (1 + b a k), where k = x/(1-x). Analogously, z2 = d c k / (1 + d c k). If the Bayes factors a and b are identical to d and c respectively, then b a = d c, and thus z1=z2.)
This somewhat strengthens the sense in which Bayes factors are independent of the priors. But in another, more intuitive sense, Bayes factor are not at all independent of the priors.
The direction and magnitude by which an observation affects the probability of a given hypothesis generally depends on the agent's background beliefs. Suppose you briefly see a table cloth in a dimly lit room. You can't clearly tell its colour, but it seems to be blue or maybe green. In the absence of relevant previous information, your new degree of belief in the hypothesis H that the cloth is blue might then change from 0.2 to 0.6. On the other hand, if this is the fifth time you've looked into the dimly lit room, your credence in H may remain almost unchanged at a little above 0.6. Alternatively, if you earlier saw the cloth in bright daylight where it looked green, your credence in H might remain unchanged at well below 0.1. Finally, if you have reason to believe that the light in the room makes green things look blue and vice versa, your credence in H might change from 0.2 to 0.1. The Bayes factor for your update is very different in these four cases. In the first it is 8, in the second and third 1, in the fourth around 0.44.
So the Bayes factors associated with an observation strongly depend on the agent's prior beliefs. That is, we cannot use Bayes factors to characterise the evidential relevance of an observation to a hypothesis in a way that is neutral on the background beliefs of the agent making the observation.
This is what Hartry Field overlooked in his 1978 paper on Jeffrey conditioning (as pointed out in Garber 1980). Strangely, it also seems to be overlooked by Good and Jeffrey when they suggest that Bayes factors can be used to communicate or transfer the effect of experience between different agents (e.g. on pp.7-9 of Probability and the Art of Judgment and pp.55-57 of Subjective Probability: The Real Thing), and by Ilho Park in his very interesting 2013 paper on how to formulate the Reflection Principle.
Consider a situation where you make an observation, and I would like to somehow take into account what you observed. Let's say you get to have a look at the table cloth in the dimly lit room and I don't. It would be nice if you changed your mind by conditioning on some observation sentence capturing the precise content of your visual experience; then I could take into account your observation by conditioning on this same sentence. According to Jeffrey, there is in general no such sentence: your experience directly changes your degree of belief in the hypothesis that the cloth is blue, without going through any observation sentences. Suppose you don't have any relevant prior information about the table cloth, and your credence in H (that the cloth is blue) changes from 0.2 to 0.6. Your new credence in H obviously depends on your prior credence, so I can't rationally take into account your observation by simply adopting your new credence in H. Instead, Jeffrey suggests that I should multiplying my own odds on H by the Bayes factor of your update, which is 8. But the Bayes factor of your update also strongly depends on your priors. In the example, it especially reflects the fact that you had no prior information about the cloth. If, unlike you, I have other reasons to believe that the cloth is blue (or not blue), then the inconclusive evidence you gathered should hardly affect my credence at all. Similarly, if I have reason to suspect that the light in the room makes green things look blue, the evidence you gathered should lower my odds on H. In either case, multiplying my prior odds by 8 would not adequately take into account your evidence.
Now Jeffrey doesn't quite say that I can always use your Bayes factors to take into account your new evidence. He says: "Others who accept your response to your experience, whether or not they share your prior opinion, can multiply their own prior odds [...] by your Bayes factor to get their posterior odds, taking account of your experience." Emphasis added. So I suspect he would have responded to the above counterexample by saying that this is not a case in which I "accept your response to your experience" in the relevant sense. The problem is that we hardly ever accept other people's response to their experience in this technical sense. The proposal works if I happen to have the exact same priors concerning the situation in which you make your observation. But then I could just as well adopt your posterior probability P'(H) instead of your Bayes factors. The proposal also works in a few special cases where we have different relevant priors, but it falls far short of a general solution.