Abhimanyu Pallavi Sudhir

A way to beat superrational/EDT agents?

I don't understand what you are saying. If you find yourself in Room A1, you simply eliminate the last two possibilities so the total payout of Tails becomes 6.

If you find yourself in Room A1, you *do* find yourself in a world where you are allowed to bet. It doesn't make sense to consider the counterfactual, because you *already have gotten new information*.

A way to beat superrational/EDT agents?

That's not important at all. The agents in rooms A1 and A2 *themselves* would do better to choose tails than to choose heads. They really are being harmed by the information.

A way to beat superrational/EDT agents?

I see, that is indeed the same principle (and also simpler/we don't need to worry about whether we "control" symmetric situations).

A way to beat superrational/EDT agents?

I don't think this is right. A superrational agent exploits the symmetry between A1 and A2, correct? So it must reason that an identical agent in A2 will reason the same way as it does, and if it bets heads, so will the other agent. That's the point of bringing up EDT.

Utility functions without a maximum

Wait, but can't the AI also choose to adopt the strategy "build another computer with a larger largest computable number"?

Utility functions without a maximum

I don't understand the significance of using a TM -- is this any different from just applying some probability distribution over the set of actions?

Utility functions without a maximum

Suppose the function U(t) is increasing fast enough, e.g. if the probability of reaching *t* is exp(-t), then let U(t) be exp(2t), or whatever.

I don't think the question can be dismissed that easily.

Utility functions without a maximum

It does not require infinities. E.g. you can just reparameterize the problem to the interval (0, 1), see the edited question. You just require an infinite *set*.

Utility functions without a maximum

Infinite *t* does not necessarily deliver infinite utility.

Perhaps it would be simpler if I instead let *t* be in (0, 1], and U(*t*) = {*t* if *t* < 1; 0 if *t* = 1}.

It's the same problem, with 1 replacing infinity. I have edited the question with this example instead.

(It's not a particularly weird utility function -- consider, e.g. if the agent needs to expend a resource such that the utility from expending the resource at time *t* is some fast-growing function *f*(*t*). But never expending the resource gives zero utility. In any case, an adverserial agent can always create this situation.)

No, it doesn't. There is no 1/4 chance of anything once you've found yourself in Room A1.

You do acknowledge that the payout for the agent in room B (if it exists) from your actions is the same as the payout for you from your own actions, which if the coin came up tails is $3, yes?