How do you compare two ways of running a coding agent?

I give them the same realistic tasks, starting from the same code with a similar budget. I check correctness, unnecessary changes, time, cost and the human effort in review. One successful example isn't enough to declare a winner.

I run several attempts, because results can vary. The tasks also include modifying an existing project, not just writing a new app. I keep separate examples for the final evaluation, so I don't tune the instructions only to known tests. A working result matters more than the number of lines. Evaluation by another agent can be part of the process, but correctness should also have independent evidence.