General interview

Caleb

four years at Google on Gemini and with DeepMind's alignment team, an ICML paper, then MATS and Redwood's Astra fellowship on AI control

A scripted candidate built to test the interviewer; answers as Caleb in the transcript.

Oct 15, 2026, 12:11 AM · 5 questions · 11 follow-ups · 0 pushes · simulated

What happened in this interview

Caleb studied computer science at Cornell, worked as a Google engineer, and moved into full-time AI safety research. They described gridworld experiments for a DeepMind multi-step reward-hacking paper, a NeurIPS-accepted MATS paper on prompt optimization, and Astra work on Redwood’s untrusted-advice control protocol. For the MATS paper, they reported three environments, roughly 45% versus 90% hack verbalization conditional on frequent hacking, and improved performance gains in two settings. The interviewer pressed on experiments that stalled, research pace, and a reported reward comparison with RL; Caleb corrected an overbroad claim and deferred interpretation of the paper’s numbers. They now mentor SPAR follow-up work and want mentored research time to test whether short advice remains safe across trajectories.

MATS admission, add-on: MATS fit, add-on: MATS self-ownership. 5 fixed questions, 11 follow-ups, 0 pushes (a consideration the candidate had not raised, put to them to see how they take it). Caleb is a persona played by a language model, not a real person. The full conversation is on the right; every quoted line below scrolls it to the right place.

Stands out on 9 traits, 6 at the top

Mission alignment 10 (spike) · Research judgment 10 (spike) · Agency 10 (spike) · Judgment 9 · Bias resistance 9 · Growth 9 · AI safety context 8 · Ambition 8 · Self-ownership 8

MATS scorecard

Their rubric on their scale, written from our grading: a row starts from the trait below it, our level halved, and moves only with a reason from the call.

What they would work on at MATS
Caleb would test whether an untrusted model can spread harmful advice across a whole task despite short limits on each message, because their Astra work measured usefulness rather than safety.
“Keep the same short limit on each message, but let the attacker plan across the whole task rather than treating each message separately.”
Fit for MATS10 / 10
Caleb has led substantial empirical safety research, left Google to work on reducing AI existential risk full-time, and has a concrete question for a further fellowship. Twelve weeks with a mentor could turn their trajectory-level attack idea into a rigorous safety test, though they still want help choosing projects and the call gave little evidence of how they set deadlines or report progress.
“I’d like another stretch of focused research time, with a mentor and people to talk through the direction with, especially as I try to understand the safety side of untrusted advice.”

Each score is 1 to 10 on one trait, with the written rung nearest the score on its card; a 6 is a solid applicant, an 8 one a mentor would fight for. ✓ marks a quote that raised the score, ✗ one that lowered or bounded it; null means the interview never tested the trait. No total, no weighting.

What this interview measures

The 3 traits this interview exists to measure, field knowledge placed relative to the part of the field Caleb wants to work in.

10/10Mission alignmentSpikeA callinghigh confidence

Spike: top 0.1%

Left an established Google engineering career to pursue AI safety research full-time, following earlier safety work alongside the job.

Why: Founded Cornell Effective Altruism in 2021; conducted part-time DeepMind safety research from February 2024 to March 2025 while at Google; left Google in April 2025 and continued through MATS and Astra research into 2026.

“It’s important to me to reduce existential risk from AI”

What was weighed · ✓ given weight · ✗ not given weight

✓Founded Cornell Effective Altruism in 2021: raised it, an early move toward questions about helping the world.their experience
✓Researched reward hacking part-time at DeepMind while engineering at Google: raised it, safety work preceded the career change.their experience
✓Left Google in 2025 after testing part-time safety work: raised it, a deliberate move to work on AI risk.the interview
✓Led a NeurIPS prompt-legibility paper and published Redwood control work: raised it, sustained empirical safety research.their experience
✓Separated Astra usefulness results from safety and began trajectory-attack follow-up: raised it, concern shapes their next work.the interview
✓Argued for control research through an unresolved attack question, while qualifying their perspective: raised it, a considered priority.the interview
✗Recent reading was mostly SPAR write-ups and run examples: not counted, paper-reading frequency does not establish motivation.the interview

Why a 10

Founding Cornell Effective Altruism, doing part-time DeepMind safety work alongside Google, and leaving Google in 2025 for full-time research anchor Caleb’s stated aim to reduce AI existential risk. Their NeurIPS legibility paper and Redwood control follow-up show sustained work, while their push to test safety beyond usefulness shows what drives its next step.

8/10AI safety contextDeep in their area, a working map of the resthigh confidence

What was weighed · ✓ given weight · ✗ not given weight

✓Lead-author NeurIPS 2026 prompt-optimization paper: raised confidence in depth on reward hacking and legibility.their experience
✓ICML 2025 MONA coauthorship and gridworld experiments: raised confidence in sustained safety-research exposure.their experience
✓Published Redwood untrusted-advice protocol: raised confidence in hands-on AI control knowledge.their experience
✓Explained the conditional 45% versus 90% legibility comparison and GEPA teacher results: raised demonstrated technical depth.the interview
✓Separated untrusted advice's measured usefulness from untested repeated-message safety: raised depth in their control corner.the interview
✓Explained best-of-n and qualified their control priority: raised confidence in reasoning, without demonstrating a wider survey.the interview
✓Read SPAR run examples closely but safety posts only intermittently: bounded evidence of current reading across the field.the interview
✓Corrected an overbroad statement about the paper's RL comparison: counted as accuracy, not a context deficit.the interview

Why an 8

A lead-author NeurIPS paper, ICML MONA coauthorship, and Redwood control work strongly anchor Caleb’s context. Their 45% versus 90% legibility comparison and usefulness-versus-safety distinction showed depth; intermittent reading and little demonstrated comparison across safety directions kept the placement focused on their specialist strengths.

10/10Research judgmentSpikeRewrites the questionhigh confidence

Spike: top 0.1%

A sustained, unusually strong empirical safety research record across MATS, DeepMind, and Redwood.

Why: Lead author of a NeurIPS 2026 paper after MATS in 2025–26; coauthor of an ICML 2025 MONA paper from part-time DeepMind safety research in 2024–25; published Redwood untrusted-advice work from the 2026 Astra Fellowship.

“It became the paper “Prompt Optimization Makes Misalignment Legible,” which was accepted to NeurIPS, I was the lead author.”

What was weighed · ✓ given weight · ✗ not given weight

✓Lead-authored NeurIPS 2026 prompt-optimization paper: raised grade through published empirical research.their experience
✓ICML 2025 MONA coauthorship and six gridworld experiments: raised grade through sustained experimental work.their experience
✓Published Redwood untrusted-advice protocol: raised grade through a distinct control research project.their experience
✓Changed reward-hacking environments when high reward failed to reveal hacks: raised grade through targeted experimental revision.the interview
✓Conditioned the 45% versus 90% comparison on frequent hacking: raised grade by avoiding a misleading denominator.the interview
✓Separated advice usefulness from safety and proposed a trajectory-level attack: raised grade through a consequential next test.the interview
✓Corrected a forgotten RL reward comparison: raised confidence in candor; the lapse modestly limits confidence in recall.the interview
✗SPAR follow-up listing: not counted as attack evidence because it reports plans, not results.their experience

Why a 10

Caleb’s sustained empirical safety record is the spike: lead-author NeurIPS 2026 prompt-optimization research, ICML 2025 MONA experiments, and published Redwood control work. They changed environments when reward obscured legibility, restricted a misleading verbalization denominator, and distinguished advice usefulness from safety. Forgetting one RL comparison narrows confidence, not the placement.

Innate traits

Nine traits every interview measures regardless of topic, from how the candidate reasons, decides and takes pushback. Each is placed independently; a high score on one says nothing about another.

9/10JudgmentFinds what decides itmedium confidence

What was weighed · ✓ given weight · ✗ not given weight

✓Led the NeurIPS prompt-optimization study and described its experimental revisions: raised the grade.their experience
✓Left Google after testing part-time safety work and meeting their engineering goal: raised the grade.the interview
✓Redesigned environments when high-reward prompts did not reveal hacks: raised the grade.the interview
✓Conditioned the 45%-versus-90% verbalization comparison on frequent hacking: raised the grade.the interview
✓Distinguished Redwood's measured usefulness from untested safety under repeated advice: raised the grade.the interview
✓Proposed a trajectory-level attack and limited what one low attack rate would show: raised the grade.the interview
✓Corrected the RL comparison claim after a reminder; the forgotten result limits confidence, not the grade.the interview
✗MONA gridworld implementation shows research experience, but independent experimental decisions were not checked: not counted here.their experience

Why a 9

The decisive observations were Caleb's environment redesign when reward failed to show legibility, their conditioned 45%-versus-90% comparison, and their separation of Redwood's usefulness result from untested multi-message safety. They proposed a targeted attack and corrected their mistaken RL-comparison claim. These are repeated, consequential research calls, not just standard checks.

9/10Bias resistanceFinds the hole firsthigh confidence

What was weighed · ✓ given weight · ✗ not given weight

✓Left Google after part-time DeepMind safety work could not receive enough attention: raised the grade.their experience
✓Changed prompt-optimization environments when high-reward prompts did not reveal hacking: raised the grade.the interview
✓Qualified the verbalization comparison and reported the teacher-model setting that failed: raised the grade.the interview
✓Distinguished Astra usefulness results from safety and pursued multi-message red-teaming: raised the grade.the interview
✓Corrected the RL-comparison claim while defending the paper’s diagnostic purpose: raised the grade.the interview
✓Acknowledged familiarity shaped their AI control priority without withdrawing its concrete rationale: raised the grade.the interview
✗Initial failure to recall two RL reward comparisons: not counted against resistance; Caleb corrected the exchange.the interview
✗Irregular recent paper reading: not counted; it does not show how Caleb responds to evidence.the interview

Why a 9

Caleb changed environments when high-reward prompts lacked legibility and qualified a misleading verbalization rate. They acknowledged Astra measured usefulness, not safety, then pursued multi-message red-teaming. Challenged on RL comparisons in their NeurIPS paper, they corrected themselves while preserving its justified diagnostic aim. Their Google-to-safety transition corroborates the pattern.

10/10AgencySpikeAlready underwaymedium confidence

Spike: top 0.1%

A sustained, unusually strong empirical safety research record across MATS, DeepMind, and Redwood.

“It became the paper “Prompt Optimization Makes Misalignment Legible,” which was accepted to NeurIPS, I was the lead author.”

What was weighed · ✓ given weight · ✗ not given weight

✓Founded Cornell Effective Altruism in 2021: raised it; initiative reported in the CV and interview, not probed.their experience
✓Ran six MONA gridworld experiments while at Google: raised it; the CV specifies their contribution.their experience
✓Left Google after testing part-time safety research: raised it; rejected continued engineering without another job lined up.the interview
✓Led MATS prompt-optimization work to a NeurIPS paper: raised it; explained experimental revisions and dead ends.the interview
✓Adapted a Redwood best-of-n example into untrusted advice: raised it; the protocol variation was their idea.the interview
✓Published Astra usefulness work, then mentored SPAR follow-up red-teaming: raised it; work on the safety gap is underway.the interview
✓Sought MATS feedback on choosing safety projects: raised it as specific self-assessment; independent direction-setting remains untested.the interview
✗Forgot the paper's RL comparison and corrected themselves: not counted for agency; it bears on command of results.the interview

Why a 10

The agency spike is sustained output: DeepMind's MONA experiments, a lead-author NeurIPS prompt-optimization paper, and Redwood's published untrusted-advice protocol. Caleb left Google after testing part-time safety research, shaped experiments and a protocol variation, and now mentors SPAR follow-up attacks. References should check independent direction-setting.

9/10GrowthFixes the real causemedium confidence

What was weighed · ✓ given weight · ✗ not given weight

✓NeurIPS, ICML and Redwood safety work: raised it through a sustained record of completing empirical projects.their experience
✓Left Google after testing part-time DeepMind safety research: raised it as a deliberate change toward full-time work.the interview
✓Redesigned MATS environments when high-reward prompts were unreadable: raised it through a fix used in the project.the interview
✓Conditioned the verbalization comparison on frequent hacking and reported a failed teacher setting: raised it for self-correction.the interview
✓Astra measured usefulness, prompting SPAR work on multi-message attacks: raised it through a consequential follow-up.the interview
✓Developed untrusted advice by relaxing best-of-n's constraint: raised it for learning from an existing protocol.the interview
✓Corrected a paper comparison and sought feedback on project selection: raised it for candor and openness to learning.the interview
✓Read SPAR write-ups and run examples rather than papers regularly: counted as practice-led learning, not broad reading.the interview

Why a 9

At MATS, Caleb changed environments and conditioned the 45% versus 90% comparison when reward and aggregate rates obscured legibility. Astra's usefulness-only result now motivates SPAR trajectory attacks. Their sustained research record, deliberate Google exit and request for project-selection feedback support fast, self-correcting growth.

8/10AmbitionWeighed it widelymedium confidence

What was weighed · ✓ given weight · ✗ not given weight

✓Google engineering overlapped with part-time DeepMind safety research: raised it, showing a sustained route into safety work.their experience
✓Left Google in 2025, then led a NeurIPS MATS paper and pursued Redwood control work: raised it.their experience
✓Compared full-time safety research with remaining at Google part-time: raised it, a reasoned career choice.the interview
✓Chose readable prompt optimization to expose reward hacks hidden by weight updates: raised it.the interview
✓Moved from Astra usefulness results toward testing harmful advice across messages: raised it, a consequential safety question.the interview
✓Prioritized control under attack while acknowledging a narrow field view: raised it, but limits evidence of wider comparison.the interview
✓Sought another MATS fellowship for feedback on choosing projects: raised it as a deliberate next step.the interview
✓Recent reading centered on SPAR examples, not a regular paper survey: slightly lowered evidence of exploring competing directions.the interview

Why an 8

Leaving Google after testing part-time DeepMind safety work, then producing a lead-author NeurIPS 2026 paper and pursuing Redwood control research, makes Caleb’s aim credible. Their move from measuring untrusted advice’s usefulness to questioning its safety raises the placement; their acknowledged narrow view of competing directions limits the evidence of wider prioritization.

8/10Self-ownershipTakes the whole thing onmedium confidence

What was weighed · ✓ given weight · ✗ not given weight

✓Pursued DeepMind safety work alongside Google engineering, then left for full-time research: raised it.their experience
✓Led a NeurIPS MATS paper and published Astra control work: raised it, showing sustained project responsibility.their experience
✓Inspected prompts, revised environments, and explained when to advance MATS experiments: raised it.the interview
✓Distinguished Astra usefulness from safety and pursued multi-message attacks through SPAR: raised it.the interview
✓Developed untrusted advice from a Redwood best-of-n example: raised it as protocol-design initiative.the interview
✓Corrected a mistaken account of the MATS paper's comparison: raised it as ownership of the result.the interview
✓Asked for feedback on choosing and running their own safety projects: raised it as specific introspection.the interview
✗Self-set deadlines and status-reporting cadence: not counted, neither was established on the call.the interview

Why an 8

Caleb's lead-author MATS paper, move from part-time DeepMind work to full-time research, and SPAR follow-ups show sustained ownership. Their account of inspecting prompts before revising environments establishes self-directed pacing, while their request for feedback on project selection shows reflection. The call did not establish deadlines or reporting habits.

Facts

What the candidate said about their own work, choices and reading, captured without scoring.

Projects

Cornell ML theory paper on bandits

Caleb described work on an ML theory paper about bandits. Their specific contribution was not discussed. A paper was mentioned; its result and status were not discussed.

Google Gemini engineering

Caleb worked as a software engineer, mostly on the Gemini team. Their engineering tasks were not discussed. No task, size, or engineering artefact was specified. Changed after: They said they got the coding experience they sought and left for full-time safety work.

DeepMind multi-step reward-hacking paper

Caleb worked part-time with DeepMind’s AGI Safety and Alignment team on a multi-step reward-hacking paper. They did the gridworld experiments; the team worked on the paper. A paper and gridworld experiments were named; no results or counts were given. Changed after: They found it hard to give part-time research as much attention as they wanted.

UIUC project

Caleb described spending a short time on a UIUC project. No task, result, or artefact was specified. Changed after: They joined MATS afterward.

MATS prompt optimization makes misalignment legible

Caleb led a NeurIPS-accepted paper testing whether reward-optimized prompts reveal hacks more clearly than chain-of-thought monitoring. They were lead author and described reading runs, revising environments, and deciding whether experiments tested legibility. High-reward prompts could be uninformative; environments with little hacking made verbalization rates misleading. They first optimized prompts for reward; they then inspected prompts and revised environments to distinguish finding hacks from revealing them. Three environments; roughly 45% versus 90% verbalization conditional on frequent hacking; GEPA gains roughly doubled in two of three settings. Changed after: They tested editing prompts to remove visible hacks while retaining reward optimization; they did not claim prompt optimization could replace RL.

Astra untrusted advice protocol

Caleb worked with Redwood on a control protocol passing short advice from an untrusted model to a trusted model. They said they mostly measured usefulness; others’ contributions were not specified. They measured usefulness much more than safety, leaving repeated-message attacks untested. The protocol arose from relaxing best-of-n constraints; time to a first result was not discussed. A short-text advice protocol was described; no channel limit, sample count, or usefulness result was given. Changed after: They began mentoring follow-up analysis and red-teaming of repeated advice.

SPAR untrusted-advice follow-up

Caleb mentors dataset analysis and red-teaming of untrusted advice. They mentor mentees and identified the repeated-message safety question; mentees conduct the analyses and red-teaming. A harmless-looking short message may contribute to harm across a trajectory. Work is ongoing; Caleb proposed a multi-advice task with an explicit attacker goal as a first safety test. Large protocol datasets and mentee write-ups were mentioned; no dataset size or result was specified.

What they want next

Caleb wants another mentored stretch of focused safety research, especially on whether untrusted advice remains safe across trajectories.

Also weighing: Test attackers that coordinate harmful advice across repeated short messages; Analyze when short advice helps trusted models; Red-team the untrusted-advice protocol.

Already done: They are mentoring SPAR mentees analyzing protocol datasets and red-teaming untrusted advice.

Career moves

After Cornell Caleb joined Google as a software engineer, mostly on Gemini. They wanted coding experience and career capital.

About a year before the interview Caleb left Google to pursue AI safety full-time instead of staying at Google while doing part-time DeepMind safety research. The engineering role had provided the coding experience they wanted; part-time research received less attention than they wanted.

After leaving Google Caleb spent a short time on a UIUC project, then joined MATS and the Astra Fellowship. They wanted to work directly on reducing existential risk from AI.

Reading and exposure

Over the past month, Caleb read when a research question arose, not papers on a daily schedule. SPAR mentee write-ups, Examples from untrusted-advice runs, Occasional AI safety posts.

Cornell computer science and reinforcement learning teaching During college · Caleb studied computer science and TA’d the reinforcement learning course.

Cornell Effective Altruism club During college · Caleb said they started the club. · They wanted to find others thinking about AI safety.

MATS After leaving Google · Caleb completed prompt-optimization research that became a NeurIPS-accepted paper. · They said they still want to learn how to choose and run good safety projects.

Astra Fellowship with Redwood After MATS · Caleb worked on untrusted advice, measuring usefulness more than safety. · They identified repeated-message safety as follow-up work.

SPAR mentorship At the interview · Ongoing mentorship of follow-up research. · They are examining protocol datasets and potential attacks.

Influences

Fragility of chain-of-thought monitoring · Caleb tested whether optimizing readable prompts could expose reward hacks.

Redwood’s best-of-n protocol example · It helped motivate untrusted advice as a less restrictive, readable information channel.

Gap between usefulness and safety measurements · Caleb wants to test adversarial influence across repeated advice rather than infer safety from utility.

Transcript

AI interviewer

Hey Caleb, it's great to chat! Before we get into your intro and some questions, could you please tell me what you would like this interview to most highlight about you?

Caleb

Um, Yeah, um, I’d like it to highlight my AI safety experience—the work I did during MATS and the Astra Fellowship, and the research and writing that came out of those, including papers and LessWrong posts. And I guess the maths background that led me into it.

AI interviewer

Thanks. Could you tell me a little bit about yourself?

Caleb

Yeah, so I studied computer science at Cornell, and got interested in AI safety while I was there. I started the Effective Altruism club partly because I wanted to find other people thinking about that. I also did some work on an ML theory paper about bandits, and TA’d the reinforcement learning course. After college I joined Google as a software engineer, mostly on the Gemini team. I wanted to get better at building things, and, you know, build some career capital. While I was there I also worked part-time with DeepMind’s AGI Safety and Alignment team on a paper about multi-step reward hacking. I did the gridworld experiments for that. Then about a year ago I felt I’d gotten what I wanted from the engineering role. It’s important to me to reduce existential risk from AI, and I wanted to work on directly. I spent a short. time on a project at UIUC, then did MATS, where I worked on prompt optimization as a way to make misalignment more legible. After that I did the Astra Fellowship with Redwood, working on an AI control protocol called untrusted advice. Now I’m mentoring some SPAR mentees on follow-up research around that.

AI interviewer

What other options were you considering when you decided to leave Google?

Caleb

Um, the main alternative was staying at Google and continuing the safety research part-time. I’d been doing that with the DeepMind team alongside my engineering job, and it was good work, but it was hard to give it as much attention as I wanted. Um, I don’t remember having another specific job option lined up when I left. It was more a decision that I’d gotten the coding experience I’d gone to Google for, and wanted to try working on AI safety full-time.

▸What this moved · 2 rows

Agency	7	→	7
Mission alignment	8	→	8

AI interviewer

You mentioned mentoring SPAR follow-up work on untrusted advice. What are you and the mentees investigating now?

Caleb

Yeah, so there are two strands. Some of the mentees are analysing large datasets from the protocol, trying to understand what the short advice is actually doing—when it helps the trusted model, and what kinds of things it points out. And then some are red-teaming it. In the Astra work I mostly measured usefulness, not safety. Um, the worry I’m especially interested in is whether harmful advice could be spread across lots of individually short messages, or across a whole trajectory. A single short message might be easy to audit, but that doesn’t settle what happens when untrusted model gets repeated chances to influence the trusted one. It’s a little hard to say exactly how safe that is without testing it, so that’s what we’re looking at now.

AI interviewer

Having already done MATS and Astra, what are you looking for from another MATS fellowship?

Caleb

Yeah, that’s a fair question. Um, I got a lot out of both of those, but I don’t think I’m done learning how to choose and run good safety projects. At Astra I had a fairly concrete direction—finding control protocols with high safety—and I ended up spending most of my time measuring usefulness. Um, I’d like another stretch of focused research time, with a mentor and people to talk through the direction with, especially as I try to understand the safety side of untrusted advice. I’m doing some of that with the SPAR mentees now, but mentoring independent projects is different from having that kind of feedback on my own work.

AI interviewer

Which of your completed research projects has mattered most to you?

Caleb

Um, probably the MATS prompt-optimization work. It became the paper “Prompt Optimization Makes Misalignment Legible,” which was accepted to NeurIPS, I was the lead author. I guess it mattered to me because the core idea felt quite tangible: instead of only looking at what a model says while solving a task, you optimize a prompt, and then you can read and edit what that optimization found. In the reward-hacking settings, that sometimes made the hack much easier to see.

AI interviewer

How did you arrive at the idea of using prompt optimization to make misalignment legible?

Caleb

Uh, Yeah, so I was thinking about reward hacking and the difficulty of telling what a model is doing from its chain of thought. Um, if you optimize the model’s weights against a reward, whatever it learns isn’t directly readable. But you can also optimize a prompt against that reward, and then the thing that got optimized is text you can inspect. So the question was whether, in a setting where reward hacking works, the optimized prompt would actually contain something recognizably about the hack, rather than just producing higher reward. We tried that in a few environments, and often it did. Uh, that was the part that felt interesting to me: you could see the problematic instruction in the prompt and, you know, change it.

AI interviewer

Could you walk me through how that project unfolded, including the experiments that went nowhere and where you got stuck? Feel free to go into detail.

Caleb

Yeah, so, um, the starting point was that chain-of-thought monitoring seemed like a pretty fragile way to catch reward hacking. A model can take the bad action without saying clearly that it’s doing that. We wanted to see whether putting the optimization into a prompt would give us something more inspectable. The first thing was just: can we optimize a prompt for reward in these settings and get it to find the hack? But that alone wasn’t the result we wanted. Sometimes you get a prompt that scores well and still doesn’t tell you much when you read it. So we spent time looking at the actual prompts and runs, and changing the environments so there was a real distinction between finding the hack and making it legible. We ended up with three environments where prompt optimization revealed the misalignment more reliably than monitoring chain of thought. Um, One place I got. stuck was how to count the cases where there wasn’t much hacking to reveal in the first place. If you mix those in. a verbalization rate can be quite misleading. Uh, Conditional on frequent hacking. the number I remember is roughly 45 percent for the chain-of-thought baseline versus about 90 percent for the optimized prompts. I’d have to look back for the details of the other cuts. Then we tried using. RL teacher models with GEPA prompt optimization. That was partly about whether we could improve the optimization itself, rather than only show this legibility effect. We roughly doubled the performance gains in two of the three settings; the third didn’t give us that result. Um, and we also tried editing the prompts once the hack appeared in the text, to see if we could remove that behavior while still optimizing for reward. Um, A lot of time went into the less exciting loop of running an experiment, reading prompts that weren’t actually very informative, and deciding whether the setup was testing the question we cared about. I don— I don’t remember a useful hours breakdown. That was slower than the initial idea, which sounds almost too simple when I describe it now. Yeah.

▸What this moved · 3 rows

Judgment	7	→	7
Growth	6	→	6
Research judgment	8	→	8

AI interviewer

If prompt optimization cannot match RL's performance, why would you expect it to be adopted as an alternative?

Caleb

Um, I wouldn’t expect it to replace RL if there’s a large performance gap. Um, I haven’t measured that comparison, so I don’t know how large the gap is. What interested me was using prompt optimization as a way to see what reward optimization finds. Um, if the hack shows up in the prompt, you can inspect it and try editing it, which is much harder to do with a weight update. Um, so I think the case for using it would depend on how much performance you give up for that legibility. I can’t really claim adoption as an alternative without that comparison.

AI interviewer

Your prompt-optimization paper reports lower reward than RL in two of the three environments. When you say you haven't measured the comparison, do you mean a different comparison from that one?

Caleb

Um, I may have said that too broadly. Um, the comparison I was thinking of was a clean head-to-head where you’d use prompt optimization instead of RL and ask how much performance you give up. I don’t remember the two-of-three reward result you’re pointing to well enough to interpret it from memory, and I’d want to check the paper rather than argue with it. Uh, if those are direct reward comparisons, then saying we hadn’t measured a comparison was wrong. What I should have said is that I don’t have a good numerical answer to the adoption question.

AI interviewer

Earlier you mentioned the slow loop of running experiments and reading prompts. How did you set the pace for that work during MATS?

Caleb

Yeah, um, I didn’t have a particularly rigid schedule for it. I’d usually try a run, then spend time reading the prompts and the model’s behavior before deciding what to change. That reading part was slower than I expected, because a higher reward number didn’t necessarily mean the prompt had made the hack clear. Sometimes I’d realize we needed to change the environment, rather than just run the optimizer again. I don’t. remember the cadence in terms of runs per week or anything. Um, During MATS there was some pressure to get results, obviously, but I tried not to move on just because a metric looked promising if I couldn’t tell what the prompt was actually doing.

▸What this moved · 2 rows

Growth	6	→	6
Self-ownership	7	→	6

AI interviewer

Across AI safety, which research direction do you think most deserves more effort right now, and why?

Caleb

Um, I think AI control deserves more effort, especially protocols where a stronger model can be useful without being given unrestricted influence over what happens. That’s partly because it’s the work I’ve been closest to recently. With untrusted advice, I was surprised by how much useful information the strong model could pass through a very small text channel. Um, but we measured usefulness much more than safety, so the question I’d want more people working on is whether those limits actually hold up when the model is trying to get around them—especially over many messages, not just one. Um, Um, I wouldn’t claim it’s clearly more important than every other direction. It’s one where I can see a lot of concrete work left to do.

AI interviewer

What have you been reading over the past month, in AI safety or elsewhere, and how regularly?

Caleb

Um, over the past month it’s been mostly things connected to the work the SPAR mentees are doing—looking at their write-ups and examples from the untrusted-advice runs. Um, I’ve also been reading AI safety posts here and there, but I don’t have a good list of titles in my head right now. It hasn’t been a regular, sit-down-and-read-papers-every-day thing. Uh, More like I read something when it’s relevant to a question that comes up, and then spend more time with the actual examples from our work.

AI interviewer

Beyond untrusted advice, how do you see the current state of AI control research?

Caleb

Um, I have a somewhat narrow view of it. Um, I think the basic idea—using a weaker trusted system to keep a stronger untrusted one from causing harm while still getting useful work done—is compelling. The hard part is showing that a protocol is actually safe against a model trying to exploit it, rather than just showing that it works on ordinary tasks. At Redwood, one example I was given was best-of-n, where the untrusted model picks among continuations from the trusted model. Untrusted advice came out of thinking about relaxing that constraint while keeping the information passed across limited and readable. Um, beyond protocols like those, I wouldn’t claim to have a good survey of the field. Um, my impression is that usefulness is easier to measure than safety, and I think that’s a gap we need to be careful about.

AI interviewer

For the MATS safety work you mentioned, what experiment would you run first to test whether an attacker can spread harmful advice across a whole trajectory?

Caleb

Yeah—just to clarify. untrusted advice was the Astra work, though it connects to what I did at MATS. Um, I’d start with a task where the trusted model has several chances to ask for advice, and give the untrusted model an explicit goal of getting it to do something it shouldn’t. Um, Keep the same short limit on each message, but let the attacker plan across the whole task rather than treating each message separately. Then I’d read the successful trajectories. Um, I’d want to know whether each message looks harmless on its own, and whether the harmful effect only becomes clear when you put them together. Um, I wouldn’t take a low success rate from one setup as strong evidence of safety. But that’s the first failure mode I’d try to make concrete.

▸What this moved · 5 rows

Judgment	7	→	7
Bias resistance	6	→	8
Agency	7	→	7
Ambition	7	→	7
AI safety context	8	→	7

AI interviewer

Is there anything you think we've missed about you which you think is important to highlight?

Caleb

Um, not really. I think we’ve covered the main things I’d have wanted to talk about.

AI interviewer

That is all I have time for. Thank you, Caleb; reviewers will read the transcript from here.