You can't make sense of the world alone with a chatbot.

About this episode

Hi there,

Welcome back to Untangled. It’s written by me, ​​​​​​​​​​​Charley Johnson​​​​​​​​​​​, and valued​​​​​​​​​​​​​​​​​​​​ by ​​​​​​​​members​​​​​​​​ like you. ​​​​​​​Help me make it better?​​​​

This week I’m sharing my conversation with Marina Nitze — former CTO of the Department of Veterans Affairs, senior White House technology advisor, and co-author (with Mikey Dickerson and Matthew Weaver) of Crisis Engineering: Time-Tested Tools for Turning Chaos into Clarity. We talk about the book, sense-making practices, and what actually helps a system change.

As always, please send me feedback on today’s post by replying to this email. I read and respond to every note.

On to the show!

Untangled HQ

* Stewarding AI: The next cohort of Stewarding AI: How to Build Responsible Principles, Workflows, and Practices is coming up. ​Get on the waitlist and be the first to hear when it opens. ​

* Ask Me Anything: Ask me about the knots in your own systems change work (”my funder wants a five-year plan but my system doesn’t work like that — what do I do?”). Ask me about stewarding AI inside your organization (”my board/boss keeps asking what our AI strategy is -- where do I start?”). Ask me about something in the news you want untangled. Or ask me something personal — what I’m reading, what I’ve changed my mind about this year, or what my high-school self — Dashboard Confessional blasting in his headphones — would make of this newsletter. No question is too small or too “I should already know this.” Just use this form.

Deep Dive

You can’t make sense of the world alone with a chatbot.

Marina and I talk about a lot of things (watch it here or on YouTube), including:

* How the Healthcare.gov crisis ended on a shared story rather than any change in the metrics, and what Marina took away from living through it.

* Why nobody in a large org holds an accurate picture of the whole system, and what it takes for a group to see it together.

* Why people can’t be argued out of a comfortable story with facts, and what actually lets a group take up a truer, stranger one.

* How crises loosen what was stuck, why most orgs waste the opening, and what leadership looks like inside that window.

* What AI adoption would look like if it took the author’s account of why organizational change usually fails seriously.

* How a leader should weigh automating against sustaining the human adaptive capacity that automation tends to engineer away.

* What AI does to an org’s ability to know itself when AI summarizes the reports and watches the metrics, and where sensemaking has to stay human.

You should give the entire episode a listen / watch, but I wanted to zoom in on one thread we kept coming back to: what actually allows systems to change amidst a crisis?

When Marina was CTO of the VA, the disability claims backlog kept growing, and the only tool the agency had was mandatory overtime. So she flew around the country and watched claims get processed from start to finish — which, at the time, nobody owned. People owned steps of the process. Nobody owned the process. And nobody had traced the entire thing.

She sat in on medical exams. Doctors would examine a veteran head to toe and then write an eight-page medical essay. When she asked them why, their answer was consistent: I’m a medical expert, writing for an even better medical expert, who will read every detail and make a nuanced decision about this veteran’s compensation. The longer the essay, the thinking went, the better served the veteran.

Except the essay didn’t go to a better medical expert. It went to a claim specialist with no medical training and a very specific form — a form that asked, say, for the number of degrees the veteran’s elbow could bend. Often the number wasn’t anywhere in the eight pages. So the claim got kicked back, and the veteran was sent in for another head-to-toe exam. Not great.

The doctors believed they were serving veterans as thoroughly as possible. So did the claim specialists. Yet together they were grinding veterans through repeat exams and growing the backlog. What fixed this system wasn’t a fact-filled memo, it was sense-making. Marina brought everyone into one room — the doctors, the claim specialists, etc. — and had them demonstrate their part of the process to the other. They left understanding how their well-intentioned actions were contributing to the overall system.

Sensemaking is about how we assemble what’s happening around us into a story we can act on. Marina and her co-authors lean on organizational theorist Karl Weick in the book, who contends that sensemaking runs on plausibility, not accuracy. Given the choice, we’ll take a familiar story that fits some of the facts over a strange or complicated story that fits all of them. The book uses the accident at Three Mile Island as an example: nearly fifty years on, there are still two incompatible stories of that accident — “we were lied to,” and “honest mistakes were made, no harm done.” Both stories are alive and well, and no fact is coming along to settle it.

The sensemaking process didn’t simply offer new information to well-intentioned actors. It was a collision with reality that went deeper than the workflow. Right, the doctors didn’t care about writing the eight-page document. They were protecting their own identity, an image of themselves as the person who serves veterans thoroughly. And now they’re in a room showing them that their contributions were an obstacle to change. A new shared story can’t change the system unless the participant’s identity shifts alongside it. From I’m the expert who serves veterans to I contribute to a process that serves veterans. Turns out, facts are downstream from the shared stories we tell. And all of that is downstream from who we think we are.

Near the end of our conversation, almost in passing, Marina said something I haven’t stopped chewing on since: “You’re not going to be able to sense-make with yourself and your LLM. That’s actually fairly dangerous.” A chatbot simulates a conversation but it doesn’t offer the friction required for you to update your picture of the world, or yourself. Your chatbot doesn’t hold a different map than you do; after a few exchanges it mostly holds yours because its incentives are set to find your story plausible. And there is no room for it to read — no raised eyebrows or dismissive smirks, no stake in the disagreement. Whereas sensemaking is fundamentally social. It travels through tone and body language and power dynamics, which is why Marina insists you cannot work a crisis over Zoom — everyone has to be in the same physical room, looking at the same reality, revealing and picking up on social cues, interpersonal issues, and all the messy things about being a human that cannot be automated away.

Organizations are concerned with the ‘AI risks’ they hear about in the media. How can we minimize ‘hallucinations?’ How can we ‘align’ it with our values? If your a long-time Untangled subscriber, you know I’ve been poking holes in these frames for a long time. Because the real danger of making sense of the world with a chatbot was never that it tells you false things or contradicts your values. It’s how smooth your own story starts to feel.

More soon,

Charley

Work With Me

Here are 3 ways I can help:

* ​​​​​​​​​Advising:​​​​​​​​ I can help you navigate uncertainty, make sense of AI, and steward change in your system.

* ​​​​​​​​​Organizational Training:​​​​​​​​ Everything you and your team need to cut through the tech-hype and implement strategies that catalyze true systems change. (For either Stewarding AI or Systems Change for Tech & Society Leaders)

* ​​​​​​​​​1:1 Leadership Coaching:​​​​​​​​ I can help you facilitate change — in yourself, your organization, and the system you work within.

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit untangled.substack.com