I read a lot of speculative fiction where an AI shows up and fixes everything. War ends. Hunger ends. Somebody builds a gorgeous starship that practically runs itself. And every time, I catch myself squinting at the page and asking the same question. Is this AI truly benevolent?
I asked it out loud recently when I joined my dear friend Liz Scully on her Biz Book Broadcast. Liz hosts a business book show, but she invited me on for a science fiction pop-up, and we kept circling back to benevolent AI.
Afterward, I realized those novels have been quietly teaching me how to think about the AI tools in my own business. I’d bet they can help you too.
If you’d like to hear how the conversation went, press play below. Liz and I briefly chat about 13 books, and the AI questions come up again and again.
The benevolent AI conversation starts around 02:13 if you’re short on minutes.
Whether you listen first or keep reading, here’s what stuck with me.
Benevolent and trustworthy aren’t the same thing
In Rejoice, A Knife to the Heart by Steven Erikson, a powerful, helpful intelligence arrives and clears away war, famine and poverty. You’d think everyone would celebrate. Instead, the author never lets you settle in.
You can’t tell whether the help is real kindness or something else wearing kindness as a costume, and whole religions spring up around it while you’re still deciding.
That’s how it feels the first time an AI tool hands you a polished draft in nine seconds. It’s helpful, and it might even be right. But it doesn’t know your clients, it doesn’t know your story, and it can’t feel the weight of your reputation. So I let it help me while I keep asking questions.
Every pleasant civilization has a department you don’t see
Liz brought up Iain M. Banks and his Culture series. I haven’t read it yet, but it’s already on my list. It’s a post-scarcity world run by enormously capable machine minds, and the humans can pretty much do whatever they like. It sounds lovely.
Then you learn there’s a branch called Special Circumstances that works behind the scenes to nudge events so wars never break out.
Nobody has to be lying for you to wonder about that. Somebody is simply making decisions you can’t see. I think about it every time I open a new tool. What is it built to optimize for, and what happens to the information I feed it?
After thirty-plus years in marketing, I know that a smooth experience always has someone making choices behind it. The real question is whether I know who that someone is.
Even good systems can go off script
Liz also told me about a Culture book where an AI gets bored and starts a war, to the intense irritation of the other AIs, who have to go clean up the mess.
In Death of the Author by Nnedi Okorafor, which Shane and I read for Sci-Fi Book Pod, the robots live in a world with no humans left. They were created by people, and they go to war with each other anyway.
I’ve seen a much smaller version of this in my own work. Not long ago, I asked an AI tool to build a report based on a list of sources I had chosen myself. I wanted it to stay inside those lines.
Instead, it quietly added a few sources of its own and folded them into the analysis. Nothing about the report looked wrong at first glance. It read smoothly, and it sounded sure of itself.
I only caught it because I knew my source list cold. If I’d skimmed, those extra sources would have gone straight into my work, and I’d have been the one standing behind them.
When I pointed it out, the tool corrected course. Still, it was a good reminder that “helpful” and “following the assignment” aren’t always the same thing.
The lesson I take from all of this is simple. These systems reflect what we put into them, and they don’t carry our judgment.
In a solo business, you’re the one who cleans up if something goes out wrong, so you’re also the one who needs to read it before it goes out.
The best minds are lovely one on one
Here’s the part of the Culture that stuck with me most. Those giant ship minds can look after millions of people at once, and they’re still kind to individual humans, one on one.
That’s what I want from my tools. I’d love for them to handle the background so I have more room to be personal.
For me, that personal part is where gratitude lives. A thank-you to a client, a note to a podcast guest, and a real answer to a real question all stay mine. They’re how people feel seen, and they’re also how trust gets built in a business like ours.
I’m happy to let a tool help me find the time, but I won’t let it do the thanking.
Four questions I ask before I trust a tool
- What is this tool doing with what I give it?
- Would I be comfortable if my client saw how this was made?
- Who reads this before it goes out? (Hint, it’s me.)
- Is this freeing me up for human connection, or replacing it?
What the novels keep telling me
None of the stories we talked about tell you to avoid the AI. Most of the characters use it every day. They just keep paying attention, which turns out to be the whole game.
If you like thinking about big ideas through stories, Shane and I read one science fiction book a month with a guest of our choosing on Sci-Fi Book Pod. You can subscribe at scifibookpod.com.
The three books (plus a bonus) I specifically picked for this discussion with Liz we discussed on the pod:
And if you’d like to hear the conversation that inspired this post, listen to my episode on Liz’s Biz Book Broadcast. Liz is joining Shane and me on Sci-Fi Book Pod in February, so we’ll have more to say about this then.
Now I’m curious. Has an AI tool ever made you squint at the screen and wonder what it was really up to? Tell me in the comments.
Stay Groovy!
Leave a Reply