With Passover fast approaching, I've been putting AI to use in my Seder prep. Specifically, I've been using it to discover and unpack different topics of conversation I could explore with my guests.
In this post, I want to showcase what you get when you unleash Claude Code, a software development tool, on this sort of research task.
The Setup
To appreciate what Claude Code brings to the table, I fed it and ChatGPT essentially the same prompt. Here's the prompt:
I'd like to do some research for the upcoming Passover Seder. Looking at around the time of the Civil War, one of the reasons given for the morality of slavery was that the Bible didn't seem to condemn it. In fact, it includes a commandment that the descendants of Ham were to be slaves. Some suggested that the African slaves were these descendants.
Can you give me some quotes and links to texts from rabbis that discuss how Judaism traditionally understood this commandment that the descendants of Ham were to be enslaved? Would this argument that was common in the US make any sense in a traditional Jewish context?
So here I'm taking some vague notation I have and I'm asking AI to flesh it out and back up its response with quotes and links.
As a side note, I find it enraging that a couple of verses in the book of Genesis could be used as moral justification for enslaving hundreds of thousands of Africans in America. But, apparently it was. Or at least that's what I wanted AI to help me unpack.
ChatGPT's Performance
You can find ChatGPT's full response here. In general, it did the job we're used to AI doing and while you can quibble with it (does it really need to start with a compliment?), the fact is that it did hours, if not days of research in a few seconds. It then presented the findings in an organized and thoughtful way. To be anything other than blown away by this result is to have lost sight of what life was like in the Pre-ChatGPT era.
And yet, with this type of response, one always needs to ask a critical follow up question: this sounds good, but is it true?
For example, did Ibn Ezra, writing in the 12th century, really directly address this issue of Black people, slavery and Noah's curse?
To ChatGPT's credit, it provides links to sources as I asked for. The Tablet Magazine article is a valuable read, and I appreciate that the link to Chabad is also Kosher. But many of the other quotes link back to Wikipedia, which any high school student can tell you isn't a reliable source. And while the Tablet and Chabad articles are helpful, are they based on accurate readings of source materials?
Claude Code's Approach
Claude Code, as the name suggests, is a coding tool. Developers use it to build software. What makes Claude Code magical isn't that it's brilliant, but that it uses tools—precisely the way software developers do—to learn and build stuff.
My prompt to Claude Code matches what I provided to ChatGPT, with one important addition (noted in bold, below):
I'd like to do some research for the upcoming Passover Seder.
Looking at around the time of the Civil War, one of the reasons given for the morality of Slavery was that the bible didn't seem to comdem it. In fact, it includes a commandment that the descendents of Ham were to be slaves.
Some suggested that the African slaves were these descendents.
Using ~/bin/safariassist - a tool to search sefaria.org - can you give me some quotes and links to texts from rabbis that discuss how judiasm traditionally understood this commandment that the descenants of ham were to be enslaved.
Would this argument that was common in the US make any sense in a traditional Jewish context?
I've asked Claude Code to use sefariaassist, a Linux command line tool that interacts with sefaria.org's API. Sefaria is a website dedicated to hosting Jewish texts and the links between them.
Unlike ChatGPT, Claude Code is pretty chatty as it tries to tackle the task I've given it. You can read its entire transcript here.
You can see that Claude gets off to a slow start as it tries to figure out what sefariaassist is all about. Incidentally, this is exactly what a human who was tasked to use this program would do: poke around, try it and fail.
Claude pushes forward, figuring out how to use sefariaassist and then starts researching the problem in earnest.
You can see that Claude gets stuff wrong - it tries to use jq on the output of sefariaassist -a text and fails. But again, like a human, it learns and moves on.
Like ChatGPT's complimentary intro, Claude gave me hints that it was on to something. Like when it announced:
● Excellent material. Let me read the full commentary links and also search for Rabbi Raphall's infamous pro-slavery sermon:
or
● I have very rich material. Let me get two more key sources:
Who doesn't love the AI powered dopamine rush that we're on the right track! I say 'we', but all I did was pose the question.
In a minute or two, Claude presented me with findings:
The findings, which I asked Claude to write up can be found here. They essentially match what ChatGPT told me. However, every statement that Claude makes is backed up with a link to Sefaria.
For example, Claude doesn't just report Ibn Ezra's quote, but provides a link to the source material
This link allows me to spot-check Claude Code to ensure it's not hallucinating. Another key win is that it lets me browse the source material to get a more complete picture. Even if ChatGPT got the Ibn Ezra quote right, perhaps earlier or later in the text, Ibn Ezra says something that undermines this quote. Or, maybe there's other useful material here that because it's not as widely cited, ChatGPT hasn't picked up on it as being useful.
In this case, the verse before the one considered famous is also quite fascinating:
Here, Ibn Ezra effortlessly dismantles another pro-slavery argument that circulated at the time of the American Civil War. That is, that the Hebrew עבד עבדים means "slave to slaves." Ibn Ezra shows this as a misreading, and that it simply means "servant like all other servants."
The win here isn't that Claude Code gave me a pithy answer to my question, but that it set me up so I could discover my own answers with ease.
And the winner is...
Comparing ChatGPT's performance to Claude Code's is notable. On one hand, ChatGPT was faster and gave me prettier results. It also provided me a link to a Tablet Magazine article which opens up a number of interesting follow-up questions.
What Claude Code lacks in terms of sexiness, it makes up for with both transparency and completeness. By watching its transcript, I can see its reasoning and what it's searching for. Most importantly, it didn't just give me a useful end product, it gave me direct links to trusted source material. This source material not only provides confidence in the answer provided, but opens up new areas to explore.
In truth, there is no winner here. Both tools are astoundingly good, and no doubt the trust issues I cited with ChatGPT will improve over time. Not that long ago, ChatGPT provided answers with no sources. Now it's providing sources, but some of them are weak. I'm sure ChatGPT and other LLMs will get better at this.
And yet, what Claude Code is doing here is something special: it has traded the know-it-all machine model for one that's a tireless assistant who's an exceedingly quick study. This makes for a helpful partnership and for research tasks like this one, is far more valuable than simply sitting at the feet of a guru.
No comments:
Post a Comment