Might there be a better way?

I wanted to see how good Claude is at diagnosing rare diseases and see if I can make it better using only public data. To do this, I tested Claude on rare-disease cases — building on the RareArena benchmark, but with 371 fresh cases published after Claude’s training data was frozen, so the model couldn’t have memorized them. Here’s what I found:

I tried different ways of setting up Claude, and each tweak gave a statistically significant improvement. However, I found it challenging to guarantee that the model wasn’t cheating on any of the cases.

Why Rare Disease Journeys Are Hard

A “rare” disease affects fewer than 1 in 2,000 people. There are 7,000+ of them. Together, they touch about 6% of the population — most people know someone affected by a rare disease.

Patients wait almost 5 years on average (4.7) from symptom onset to diagnosis (EURORDIS Rare Barometer). They see on average 7-8 doctors. They’re misdiagnosed 2-3 times (Shire Rare Disease Impact Report, 2013). Primary care doctors and specialists rule out the diseases they know about, but don’t consider the diseases they don’t know about. And often patients hop from doctor to doctor until they find an answer (or give up).

In Berlin, I started having symptoms in May 2019. While I was still in Germany, I saw a family doctor, a histamine intolerance expert, a gastroenterologist, and a nutritionist. Back in San Francisco, starting in 2020, I saw another family doctor and two different complex disease specialists.

Only the last specialist looked broadly enough to find an answer (after asking me to draw 40 tubes of blood in one session!)

Benchmarking And Improving Claude

I built on RareArena (Zhao et al., Lancet Digital Health 2026): 8,562 rare-disease screening cases derived from PubMed case reports, with verified diagnoses from the Orphanet rare-disease database.

Each test gives the model a clinical case write-up — symptoms, history, and physical findings, with the diagnosis stripped out — and asks for a ranked list of the top 5 candidate rare diseases. Here’s what one looks like:

An 11-year-old girl presented with retinal findings on widefield optical coherence tomography imaging. Examination revealed increased thickening of the midperipheral retina bilaterally. In areas of retinal thickening, the retinal architecture showed coarse delineation with thinned and disrupted photoreceptor outer segments. The retinal pigment epithelium appeared normal on autofluorescence imaging. Retrospective review of conventional OCT imaging from age 3 demonstrated that the retinal thickening was already present at that earlier timepoint.

— One of the 371 cases (diagnosis hidden; the model has to produce a top‑5 guess)

Trying for a Valid Benchmark

Two challenges make it hard to run a benchmark using public case data: models might find the exact same case while exploring, or they might memorize case details from their training data.

On the first: Claude with a PubMed or web search tool can find the exact same case at runtime. When I gave it free rein, on about half the cases it looked up the actual case while diagnosing it — a.k.a. cheating. To try to avoid this, I implemented custom PubMed and web search tools that filtered out the case Claude was meant to diagnose, and I had tool-use audits in place to check whether filtering worked correctly. The DeepRare paper took a different and stronger approach: they evaluated their system on newly-collected hospital cohorts that were never published online (so there’s nothing for the model to find), plus ablations with web search disabled.

On the second: the model may have memorized some aspect of the PubMed case during its training. To avoid this, I collected 371 rare-disease cases published after Claude Opus 4.6’s August 2025 training cutoff, with diagnosis-framing language stripped out by a separate, smaller Anthropic model (Haiku) so Claude couldn’t reverse-engineer the answer. I can’t fully guarantee these PubMed cases didn’t leave some online trace that was included in the training data, or that the Haiku cleaning pipeline was perfect — but it’s the best I could do with public data.

Results

To assess how good each setup is, I looked at Top-1 = the model’s first guess is the correct diagnosis, and Top-5 = the correct diagnosis appears anywhere in the model’s top 5 guesses.

I tested three different ways of setting up Claude:

1. Vanilla Claude (no tools) — like opening Claude Desktop and asking a question. Got 47.7% right on the first guess, already better than a typical first-visit family doctor.
2. Extended thinking — same as vanilla, with extended thinking turned on. Lifted accuracy to 54.9%.
3. Tool use — plug in two free public medical databases (PubMed for the literature, the Human Phenotype Ontology for clinical signs and symptoms). Pushed accuracy to 76.0% at roughly $0.15 per consult.

Uncertainty is about ±5pp at this sample size, and each step-up is statistically significant under paired same-case tests (p ≤ 0.001). One important caveat on the tools number: on a 25-case rerun under identical conditions, Claude agreed with itself on only 72% of cases — about a quarter of verdicts flip on a fresh run, since each tool-using run takes a slightly different path. The honest read is that tools-with-public-medical-data lands in the mid-to-high 70s on this set of cases, not exactly 76%.

A few reference points for scale:

• General physician first-visit: ~26% Top-1 — as reported in the RareArena paper.
• DeepRare (Zhao et al., Nature 2026): 57.18% Top-1 averaged across HPO-only public benchmarks, 64.4% on a 163-case real-world hospital cohort (physicians at 54.6%). Different cohorts than I tested on, so this is a ballpark reference, not a head-to-head comparison.

Opportunities & Risks

These are encouraging results, but this is an artificial test. Finding candidate diagnoses is only one step in the medical journey — confirming a diagnosis and starting treatment takes much longer.

Over time, though, it seems likely that Claude might be helpful in the following ways:

• Patient advocacy. A patient can go to their primary care doctor (PCP) with well-sourced candidate diagnoses and have a better chance of going to the right specialist on the first try. Or they may be able to plan confirmatory tests with their PCP first, if the tests are affordable or specialists are not easy to access (e.g., in developing countries).
• Doctor assistance. Doctors might be able to support their patients with faster or higher-quality diagnoses. Or the electronic health records system could spot patterns over time and alert the right doctor.

Medicine is complex, with unintended side effects, so there are many ways that the methods above may fail or not help:

• Real medicine is messy. Claude still gets many cases wrong (24% wrong on Top-1, 16% wrong on Top-5), and it will do even worse with real medical records. Medical history is often not as clean or complete as our test cases. Or it might be affected by patient or doctor biases.
• Common diseases are more common. Techniques optimized for surfacing rare diseases might shift focus too much away from common diseases. Common diseases are often easier to diagnose and cheaper to treat. Chasing a rare disease often means expensive testing and months of stress.
• Privacy. Inputs go to Anthropic, and Anthropic is subject to government pressure and data residency rules.

So this tool may be useful, but mainly as a way to help speed up discussions with a doctor.

Conclusion

From this experiment, I learned that anyone can reach around 70% accuracy in diagnosing rare diseases from PubMed cases — in the same order of magnitude as DeepRare’s reported numbers (though on different cohorts, so not a head-to-head comparison), using only Claude and public medical search tools, a much simpler setup than their multi-agent system with more than 40 specialized tools. It was also humbling to learn how hard it is to know whether the benchmarks are real. It took a couple of rounds of running the benchmark and auditing to lock down different ways Claude might have memorized or cheated.

Try a Diagnosis Yourself With Claude Code

If this might be useful to you, please give this a try and let me know how it works out!

Prerequisites

• uv for the HPO MCP server’s Python dependencies
• node & npx for the PubMed MCP server
• Claude Code installed and authenticated

Steps

1. Open Claude Code in a folder for this session. If you have medical records you want Claude to read — PDFs of lab results, appointment notes, imaging reports — put them in a folder and start Claude Code there. If you don’t have any yet, an empty folder works too; you’ll describe symptoms in your own words during the consultation.

mkdir -p ~/records       # or skip if you already have one
cd ~/records
claude

1. Install the plugin. In the Claude Code session, paste this and press enter:

“Please install the rare-disease-consult plugin from https://github.com/fryanpan/rare-disease-consult-plugin. Follow the instructions in its AGENT_INSTALL.md file.”

Claude will show a disclaimer, check that you have uv and node/npx (and offer to install them if missing), clone the plugin, register it, and tell you when to restart.

1. Start the consultation:

/consult

During intake, describe symptoms in your own words. If you put records in the folder, you can also point Claude at them: “Please read the files in this folder and use them in the intake.”

If you’d rather install the plugin manually, see the plugin README’s Installation section for the clone + register flow.

A few notes: the first invocation downloads HPO data (~50 MB, ~30 seconds, one-time). Currently Claude Code only — the plugin uses local helper tools (uv and node/npx) that aren’t yet available in Claude Cowork’s hosted runtime. Let me know if you want this to work in Cowork and I can take a look.

Links

• Benchmark + methodology: github.com/fryanpan/rare_disease_benchmark (CC BY-NC-SA 4.0) — full audit and reproduction notes.
• Read the caveats first: the plugin README covers the full risk list. This is v0.1 — it has not been validated on real patient cases.

Asked my team lead agent for a hit list of what needed my attention. Reviewed a draft of this blog post and sent some edit requests. Checked an interactive mockup, left some comments. Looked over some training examples for one of my projects and kicked off the next round. Five minutes later, I was done with my banana bread, and my agents had what they needed for the next chunk of work.

In the past two months in Berlin, I’ve made exploring a priority. But I’ve also wanted to keep my side projects rolling, so I’ve been tuning my agent setup to let me review when my agents need input, and I have time to spare.

What I’ve done with this team in the last six weeks

Here’s what this workflow has helped me do since early April, with around 15 hours a week of hands-on time:

• Started getting back in touch with old friends and colleagues. It really has been a joy!
• Tested how well Claude can diagnose rare diseases (follow-up to RareArena)
• Made my family bike map usable day-to-day in Berlin
• Honed my approach for how I spend my time each week
• Worked with our H&R Block rep to do our taxes
• Set up a personal site and writing workflow. I’ll try to post at least once a week!
• Built a way to measure agent-team effectiveness and continuously improve
• Planned many activities around Berlin with my boo on Discord
• Started helping friends out at a nonprofit with Android development
• And…built this agent workflow to meet my needs!

I wanted to share this post as background context. After this, I promise no more agent infrastructure posts for the next two months. I think it’s more interesting to talk about what I try to do with my agents, rather than how the agents are set up.

If you really want to know about my evolving agent workflow, follow along in my open-source repos linked throughout this post…

The agent team

I run a team of long-running Claude Code sessions on a Mac mini at home — one per project I care about, plus a team lead that coordinates them. Claude Code is Anthropic’s coding agent; you can run it on your own machine.

Keeping each session running around the clock means each agent remembers what we’ve been discussing, plus longer-term notes I keep with them in markdown files or Notion docs.

The team today:

• A writing assistant for blog posts and my personal site. Also gives guidance to other agents when they need help writing.
• A personal assistant — an assistant that Joanna and I both have a Discord chat with. Helps us research things like what to do in Berlin, where to go for Bea’s after-school outings, or how to fix a flat tire on my Helix folding bike.
• A personal CRM agent that helps me reconnect with people from the past — I’m finally trying to get better about that.
• A finance agent that helps with taxes, donations, and financial planning.
• A research team that runs on-demand investigations and scheduled monitoring of high-signal sources I care about.
• Half a dozen coding agents for various side projects

flowchart TD
  subgraph User["Phone or laptop"]
    UApp["Claude app or browser"]
  end

  RC["Remote Control"]
  TS["Tailscale"]
  Events["Discord, GitHub, Notion events"]

  subgraph Mac["Mac mini at home"]
    TL["Team lead"]
    PA["Project agents"]
    LF["Live Feedback server"]
    TL <-->|"claude-hive"| PA
    PA -->|"starts review sessions"| LF
  end

  UApp -->|"chat"| RC
  RC --> TL
  RC --> PA
  UApp -->|"review work"| TS
  TS --> LF
  Events -->|"event bridges"| PA

  style User fill:#fef3c7,stroke:#999
  style Mac fill:#e0f2fe,stroke:#999

The weekly rhythm

Each week, the team lead and I plan major goals. The plan goes to Notion. The team lead sends it to the team to vet; I leave comments in Notion; the plan adjusts from there.

When the plan looks good, the team lead kicks off the work.

And then throughout the week, whenever I have time to help, I ask the team lead what needs my attention, and in what order.

The tools that let me work on the go

I don’t sit at the Mac mini anymore. I tap into it from wherever I am. Here are the pieces that make it work:

Remote Control — chat with my agents on the go

Claude Code sessions run around the clock on the Mac mini. To connect from a laptop or phone via the official Claude app, I’ve set Remote Control to on for all sessions in my Claude Code configuration.

The team lead coordinates the fleet

One of those Claude Code sessions is the team lead. It works with me each week to set shared goals across the whole agent team. And then the team lead keeps everyone moving on tasks, troubleshoots issues, and relays context between teammates. That makes it easier for me to work on the go since I can write a brief request to the team lead, and it handles the rest.

The team lead also helps the team reflect and improve using months of Claude Code transcripts.

The closest existing tool is Claude Cowork, Anthropic’s agentic tool for non-technical users. My team lead running in Claude Code is more customizable.

Open source team lead: fryanpan/ai-team-lead

Under the hood, the team lead and agents on the team use claude-hive to message each other.

Live Feedback — review docs, mockups, and dev servers from any device

Before this plugin, reviewing what Claude sent me was tedious. Claude would send me markdown docs or interactive mockups during planning, or dev server links while implementing and testing. I’d have to figure out how to access these links remotely, and then jump between the link and chatting with Claude to give feedback.

My live feedback plugin collapses that loop. Now I can comment directly on the doc, running app, or mockup; the agent on the other side makes updates, and they show up immediately while I’m still reviewing.

Open source feedback plugin: fryanpan/claude-live-feedback-plugin

The plugin uses Claude Channels (Anthropic’s real-time notification feature), released in March 2026, for pushing notifications into a running Claude session.

Tailscale — the private network underneath

Tailscale is a private mesh network between my devices. My phone, laptop, and the Mac mini all show up on the same network without a public tunnel between them. Live Feedback and any dev servers I spin up on the Mac mini are reachable from anywhere I am. No port-forwarding, no public exposure, no surprise scanners.

Responding to Other Events: Discord, GitHub

Three other pieces let the team pick up events and kick off work on their own, without me having to relay a prompt.

• A Discord plugin lets Joanna and me chat with the personal assistant agent from anywhere.
• A GitHub event bridge lets coding agents take a feature all the way from code review through addressing reviewer comments to merging and deploying — similar to Claude’s Automerge feature but covering more steps.
• A Notion bridge lets the agents listen to pages or whole sections in Notion and react to comments and edits on those pages.

What this setup gives me

This setup lets me work from anywhere and give input in batches — no more constantly juggling open chats.

It does not, however, make me any more productive. I’ve measured my team’s output using historical transcripts. For each hour of hands-on effort I put in, my agents produce about the same as they would if I only worked from my desk — the setup doesn’t change the per-hour ratio, it just lets me work on the move.

What’s still hard

Multi-agent setups introduce flakiness. This is more stable than my March setup (custom-built agents running on cloud servers, chatted to through Slack, before Remote Control and Dispatch existed), but it still pulls 1–2 hours of orchestration and maintenance time out of my week. The hive messaging went down a few days ago due to a bad configuration change; for a few hours, all my agents were sending each other messages with no deliveries, which led to mass confusion until I asked the team lead what was up and went to fix it. Plus, several Claude components I depend on, like Remote Control or Channels, are flaky too and seem to fail once a week or so. Over time, I’ll make these pieces more stable, or hopefully, can depend on 3rd parties to build more robust versions!

Managing the agents’ memory is an ongoing challenge. With sessions running for days, the conversation history piles up. That history costs money, plus it isn’t organized in a way that will work as I dump more knowledge in. When I feel this is painful enough, I’ll bring in more complicated memory structures — everyone seems to be working on these.

What’s next

I’m happy so far with how smoothly this setup lets me work on the go. In the coming weeks, I’ll keep reviewing with the team to see what’s worth improving next.

A few areas already on my list:

• Making as-good-or-better decisions with less hands-on time from me
• Organizing my work so I stay in flow and focus on one area at a time
• Measuring and honing how the team delivers value (this will be hard!)
• Fixing infrastructure reliability or usability

I’ll start writing more about what I’ve been doing with my agents — rare disease diagnosis is probably up next.

I’ll continue to post to Octoturtle blog for things related to the family, but this blog will have more personal or technical topics for friends, family, and anyone interested in following along.

Plus for anyone who might want to work with me, there’s an “About Me” page in human-readable format.