Last week TypeSafe released Jev, an AI model that can't write. You give it some text and a list of questions, and it answers with things like "yes, 0.96" or "option B." It won't summarize the text or explain itself, and an answer usually comes back in about a sixth of a second. TypeSafe calls it a System One model, after the fast, gut-reaction half of Kahneman's two kinds of thinking.

Their launch demos have it playing Doom and racing through Wikipedia links. We had a duller job in mind. Every rule on Court Rules comes with the judge's own words attached, so that a reader can check our summary against the sentence it came from. A model that does nothing except compare a claim to a piece of text is pretty much a proofreader, and we had about 64,000 rules that had never had a second pair of eyes on them all at once.

Twenty minutes and $1.34

Each rule we publish starts with the judge's sentence, quoted word for word. Under that is our plain-English summary, and under that a handful of structured fields (a page limit, say, or a number of days). The quote gets matched against the source document before a rule goes live. The summary and the fields are written by an extraction model, and until now the only way to double-check them was to pay a second large model to reread everything. That is slow and expensive enough that we had only ever done it in samples.

Jev read all of it in twenty minutes, and the bill was $1.34. For each rule we asked it two things: does the summary say what the quote says, and is each structured field something the quote states?

What it caught

About 95 percent of rules came back clean. In the rest, the quote and the citation were fine, and the trouble was a summary or a field that said slightly more than the judge did. One judge wrote that a party "may submit" two courtesy copies of a compendium of authorities, and our summary had turned that into a requirement. Another judge encourages junior attorneys to argue motions, and our structured data had attached a years-of-experience cutoff the judge never gave. A rule about tabbing exhibit binders had picked up page margins from somewhere.

A litigator mostly reads a rule to find out whether something is required or just allowed, so we took these seriously. We went through a sample by hand, agreed with Jev, and fixed the flagged rules the same day.

We also changed the form our extraction model fills in. It used to have required blanks for things like margins and experience cutoffs, and a model handed a required blank will find something to put in it. Those fields are optional now, and an empty one means the judge didn't say.

Where it was bad

Our first multiple-choice question was a flop, and it was our fault. We gave Jev a menu of eleven document types and asked which one a page was, and it decided that nearly every blank template order was an order entered in one particular named case. We blamed the model and switched to separate yes-or-no questions, which worked.

Then Jerry Liu published DocJev, which classifies documents with Jev by defining each category in one sentence about the document's main purpose. Our eleven options had been bare labels with nothing telling them apart. We rewrote the menu that way and ran it on a batch of documents, five of which we had already read by hand. It agreed with our reading on four. The miss was a links page that talks about filing rules without stating any, which a second yes-or-no question now catches. Parallel found that very long label lists still hurt it, so we keep the menus short.

Its confidence scores only mean something at the ends. When it was 95 percent sure a summary contradicted its quote, it was usually right. At 60 percent it was right about one time in four, and once it insisted that "at least 21 days before" and "no later than 21 days before" were in conflict. Another time it pulled the correct number out of a 90-minute time limit and then told us the unit was hours.

And it has no ideas of its own. It will count any category you hand it across every rule we have, for pennies, but we had to bring the categories.

Questions we couldn't afford before

The cheapness changes what you bother to ask. We would never have paid a big model to find out how judges phrase deadlines, but at these prices we just ran the question across the nearly nine thousand rules that have one. More than a third just say "days" and leave you to work out which kind. About one in fourteen say "court days," which skip court holidays and are not the same thing as business days. We are reworking our deadline fields around what that survey found.

We also ran it over about four thousand source documents with a single question: does this page tell a litigant to do something? That took 70 seconds. It turned up several hundred pages we had set aside, most of them the blank template orders judges post for parties to fill in, which do carry requirements if you read them. A second model that had to quote its evidence agreed on 39 of the 40 we pulled at random. Those documents are in the extraction queue.

What it does now

Jev now gets the first say on whether a document we collect contains rules at all. The large model only weighs in when Jev is unsure. It also has the proofreading job for good. Every new rule's summary and values are checked against the judge's sentence before the rule shows up on the page you search, and a rule it can't stand behind gets flagged for us to look at. We will write again once that has been running for a while and we can say how much it catches.

Originally published on Court Rules.

The shape of a row

A row is identified by the pair (court_slug, url). The judge name alone does not identify a row.

The audit

// Check a judge index JSON file for rows that cannot be identified.
//
// Usage: node row-identity.js [judges.json]
//
// Reports rows with an empty court_slug or url, and duplicate (court_slug, url)
// pairs. Rows are never collapsed by judge_name, so two judges who share a name
// stay separate. Standard library only.
const fs = require("fs");

const path = process.argv[2] || "sample-judges.json";
const rows = JSON.parse(fs.readFileSync(path, "utf8"));

const missingCourt = [];
const missingUrl = [];
const seen = new Map();
const duplicates = [];

rows.forEach((row) => {
  const court = (row.court_slug || "").trim();
  const url = (row.url || "").trim();
  const label = (row.judge_name || "(no name)") + " [" + (court || "no court") + "]";
  if (court === "") missingCourt.push(label);
  if (url === "") missingUrl.push(label);
  if (court !== "" && url !== "") {
    const key = court + "\u0000" + url;
    if (seen.has(key)) duplicates.push(seen.get(key) + " and " + label + " -> " + court + " " + url);
    else seen.set(key, label);
  }
});

console.log("rows read = " + rows.length);
console.log("missing court_slug = " + missingCourt.length);
missingCourt.forEach((item) => console.log("  " + item));
console.log("missing url = " + missingUrl.length);
missingUrl.forEach((item) => console.log("  " + item));
console.log("duplicate (court_slug, url) pairs = " + duplicates.length);
duplicates.forEach((item) => console.log("  " + item));

const names = new Map();
rows.forEach((row) => {
  const name = (row.judge_name || "").trim();
  if (!names.has(name)) names.set(name, []);
  names.get(name).push((row.court_slug || "").trim());
});
const repeated = [...names.entries()].filter(([, courts]) => courts.length > 1);
console.log("repeated judge names kept separate = " + repeated.length);
repeated.forEach(([name, courts]) => console.log("  " + name + " -> " + courts.join(", ")));

process.exitCode = missingCourt.length || missingUrl.length || duplicates.length ? 1 : 0;

Run it

node row-identity.js judges.json

The fixture output shows each check firing once.

rows read = 5
missing court_slug = 1
  Example Judge Missing Court [no court]
missing url = 1
  Example Judge Missing Url [ex-district]
duplicate (court_slug, url) pairs = 1
  Example Judge With Date [ex-district] and Example Duplicated Row [ex-district] -> ex-district https://example.invalid/courts/ex-district/example-judge-with-date
repeated judge names kept separate = 1
  Example Duplicated Row -> ex-district, ex-circuit

The fixture is synthetic and sits beside the script. One row has no court, one has no URL, two rows share a court-and-URL pair, and one judge name appears in two courts. The last line is the point: the two Example Duplicated Row entries stay two rows because their courts differ.

Against the real snapshot the audit reports no missing fields and no duplicate pairs, and five repeated judge names kept separate, including Barbara J. Rothstein in ddc and wdwa.

Why judge_name is not the key

Names repeat across courts and change spelling over time. Collapsing rows by judge_name would merge two real pages into one and hide a duplicate court-and-URL pair at the same time. The pair check fails loudly when the same page is listed twice.

What the audit cannot show

• Duplicate pairs mean one page is listed twice.
• A null rules_last_changed stays unknown. The script prints nothing about it by design.
• Row identity says nothing about whether a court requirement applies to a case.
• The audit checks the file only. The court documents behind the pages are out of scope.

Source and license

Every row keeps its page URL. The Michelle Lim's source page traces to the court's own document listing at lacourt.ca.gov, and each row in the file carries the same kind of pointer. CC BY 4.0 covers judges.csv, judges.json and the dataset README only. Credit Court Rules and preserve every row's source URL.

We started with a simple problem. My co-founder is a litigator. I am an engineer. He was spending hours every week digging through PDFs on federal court websites to find judge-specific rules, local filing procedures, and deadline information. The PDFs were inconsistently formatted. Searchable text sat next to scanned images from 1995, some updated quarterly, some untouched for five years.

He asked me if I could build something to aggregate it all. I said sure. How hard could it be?

Famous last words.

The scope problem

There are 94 U.S. federal district courts. Each has local rules. Many of those local rules span hundreds of pages. Then there are the individual judge practices, chamber-specific documents that live on each judge's personal page on the court website. Not every judge publishes them, and courts organize them differently; some bury the files under four layers of navigation.

Our first attempt at scope was naive. We thought we would scrape the rules, parse the PDFs, and build a searchable index. We did not realize that "scrape the rules" meant writing 94 different scrapers, each handling a different CMS, different URL structure, different PDF format, and different update cadence.

The Southern District of New York alone has over two dozen active district judges. Each has an individual practices PDF. Some update them every few months. The URL changes when they update. The filename changes. Sometimes the PDF is replaced inline with no versioning. Tracking changes across that surface area is a full-time job.

PDF parsing is the real enemy

PDF parsing is a solved problem in computer science the same way sailing is a solved problem. The theory is straightforward. The practice involves a lot of cursing and unexpected water.

We started with Python and PyPDF2. It worked on maybe 30 percent of documents. The rest were image-based PDFs with no text layer. For those we added OCR using Tesseract. That got us to about 60 percent. But OCR on legal documents is messy. Footnotes confuse it. Tables break it. Multi-column layouts destroy it.

Then we found a whole category of PDFs that were text-based but had broken encoding. The text extracted as gibberish while the visual render was fine, because the underlying text was corrupted. These were usually documents that had been printed, scanned, OCR'd, and re-saved multiple times over a decade. Each step introduced more noise.

Our current pipeline uses a hybrid approach. Text-based PDFs go through pdfplumber, which handles tables and layout better than PyPDF2. Image-based PDFs go through Tesseract with layout analysis. When the encoding is broken, we fall back to full-page OCR of the rendered image, slower but more reliable than trusting the corrupt text layer.

Even with all of that, we still manually review a percentage of documents. Some things are just faster for a human to verify.

Normalizing judge data

Court websites do not have APIs. They have HTML. Sometimes very old HTML.

We built a judge directory covering every active federal district judge. For each judge, we track name, court, appointment date, chamber location, and links to individual practices documents. This sounds simple. It is not.

Some courts list judges on a single page with clean semantic HTML. Others use a table layout from 2003. Others require clicking through to a bio page, then finding a "chambers" tab, then downloading a PDF from a link that has no predictable structure.

We ended up with a per-court configuration system. Each court gets a scraper definition file that specifies:

• The URL pattern for the judges list page
• The CSS selector or XPath to extract judge names and profile links
• The pattern for individual practices document URLs
• The expected update frequency
• A human reviewer assignment

This lets us add new courts without rewriting the entire system. It also means we have 94 configuration files, each slightly different, each requiring maintenance when the court redesigns its website.

Which happens more often than you would think.

The deadline calculator

The most technically interesting part of the project is the deadline calculator. Federal court deadlines depend on:

• The triggering event (service, order entry, filing)
• The rule that governs the period (Rule 12, Rule 56, Rule 59, etc.)
• The type of days (calendar days, business days, court days)
• Federal holidays
• Local court closures
• Whether the last day falls on a weekend or holiday

We built a rules engine that encodes the Federal Rules of Civil Procedure and the Federal Rules of Appellate Procedure. The engine takes a trigger date, a rule reference, and a court identifier. It returns the calculated deadline with an explanation of each step.

The tricky part is local variations. Districts modify the default federal periods through local rules, individual judges add scheduling requirements through standing orders, and courts close for local events that are not federal holidays. We maintain a per-court overrides file that layers local rules on top of the federal defaults.

For holidays, we pull the OPM federal holiday calendar. But we also track observed holidays separately from calendar holidays. When New Year's Day falls on a Saturday, the observed holiday is Friday. When it falls on Sunday, it is Monday. The calculator handles this correctly, which sounds trivial until you realize how many generic date libraries get it wrong.

Inauguration Day is its own special case. It is a federal holiday only in the Washington D.C. area. Our calculator knows that and applies it only to D.D.C. and D.C. Circuit deadlines. Everywhere else, January 20 is a normal day.

Architecture

The stack is intentionally boring. We use Node.js for the API, PostgreSQL for structured data, and Elasticsearch for full-text search across rules documents. The frontend is React. Nothing exotic.

Documents are stored in S3. Parsed text is indexed in Elasticsearch with metadata tags for court, judge, document type, and effective date. We version every document. When a judge updates their individual practices, we keep the old version and mark the new one as current. This lets us show change history and answer questions like "what did Judge Liman's practices say in 2023?"

The scrapers run on a schedule. Most courts are checked weekly. High-update courts, like SDNY, are checked daily. The scraper jobs are idempotent. If nothing changed, nothing changes. If a PDF was updated, it gets downloaded, parsed, indexed, and queued for human review if the diff is significant.

We use GitHub Actions for CI and deployment. The whole thing runs on AWS. Total monthly infrastructure cost is under $200. The biggest expense is the human review time, not the servers.

What we got wrong

Our first version tried to auto-detect document type from PDF content. That failed badly. A document titled "Individual Practices" might actually be a standing order. A "Local Rules" PDF might contain only criminal rules. We now require manual classification for new document types. It is slower. It is also correct.

We also underestimated the maintenance burden. Court websites change. Judges retire. New judges get appointed. PDFs get reorganized. We built a system that is 80 percent automated and 20 percent human curation. I thought we could get to 95 percent. I was wrong. The long tail of edge cases is endless.

Another mistake: we initially stored parsed text as plain strings. When we added full-text search, we realized we had destroyed paragraph structure, footnote references, and table data. We had to re-parse every document with layout preservation. That took three weekends.

Current numbers

The database covers all 94 federal district courts. We track approximately 680 active district judges. We have parsed and indexed over 4,200 local rules and individual practices documents. The deadline calculator handles 40+ federal rule references with district-specific overrides.

Traffic is modest but growing. About 12,000 unique visitors per month, mostly litigators, paralegals, and law students. The calculator is the most-used feature. The judge directory gets heavy use from associates who just got case assignments and need to know their judge's quirks.

We have had zero venture capital. No paid marketing. Growth is entirely organic through legal forums, Reddit, and word of mouth in litigation departments.

Why we keep doing it

The honest answer is that we use it ourselves. My co-founder relies on it for his practice. I rely on it as a sanity check when he asks me whether a deadline calculation looks right. Building tools you actually use changes your priorities. You do not add features because they sound impressive. You add them because you needed them yesterday.

The federal court system is public, the rules are public, and the judge practices are public. But they are scattered across 94 websites, in formats that resist aggregation. We are just trying to put them in one place, in a form that is actually usable.

It is not the kind of project that gets written up in TechCrunch. It will not scale to a billion users. But it solves a real problem for a specific group of people. And for a side project, that is enough.

If you are interested in the technical details or want to contribute, the project is at courtrules.app. We are not open source yet, though we have talked about it. The dataset might be useful to researchers, journalists, or other developers working in the legal space.