From Permit Rows to a Focused Publication

A dataset is a source, not an article

San Francisco publishes building permit applications through the Building Permits dataset. The source is useful because it preserves operational fields and identifiers. It is difficult to read as a record of meaningful construction activity.

A permit can appear in multiple rows. The scope can be terse, inconsistent, or spread across related filings. Valuation does not always mean the same thing across permit types. Addresses, applicants, status, and dates need to remain connected to the source record.

SF Building Permits turns selected filings into focused reports. That requires a pipeline with separate collection, editorial, record, and rendering responsibilities.

Preserve source state first

The workflow collects records from San Francisco OpenData through its Socrata interface. Source identifiers and URLs stay attached to the local record so a report can be checked against the municipal source.

SQLite stores the working state. That database separates three questions:

1. Has the source record been seen?
2. Has it been evaluated for editorial significance?
3. Has a report been produced and published?

Keeping those states outside generated HTML means the site can be rebuilt without forgetting what the workflow has already processed. It also makes duplicate handling and correction work possible.

Select before writing

The publication does not turn every permit row into a page. It looks for major filings and small related clusters that justify a focused report.

The editorial record captures the fields a reader needs to understand the filing:

• Location and neighborhood context
• Permit scope
• Declared valuation when present
• Applicant or responsible party when present
• Filing and status information
• Source identifiers and a link back to San Francisco OpenData

One report covers one notable permit or a tightly related cluster. This keeps the article connected to a concrete event instead of producing bulk summaries that hide the underlying record.

Render static output

Jinja2 turns approved records into static pages. A Cloudflare Worker serves the generated output.

The static delivery model keeps the public system small. It has no database connection, source API credential, or publishing interface exposed to visitors. Collection and drafting happen before deployment, while the public site serves completed reports and source links.

The system uses Python 3.12 managed with uv, SQLite for records, Jinja2 for templates, and a static output directory for deployment. As of July 2026, twenty-six reports are public.

The human and agent boundary

The original workflow stopped before drafting and waited for a human selection. That was an editorial policy rather than a technical limitation. When no agent session ran, the publication stopped producing even though the source and code remained healthy.

This distinction matters because a dormant publication and a broken automated pipeline look similar from the outside. SF Building Permits had no scheduler, cron trigger, or continuous publishing service to repair. It had a written procedure that needed an active operator.

The system can move toward more automated drafting and publishing, but the source, scoring, and records boundaries remain necessary. Automation changes who advances a record through the workflow. It does not remove the need for provenance, duplicate control, editorial criteria, and correction paths.

The reusable pattern

A focused publication over public operational data needs more than a feed reader:

• Preserve source identifiers and links.
• Separate collected, evaluated, drafted, and published state.
• Group related source rows before writing.
• Define why a record deserves a report.
• Keep generated public output separate from the editorial database.
• Make corrections traceable to the original source.

The result is a publication whose claims can be followed back to municipal records and whose build can be repeated without scraping its own pages for memory.

Explore the SF Building Permits case study or open the live publication.