Podesta Email Corpus Analysis

Methodology: the full wikileaks-full.mbox parsed with Python stdlib mailbox / email in a single pass, normalized to JSONL, attachments content-addressed to disk. All counts below come from the parsed JSONL, not the raw mbox. Pass generated on 2026-04-13.

1. Corpus overviewlink

2. Provenance & source reconciliationlink

The kanban card originally called for a three-way overlap matrix across three WikiLeaks mirror snapshots (wikileaks-full, archive-small, archive-zip). Phase-1 exploration showed this work would produce no signal:

• archive-small is a truncated byte-prefix of wikileaks-full. Every message in archive-small is byte-identical to the corresponding prefix of the canonical mbox. No reconciliation needed.
• archive-zip is an HTML export, not a mail store. The HTML pages carry no Message-ID headers, and messages larger than ~10 KB have their bodies omitted from the export. There is no usable join key with the mbox corpus.

This report therefore treats wikileaks-full (50,887 messages, 4.67 GB on disk) as the canonical source and ignores the other two snapshots. The decision was approved before parsing began.

3. Message-ID healthlink

Synthetic IDs are deterministic: sha256("{from}|{date}|{subject}|{body[:200]}")[:16] + "@synthetic.pmail". Replaying the parse on identical input produces identical synthetic IDs, so the DB loader (#621) can treat synthetic IDs as stable primary keys.

Duplicate-ID handling in #621: first occurrence wins; subsequent occurrences are either dropped or stored under a suffix depending on body hash equality.

4. Date distributionlink

By yearlink

By monthlink

5. Top senders (top 50)link

Corpus contains 3,844 unique sender addresses.

6. MIME structurelink

Top-level Content-Type of each message (what the outer envelope advertises):

Content-Transfer-Encoding distribution across all MIME parts (not just top-level):

7. Attachmentslink

Totals: 14,119 attachment instances, content-dedup'd to 10,276 unique blobs totaling 2.7 GB on disk.

By content type (top 20)link

By filename extension (top 20)link

Largest attachments (top 15)link

8. Threadinglink

Threading derivation (root detection, union-find, canonical thread IDs) is deferred to #621. This report only profiles the raw coverage so the loader knows how many messages will need synthesized thread roots.

9. Character encodingslink

Non-UTF-8 charsets are decoded with errors='replace' during parse. The HTML sanitizer in #621 will re-normalize to UTF-8 on write.

10. Anomalieslink

11. Readiness for ingestlink

Parser-emitted warnings the #621 DB loader must handle:

The full JSONL (ingest/pmail/staged/wikileaks-full.jsonl) is the input to #621 — the loader reads one line at a time, hits the attachment blobs by sha256, and inserts into pmail.db per the schema already shipped in 9c66910.