---
title: "Podesta Email Corpus Analysis"
subtitle: "Parse, dedup, and profile of the 50,887-message WikiLeaks release"
description: "Full shape of the Podesta mbox before ingest: message counts, date distribution, MIME mix, sender graph, attachments, threading health."
color: "#2d3f54"
image: /og/report-pmail-corpus-analysis.jpg
sortDate: 2016-11-06
stats: {"messages": 50887, "attachments": 14119, "dateRange": "2000-06-29 — 2016-03-21"}
verdict: "Corpus is ingest-ready: 0.00% of messages need synthetic IDs, 0.08% duplicate Message-IDs, 426 DSN/bounce messages to flag, 14,119 attachments dedupe to 10,276 unique blobs (2.7 GB)."
---

# Podesta Email Corpus Analysis

*Methodology: the full `wikileaks-full.mbox` parsed with Python stdlib `mailbox` / `email` in a single pass, normalized to JSONL, attachments content-addressed to disk. All counts below come from the parsed JSONL, not the raw mbox. Pass generated on 2026-04-13.*

---

## 1. Corpus overview

| | |
|---|---|
| Messages parsed | **50,887** |
| Date range | **2000-06-29 — 2016-03-21** |
| Unique senders | **3,844** |
| Attachments (total instances) | **14,119** |
| Attachments (unique content-hashed blobs) | **10,276** |
| Attachment storage (dedup'd) | **2.7 GB** |
| Messages with In-Reply-To | **20,606** (40.5%) |
| Messages with References | **21,036** (41.3%) |
| Synthetic Message-IDs | **1** (0.00%) |
| Duplicate Message-IDs | **42** (0.08%) |
| DSNs / bounces | **426** |
| Malformed Date headers | **0** |

---

## 2. Provenance & source reconciliation

The kanban card originally called for a three-way overlap matrix across three WikiLeaks mirror snapshots (`wikileaks-full`, `archive-small`, `archive-zip`). Phase-1 exploration showed this work would produce no signal:

- **`archive-small`** is a truncated byte-prefix of `wikileaks-full`. Every message in `archive-small` is byte-identical to the corresponding prefix of the canonical mbox. No reconciliation needed.
- **`archive-zip`** is an HTML export, not a mail store. The HTML pages carry no Message-ID headers, and messages larger than ~10 KB have their bodies omitted from the export. There is no usable join key with the mbox corpus.

This report therefore treats `wikileaks-full` (50,887 messages, 4.67 GB on disk) as the canonical source and ignores the other two snapshots. The decision was approved before parsing began.

---

## 3. Message-ID health

| Condition | Count | % of corpus |
|-----------|-------|-------------|
| Message-ID header present | 50,886 | 100.00% |
| Synthetic Message-ID generated | 1 | 0.00% |
| Duplicate Message-ID (first-wins kept) | 42 | 0.08% |

Synthetic IDs are deterministic: `sha256("{from}|{date}|{subject}|{body[:200]}")[:16] + "@synthetic.pmail"`. Replaying the parse on identical input produces identical synthetic IDs, so the DB loader (#621) can treat synthetic IDs as stable primary keys.

Duplicate-ID handling in #621: first occurrence wins; subsequent occurrences are either dropped or stored under a suffix depending on body hash equality.

---

## 4. Date distribution

### By year

| Year | Messages |
|------|----------|
| 2000 | 1 |
| 2004 | 1 |
| 2005 | 1 |
| 2006 | 7 |
| 2007 | 350 |
| 2008 | 6,007 |
| 2009 | 1,147 |
| 2010 | 1,171 |
| 2011 | 1,027 |
| 2012 | 1,288 |
| 2013 | 1,286 |
| 2014 | 7,263 |
| 2015 | 24,672 |
| 2016 | 6,665 |

### By month

| Month | Messages |
|-------|----------|
| 2000-06 | 1 |
| 2004-06 | 1 |
| 2005-06 | 1 |
| 2006-02 | 2 |
| 2006-10 | 4 |
| 2006-11 | 1 |
| 2007-05 | 1 |
| 2007-06 | 8 |
| 2007-07 | 1 |
| 2007-09 | 50 |
| 2007-10 | 19 |
| 2007-11 | 93 |
| 2007-12 | 178 |
| 2008-01 | 243 |
| 2008-02 | 316 |
| 2008-03 | 117 |
| 2008-04 | 158 |
| 2008-05 | 296 |
| 2008-06 | 455 |
| 2008-07 | 448 |
| 2008-08 | 494 |
| 2008-09 | 546 |
| 2008-10 | 1,258 |
| 2008-11 | 1,281 |
| 2008-12 | 395 |
| 2009-01 | 245 |
| 2009-02 | 5 |
| 2009-03 | 9 |
| 2009-05 | 112 |
| 2009-06 | 120 |
| 2009-07 | 99 |
| 2009-08 | 42 |
| 2009-09 | 138 |
| 2009-10 | 141 |
| 2009-11 | 134 |
| 2009-12 | 102 |
| 2010-01 | 126 |
| 2010-02 | 111 |
| 2010-03 | 124 |
| 2010-04 | 75 |
| 2010-05 | 82 |
| 2010-06 | 90 |
| 2010-07 | 57 |
| 2010-08 | 90 |
| 2010-09 | 108 |
| 2010-10 | 180 |
| 2010-11 | 71 |
| 2010-12 | 57 |
| 2011-01 | 66 |
| 2011-02 | 57 |
| 2011-03 | 74 |
| 2011-04 | 65 |
| 2011-05 | 73 |
| 2011-06 | 57 |
| 2011-07 | 51 |
| 2011-08 | 36 |
| 2011-09 | 19 |
| 2011-10 | 18 |
| 2011-11 | 275 |
| 2011-12 | 236 |
| 2012-01 | 165 |
| 2012-02 | 62 |
| 2012-03 | 46 |
| 2012-04 | 43 |
| 2012-05 | 28 |
| 2012-06 | 17 |
| 2012-07 | 94 |
| 2012-08 | 181 |
| 2012-09 | 198 |
| 2012-10 | 209 |
| 2012-11 | 121 |
| 2012-12 | 124 |
| 2013-01 | 113 |
| 2013-02 | 115 |
| 2013-03 | 128 |
| 2013-04 | 109 |
| 2013-05 | 128 |
| 2013-06 | 118 |
| 2013-07 | 124 |
| 2013-08 | 119 |
| 2013-09 | 128 |
| 2013-10 | 70 |
| 2013-11 | 39 |
| 2013-12 | 95 |
| 2014-01 | 391 |
| 2014-02 | 381 |
| 2014-03 | 350 |
| 2014-04 | 405 |
| 2014-05 | 386 |
| 2014-06 | 338 |
| 2014-07 | 535 |
| 2014-08 | 910 |
| 2014-09 | 865 |
| 2014-10 | 1,010 |
| 2014-11 | 827 |
| 2014-12 | 865 |
| 2015-01 | 1,041 |
| 2015-02 | 1,500 |
| 2015-03 | 2,163 |
| 2015-04 | 2,296 |
| 2015-05 | 1,979 |
| 2015-06 | 2,165 |
| 2015-07 | 2,336 |
| 2015-08 | 2,127 |
| 2015-09 | 2,271 |
| 2015-10 | 2,484 |
| 2015-11 | 2,045 |
| 2015-12 | 2,265 |
| 2016-01 | 2,549 |
| 2016-02 | 2,432 |
| 2016-03 | 1,684 |

---

## 5. Top senders (top 50)

| # | Sender | Messages |
|---|--------|----------|
| 1 | `john.podesta@gmail.com` | 6,553 |
| 2 | `mfisher@hillaryclinton.com` | 1,329 |
| 3 | `cheryl.mills@gmail.com` | 735 |
| 4 | `re47@hillaryclinton.com` | 705 |
| 5 | `info@barackobama.com` | 627 |
| 6 | `slatham@hillaryclinton.com` | 534 |
| 7 | `eryn.sepp@gmail.com` | 517 |
| 8 | `jpalmieri@hillaryclinton.com` | 513 |
| 9 | `hms@sandlerfoundation.org` | 438 |
| 10 | `burns.strider@americanbridge.org` | 419 |
| 11 | `podesta@podesta.com` | 417 |
| 12 | `dschwerin@hillaryclinton.com` | 408 |
| 13 | `kofferdahl@hillaryclinton.com` | 382 |
| 14 | `funk@americansunitedforchange.org` | 329 |
| 15 | `ntanden@gmail.com` | 326 |
| 16 | `robbymook2015@gmail.com` | 325 |
| 17 | `ha16@hillaryclinton.com` | 298 |
| 18 | `sara.latham@ptt.gov` | 292 |
| 19 | `feedback@lcv.org` | 279 |
| 20 | `admin@endcitizensunited.org` | 278 |
| 21 | `jsullivan@hillaryclinton.com` | 277 |
| 22 | `podesta.mary@gmail.com` | 271 |
| 23 | `robbymook@gmail.com` | 261 |
| 24 | `jbenenson@bsgco.com` | 255 |
| 25 | `nbudzinski@hillaryclinton.com` | 244 |
| 26 | `messenger@truthout.org` | 236 |
| 27 | `jennifer.m.palmieri@gmail.com` | 233 |
| 28 | `gpodesta@gmail.com` | 223 |
| 29 | `arenteria@hillaryclinton.com` | 220 |
| 30 | `info@votevets.org` | 216 |
| 31 | `clu@barackobama.com` | 215 |
| 32 | `tina@presidentclinton.com` | 214 |
| 33 | `nmerrill@hrcoffice.com` | 209 |
| 34 | `cle@americanbar.org` | 209 |
| 35 | `mpbonner@bonnergrp.com` | 208 |
| 36 | `jake.sullivan@gmail.com` | 205 |
| 37 | `mrooney@hillaryclinton.com` | 200 |
| 38 | `meganrouse@gmail.com` | 189 |
| 39 | `aelrod@hillaryclinton.com` | 188 |
| 40 | `tom@zzranch.com` | 188 |
| 41 | `info@seanmaloney.com` | 184 |
| 42 | `brentbbi@webtv.net` | 174 |
| 43 | `mpodesta@gmail.com` | 169 |
| 44 | `footlocker@e.footlocker.com` | 165 |
| 45 | `kschake@hillaryclinton.com` | 164 |
| 46 | `creamer2@aol.com` | 164 |
| 47 | `gruncom@aol.com` | 164 |
| 48 | `info@kirkpatrickforarizona.com` | 164 |
| 49 | `tcarrk@hillaryclinton.com` | 160 |
| 50 | `mond1967@zahav.net.il` | 159 |

Corpus contains **3,844 unique sender addresses**.

---

## 6. MIME structure

Top-level `Content-Type` of each message (what the outer envelope advertises):

| Top-level content type | Count |
|------------------------|-------|
| `multipart/alternative` | 34,629 |
| `text/plain` | 6,356 |
| `multipart/mixed` | 5,960 |
| `multipart/related` | 2,531 |
| `text/html` | 1,023 |
| `multipart/report` | 376 |
| `text/calendar` | 8 |
| `image/jpeg` | 1 |
| `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | 1 |
| `multipart/signed` | 1 |
| `application/pdf` | 1 |

Content-Transfer-Encoding distribution across all MIME parts (not just top-level):

| Content-Transfer-Encoding | Parts |
|---------------------------|-------|
| `quoted-printable` | 64,844 |
| `base64` | 20,380 |
| `7bit` | 11,125 |
| `8bit` | 731 |
| `binary` | 13 |

---

## 7. Attachments

Totals: **14,119 attachment instances**, content-dedup'd to **10,276 unique blobs** totaling **2.7 GB** on disk.

### By content type (top 20)

| Attachment content type | Count |
|-------------------------|-------|
| `image/jpeg` | 4,171 |
| `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | 2,540 |
| `image/png` | 1,925 |
| `application/pdf` | 1,734 |
| `application/msword` | 1,014 |
| `application/ics` | 701 |
| `image/gif` | 674 |
| `application/octet-stream` | 397 |
| `text/html` | 186 |
| `text/plain` | 129 |
| `text/calendar` | 97 |
| `application/vnd.ms-excel` | 96 |
| `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | 73 |
| `image/bmp` | 52 |
| `application/ms-tnef` | 42 |
| `application/vnd.ms-powerpoint` | 40 |
| `image/jpg` | 40 |
| `application/vnd.openxmlformats-officedocument.presentationml.presentation` | 28 |
| `image/pjpeg` | 20 |
| `image/x-png` | 18 |

### By filename extension (top 20)

| Extension | Count |
|-----------|-------|
| `.jpg` | 4,136 |
| `.docx` | 2,573 |
| `.png` | 1,943 |
| `.pdf` | 1,898 |
| `.doc` | 1,082 |
| `.ics` | 800 |
| `.gif` | 674 |
| `.htm` | 118 |
| `.jpeg` | 114 |
| `.xls` | 102 |
| `.txt` | 86 |
| `.xlsx` | 77 |
| `.bmp` | 52 |
| `.dat` | 43 |
| `.ppt` | 39 |
| `.pptx` | 27 |
| `.vcf` | 23 |
| `.zip` | 23 |
| `.rtf` | 18 |
| `.emz` | 15 |

### Largest attachments (top 15)

| Size | Filename | Type |
|------|----------|------|
| 22.3 MB | `Barbara Walters Carol Birthday Greeting.mov` | `video/quicktime` |
| 20.8 MB | `Univision Hispanic Electorate Poll - Presentation 7.13.15 FINAL.pdf` | `application/pdf` |
| 19.8 MB | `2016ElectionPreviewEZ.pdf` | `application/msword` |
| 18.2 MB | `Covers(A-M)_20150311.docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` |
| 17.4 MB | `Covers(N-Z)_20150311.docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` |
| 14.8 MB | `ClimateWorks Board orientation binder_2015.pdf` | `application/pdf` |
| 13.9 MB | `2012 Budget Meeting Documents 2.zip` | `application/zip` |
| 13.9 MB | `2012 Budget Meeting Documents 2.zip` | `application/zip` |
| 13.9 MB | `2012 Budget Meeting Documents 2.zip` | `application/zip` |
| 12.1 MB | `3rd Commission Meeting Agenda and Papers.pdf` | `application/pdf` |
| 11.6 MB | `popup nola rough 4-24.mov` | `video/quicktime` |
| 11.3 MB | `LYVNOIWIExperimentBriefing.pdf` | `application/pdf` |
| 10.9 MB | `Theresa1.png` | `image/png` |
| 10.8 MB | `REVISED HFA Presentation 12.17.15.pptx` | `application/vnd.openxmlformats-officedocument.presentationml.presentation` |
| 10.5 MB | `B462E513-9DFC-4801-B7A7-18560D8EDBB8.png` | `image/png` |

---

## 8. Threading

| | |
|---|---|
| Messages with `In-Reply-To` header | 20,606 (40.5%) |
| Messages with `References` header | 21,036 (41.3%) |
| Orphan replies (IRT → unknown parent) | 5,714 |
| Largest reference-chain thread (node count) | 48 |

Threading derivation (root detection, union-find, canonical thread IDs) is deferred to #621. This report only profiles the raw coverage so the loader knows how many messages will need synthesized thread roots.

---

## 9. Character encodings

| Charset | Parts |
|---------|-------|
| `utf-8` | 54,441 |
| `us-ascii` | 17,095 |
| `iso-8859-1` | 14,809 |
| `windows-1252` | 5,499 |
| `windows-1255` | 287 |
| `big5` | 136 |
| `windows-1256` | 111 |
| `iso-2022-jp` | 58 |
| `koi8-r` | 52 |
| `iso-8859-7` | 32 |
| `iso-8859-8-i` | 32 |
| `gb2312` | 26 |
| `iso-8859-15` | 24 |
| `iso-8859-2` | 23 |
| `windows-1250` | 20 |
| `euc-kr` | 20 |
| `ascii` | 12 |
| `cp1252` | 10 |
| `iso-8859-9` | 6 |
| `gbk` | 6 |

Non-UTF-8 charsets are decoded with `errors='replace'` during parse. The HTML sanitizer in #621 will re-normalize to UTF-8 on write.

---

## 10. Anomalies

| Anomaly | Count |
|---------|-------|
| Synthetic Message-ID needed | 1 |
| Duplicate Message-ID | 42 |
| Malformed Date header | 0 |
| DSN / bounce (by MIME type or subject) | 426 |
| Attachment decode failure | 0 |

---

## 11. Readiness for ingest

Parser-emitted warnings the #621 DB loader must handle:

| Warning tag | Meaning | Loader behavior |
|-------------|---------|-----------------|
| `synthetic_id` | Original message had no `Message-ID` header | Accept synthetic ID as primary key; flag on message row |
| `malformed_date` | Date header present but unparseable | Store `date_raw`; leave `date_iso` NULL; sort by byte_offset fallback |
| `dsn_message` | MIME part `message/delivery-status` detected | Flag as bounce; do not thread into user conversations |
| `dsn_subject` | Subject matches bounce regex but no DSN MIME | Soft-flag; still index |
| `attachment_decode_failed` | Attachment payload could not be decoded | Record metadata row with NULL sha256; skip blob write |
| `attachment_write_failed` | Blob write to disk failed mid-pass | Retry on next run (idempotent via content address) |

The full JSONL (`ingest/pmail/staged/wikileaks-full.jsonl`) is the input to #621 — the loader reads one line at a time, hits the attachment blobs by sha256, and inserts into `pmail.db` per the schema already shipped in `9c66910`.