# TR-2026-0411-001: ChatGPT Transcript Extraction System — Implementation and Methodology
**Authors**: Surf CLI Research Group
**Date**: April 11, 2026
**Classification**: Technical Report
**Status**: Final
**Distribution**: Internal Research Archive
---
## Abstract
This report documents the design, implementation, and validation of the ChatGPT transcript extraction system within the Surf CLI browser automation framework. The system enables both interactive query-response cycles and post-hoc transcript export from ChatGPT conversations through DOM-based extraction, avoiding dependency on undocumented backend APIs or authentication tokens. We describe the discovery methodology involving seventeen systematic probe scripts, the deduplication algorithm for handling ChatGPT's redundant DOM structures, the Activity flyout extraction mechanism for reasoning models, and the dual-mode Glazed command architecture. The report includes a detailed failure analysis of the citation-fragment bug that affected early provider implementations, the turn-based selection algorithm that resolved it, and the validation framework used to verify extraction correctness against ground-truth DOM state.
---
## 1. Introduction and Problem Statement
### 1.1 Background
The Surf CLI project requires browser-based automation for multiple AI providers including ChatGPT, Claude, Gemini, Perplexity, and Grok. Unlike providers with stable API endpoints, ChatGPT's web interface requires DOM-level interaction. The project needed both:
1. **Interactive provider capability**: Send prompts to ChatGPT and capture responses in real-time
2. **Post-hoc transcript export**: Extract full conversation history from existing ChatGPT conversation pages
### 1.2 Technical Constraints
Several constraints shaped the implementation:
- **No stable API**: OpenAI does not provide a public API for ChatGPT's web interface
- **Authentication barriers**: Backend endpoints require bearer tokens not accessible to page-context JavaScript
- **DOM complexity**: ChatGPT renders conversation data with redundant wrapper structures and duplicate nodes
- **Dynamic content**: Reasoning models (o3, o4-mini) expose thought traces through interactive flyouts requiring user simulation
- **Provider isolation**: The browser extension launches a separate native host binary, meaning code changes require explicit installation
### 1.3 Research Questions
This investigation addressed four primary questions:
1. Can transcript data be reliably extracted from ChatGPT's DOM without backend API access?
2. What algorithm correctly deduplicates ChatGPT's redundant message node structures?
3. How can reasoning chain content be extracted from Activity flyouts that require user interaction?
4. What validation methodology distinguishes extraction algorithm bugs from completion-state timing issues?
---
## 2. Methodology: Systematic Discovery Through Probe Scripts
### 2.1 Research Approach
The investigation followed a systematic falsification methodology: probe each potential extraction vector, document failure modes, and converge on the viable path. Seventeen probe scripts were developed and executed through the `surf-go js` command against live ChatGPT conversations.
### 2.2 Phase I: Infrastructure Validation
**Objective**: Establish baseline JavaScript execution capability through the Surf CLI native messaging bridge.
**Script**: `js` command validation (inline)
**Command**:
```bash
go run ./cmd/surf-go js 'return document.title' \
--socket-path /home/manuel/snap/chromium/common/surf-cli/surf.sock
```
**Critical Finding**: The extension's `EXECUTE_JAVASCRIPT` handler wraps user code in an async IIFE:
```typescript
const expression = "(async () => {\n'use strict';\n" + message.code + "\n})()";
```
This has two implications:
- Scripts must use explicit `return` statements to emit values
- Template literals in user code can be corrupted if not handled correctly (see Section 6.3)
### 2.3 Phase II: DOM Inventory
**Script**: `chatgpt_transcript_dom_summary.js`
**Purpose**: Enumerate message nodes and their attributes
**Methodology**:
```javascript
const turns = Array.from(document.querySelectorAll(
'[data-message-author-role], [data-turn="assistant"], [data-turn="user"]'
));
```
**Observations**:
- Nodes carry `data-message-author-role` ("user" | "assistant" | "system")
- Nodes carry `data-message-id` (UUID format)
- Assistant nodes carry `data-message-model-slug` (e.g., "gpt-5-4-thinking")
- **Critical**: Both wrapper containers and content nodes match the selector, producing duplicates
**Raw DOM Structure**:
```
section[data-testid="conversation-turn-0"]
└─ div[data-message-author-role="user"][data-message-id="abc-123"]
└─ div[data-message-author-role="user"] ← duplicate inner node
section[data-testid="conversation-turn-1"]
└─ div[data-message-author-role="assistant"][data-message-id="def-456"][data-message-model-slug="gpt-4o"]
└─ div[data-message-author-role="assistant"] ← duplicate inner node
└─ div.markdown ← actual content
```
### 2.4 Phase III: Backend Endpoint Reconnaissance
This phase applies the systematic [[ARTICLE - surf-go Browser Verbs - Using JS Probes to Build Reliable Web Automation|JS probe methodology]]: iterate on live pages, preserve probe scripts, identify selectors through evidence rather than inspection.
**Scripts**:
- `chatgpt_transcript_resource_scan.js`
- `chatgpt_transcript_backend_probe.js`
- `chatgpt_transcript_textdocs_probe.js`
**Endpoint Survey**:
| Endpoint | Response | Assessment |
|----------|----------|------------|
| `/backend-api/conversation/<id>` | 404 Not Found | Non-existent or requires different ID format |
| `/backend-api/share/<id>` | 404 Not Found | Sharing not enabled for target conversation |
| `/backend-api/public/conversation/<id>` | 404 Not Found | Not a public conversation |
| `/backend-api/conversation/<id>/textdocs` | 401 Unauthorized | **Endpoint exists but requires bearer token** |
**Authentication Surface Scan** (`chatgpt_transcript_auth_surface_scan.js`):
- `localStorage`: No bearer token found
- `sessionStorage`: No bearer token found
- `document.cookie`: Contains `__Secure-next-auth.session-token` but not API bearer token
- Global window keys: No obvious Apollo/GraphQL auth caches
**IndexedDB Survey** (`chatgpt_transcript_indexeddb_scan.js`):
- No ChatGPT application database with transcript content visible to page context
**Conclusion**: Backend replay is infeasible without network-layer token capture. DOM extraction remains the viable path.
### 2.5 Phase IV: Activity/Thought Trace Discovery
**Scripts**:
- `chatgpt_thought_trace_scan.js` — Text pattern matching for thought indicators
- `chatgpt_activity_button_map.js` — Enumerate thought buttons per conversation turn
- `chatgpt_thought_trace_click_probe.js` — Test click behavior and flyout detection
**Key Discovery**: The "Thought for N seconds" button is a plain `<button>` element with text matching `/Thought for/i`. When clicked, it opens a flyout panel with:
- Header containing "Activity" and the duration
- Full reasoning chain text
- Sources section (if browsing enabled)
**Flyout DOM Structure**:
```
body > div:nth-child(5) > div > ... > div.stage-thread-flyout-preset-default
└─ div > div > section
├─ text: "Activity"
├─ text: "Thought for 12 seconds"
└─ div ← reasoning content
```
**Race Condition Discovery**: The flyout takes 400-1200ms to appear after click. The Activity header must be matched against the button's duration text to confirm the correct flyout opened (multiple assistant turns may have different durations).
### 2.6 Probe Script Catalog
| Script | Phase | Purpose | Lines |
|--------|-------|---------|-------|
| `chatgpt_transcript_dom_summary.js` | II | Raw DOM inventory | 14 |
| `chatgpt_transcript_extract_dom.js` | II | Deduplication algorithm test | 25 |
| `chatgpt_transcript_resource_scan.js` | III | Resource and endpoint discovery | 18 |
| `chatgpt_transcript_backend_probe.js` | III | Backend API probing | 35 |
| `chatgpt_transcript_textdocs_probe.js` | III | Specific `/textdocs` test | 12 |
| `chatgpt_transcript_auth_surface_scan.js` | III | Token/cache discovery | 20 |
| `chatgpt_transcript_cache_scan.js` | III | localStorage/sessionStorage scan | 22 |
| `chatgpt_transcript_indexeddb_scan.js` | III | IndexedDB enumeration | 25 |
| `chatgpt_thought_trace_scan.js` | IV | Thought pattern detection | 28 |
| `chatgpt_activity_button_map.js` | IV | Button enumeration per turn | 32 |
| `chatgpt_thought_trace_click_probe.js` | IV | Click behavior analysis | 38 |
| `chatgpt_activity_open_single.js` | IV | Single flyout extraction | 85 |
| `chatgpt_activity_open_sequence.js` | IV | Sequential multi-flyout | 45 |
| `chatgpt_activity_export_first_three.js` | IV | Batch activity export | 92 |
| `chatgpt_activity_sidebar_extract.js` | IV | Sidebar content detection | 24 |
| `chatgpt_transcript_export_with_activity.js` | V | Final integrated extractor | 167 |
---
## 3. Implementation: DOM Extraction Algorithms
### 3.1 Turn-Based Deduplication Algorithm
**Problem**: ChatGPT renders each logical message as multiple DOM nodes (wrapper + content containers), all with the same `data-message-author-role` and `data-message-id` attributes.
**Anti-Pattern (Global Selection)**:
```javascript
// FAILS: Returns duplicates, may select citation fragments
const allNodes = document.querySelectorAll('[data-message-author-role="assistant"]');
const lastNode = allNodes[allNodes.length - 1]; // May be citation, not main body
```
**Correct Algorithm (Turn-Based Longest-Text Selection)**:
```javascript
function extractSectionMessage(section, index) {
// 1. Collect all role-marked nodes within the turn boundary
const candidates = Array.from(section.querySelectorAll('[data-message-author-role]'));
// 2. Deduplicate by message ID, keeping longest text per ID
const byMessageId = new Map();
for (const node of candidates) {
const role = node.getAttribute('data-message-author-role') || 'unknown';
const messageId = node.getAttribute('data-message-id')
|| `${role}:${index}:${byMessageId.size}`;
const model = node.getAttribute('data-message-model-slug') || null;
const text = (node.innerText || '').trim();
if (!text) continue;
const existing = byMessageId.get(messageId);
if (!existing || text.length > existing.text.length) {
byMessageId.set(messageId, { role, model, messageId, text, textLength: text.length });
}
}
// 3. Select best candidate for this turn (longest text)
const items = Array.from(byMessageId.values())
.sort((a, b) => b.textLength - a.textLength);
return items[0] || null;
}
// 4. Iterate conversation turns in DOM order
const sections = Array.from(document.querySelectorAll(
'section[data-testid^="conversation-turn-"]'
));
const transcript = sections.map((section, i) => extractSectionMessage(section, i))
.filter(Boolean);
```
**Algorithmic Properties**:
- **Time Complexity**: O(n × m) where n = turn count, m = nodes per turn
- **Space Complexity**: O(k) where k = unique message IDs per turn
- **Correctness**: Guaranteed to select content over wrappers because content nodes contain strictly more text
- **Robustness**: Citation fragments have short text; main assistant response has long text
### 3.2 Activity Flyout Extraction Algorithm
**Challenge**: Reasoning model thought traces are hidden behind interactive buttons that open flyouts. The flyout:
- Takes non-deterministic time to appear (400-7000ms)
- May not open on first click (race condition with React event handling)
- Must be matched by duration string to confirm correct flyout opened
**Algorithm**:
```javascript
const ACTIVITY_SELECTORS = [
'body > div:nth-child(5) > div > ... > section', // Exact path from DOM analysis
'[class*="stage-thread-flyout"] section', // Fallback class match
];
const ACTIVITY_HEADER_RE = /Activity\s*[^\w\n]?\s*[^\n]+/i;
function getActivitySection() {
for (const selector of ACTIVITY_SELECTORS) {
for (const node of document.querySelectorAll(selector)) {
const text = (node.innerText || '').trim();
if (ACTIVITY_HEADER_RE.test(text) || text.includes('Thinking\n')) {
return { selector, node, text };
}
}
}
return null;
}
function activityMatchesDuration(activityText, duration) {
const escaped = duration.replace(/[.*+?^${}()|[\]\\]/g, '\\
amp;');
return new RegExp(`Activity\s*[^\\w\\n]?\s*${escaped}`, 'i').test(activityText);
}
async function normalizeAndOpen(button) {
const buttonText = (button.textContent || '').trim().replace(/\s+/g, ' ');
const duration = durationFromButtonText(buttonText); // "12 seconds" from "Thought for 12 seconds"
const previousActivity = getActivitySection();
button.scrollIntoView({ block: 'center', inline: 'center' });
await sleep(250); // Ensure element is interactable
// Retry up to 2 times
for (let attempt = 1; attempt <= 2; attempt++) {
button.click();
const hit = await waitForCondition(() => {
const activity = getActivitySection();
if (!activity) return null;
// Match duration to confirm correct flyout; ignore stale flyout on retry
if (!activityMatchesDuration(activity.text, duration)) return null;
if (previousActivity && activity.text === previousActivity.text && attempt === 1)
return null; // Stale flyout from previous turn
return activity;
}, 7000);
if (hit) return { attempt, duration, text: hit.value.text, ... };
await sleep(400);
}
return { attempt: 2, text: null }; // Failed to open
}
```
**Timing Parameters** (determined empirically):
- `scrollIntoView` delay: 250ms
- Post-click wait interval: 250ms polling
- Maximum wait per attempt: 7000ms
- Inter-attempt delay: 400ms
- Inter-turn delay (batch mode): 500ms
### 3.3 Embedded Script Architecture
The Go command embeds the JavaScript using Go 1.16+ `embed` directive:
```go
//go:embed scripts/chatgpt_transcript.js
var chatGPTTranscriptScript string
func buildChatGPTTranscriptCode(s *ChatGPTTranscriptSettings) (string, error) {
options := map[string]any{
"withActivity": s.WithActivity,
"activityLimit": s.ActivityLimit,
}
b, _ := json.Marshal(options)
// Prepend options as global SURF_OPTIONS object
return fmt.Sprintf("const SURF_OPTIONS = %s;\n%s", string(b), chatGPTTranscriptScript), nil
}
```
**Execution Flow**:
1. Go command builds script with embedded options
2. Script sent through `js` tool via native messaging
3. Extension wraps in async IIFE: `(async () => {\n'use strict';\n<script>\n})()`
4. CDP `Runtime.evaluate` executes in page context
5. Result serialized and returned through Glazed pipeline
---
## 4. Implementation: Command Architecture
### 4.1 Dual-Mode Glazed Command Design
The `chatgpt-transcript` command implements Glazed's dual-mode pattern:
```go
type ChatGPTTranscriptCommand struct {
*cmds.CommandDescription
}
// Writer mode: human-readable Markdown
var _ cmds.WriterCommand = (*ChatGPTTranscriptCommand)(nil)
// Glaze mode: structured data rows
var _ cmds.GlazeCommand = (*ChatGPTTranscriptCommand)(nil)
```
**Mode Selection**:
- Default (`--with-glaze-output=false`): `RunIntoWriter` emits Markdown transcript
- Structured (`--with-glaze-output=true`): `RunIntoGlazeProcessor` emits row data
**Critical Implementation Detail**:
When implementing both interfaces, the command builder must use:
```go
cli.WithDualMode(true)
cli.WithGlazeToggleFlag("with-glaze-output")
```
Otherwise the writer path takes precedence and Glaze output is never reached.
### 4.2 Shared Transcript Fetch Helper
Both modes use a common fetch helper:
```go
func fetchChatGPTTranscript(ctx context.Context, s *ChatGPTTranscriptSettings) (*chatGPTTranscriptData, error) {
code, err := buildChatGPTTranscriptCode(s)
if err != nil {
return nil, err
}
resp, err := ExecuteTool(ctx, client, "js", map[string]any{"code": code}, tabID, windowID)
if err != nil {
return nil, err
}
return parseChatGPTTranscriptResponse(resp)
}
```
This ensures consistent behavior between human-readable and machine-readable outputs.
### 4.3 Export Pipeline
**Artifact Export** (`--export-file`):
```go
func writeChatGPTTranscriptExport(path string, format string, data *chatGPTTranscriptData) error {
switch format {
case "json":
body, _ = json.MarshalIndent(data.Raw, "", " ")
case "markdown":
body = []byte(renderChatGPTTranscriptMarkdown(data.Raw))
}
return os.WriteFile(path, body, 0644)
}
```
**Row Expansion**:
Each transcript turn becomes one Glazed row with flattened metadata:
```go
func chatGPTTranscriptDataToRows(data *chatGPTTranscriptData) []types.Row {
dataMap := data.Raw
items, _ := dataMap["transcript"].([]any)
for _, item := range items {
m, _ := item.(map[string]any)
rowMap := map[string]any{
"href": dataMap["href"], // Conversation URL
"title": dataMap["title"], // Page title
"turnCount": dataMap["turnCount"],
"withActivity": dataMap["withActivity"],
"activityLimit": dataMap["activityLimit"],
"activityExported": dataMap["activityExported"],
// ... per-turn fields
"index": m["index"],
"role": m["role"],
"model": m["model"],
"messageId": m["messageId"],
"textLength": m["textLength"],
"text": m["text"],
"hasThought": m["hasThought"],
"thoughtButtonText": m["thoughtButtonText"],
"activityFound": m["activityFound"],
"activityText": m["activityText"],
"activityAttempts": m["activityAttempts"],
"activityWaitedMs": m["activityWaitedMs"],
}
rows = append(rows, types.NewRowFromMap(rowMap))
}
return rows
}
```
### 4.4 Command Flag Specification
| Flag | Type | Default | Description |
|------|------|---------|-------------|
| `with-activity` | bool | false | Open and scrape thought trace flyouts |
| `activity-limit` | int | 0 | Max flyouts to open (0 = unlimited) |
| `export-file` | string | "" | Write transcript to file |
| `export-format` | choice | "markdown" | "markdown" or "json" |
| `socket-path` | string | platform default | Native host socket path |
| `timeout-ms` | int | 120000 | Socket request timeout |
| `tab-id` | int | -1 | Override target tab |
| `window-id` | int | -1 | Override target window |
| `debug-socket` | bool | false | Log socket frames to stderr |
| `with-glaze-output` | bool | false | Emit structured rows vs Markdown |
---
## 5. Implementation: Interactive Provider
### 5.1 Provider Architecture
The interactive `chatgpt` provider (in `go/internal/host/providers/chatgpt.go`) operates differently from the transcript command:
- **Transcript command**: Client-side Go command → Extension → Page JS
- **Interactive provider**: Extension launches `surf-host-go` → Provider polls page via CDP
This distinction is critical: provider changes require rebuilding the installed host binary.
### 5.2 Conversation Turn Polling Algorithm
The provider polls for assistant response completion:
```javascript
const expr = `(() => {
// 1. Find all conversation turns
const turns = Array.from(document.querySelectorAll(
'section[data-testid^="conversation-turn-"], article[data-testid^="conversation-turn"]'
));
// 2. Extract best assistant from each turn
function extractBestAssistantFromTurn(turn) {
const candidates = Array.from(turn.querySelectorAll(
'[data-message-author-role="assistant"], [data-turn="assistant"]'
));
if (candidates.length === 0) return null;
// 3. Deduplicate by message ID
const byMessageId = new Map();
for (const node of candidates) {
const messageId = node.getAttribute('data-message-id')
|| ('assistant:' + byMessageId.size);
const contentRoot = node.querySelector('.markdown')
|| node.querySelector('[data-message-content]')
|| node.querySelector('.prose')
|| node;
const text = (contentRoot?.innerText || '').trim();
if (!text) continue;
const existing = byMessageId.get(messageId);
const candidate = {
node, messageId, text, textLength: text.length,
hasMarkdown: Boolean(node.querySelector('.markdown')),
hasProse: Boolean(node.querySelector('.prose')),
};
if (!existing || candidate.textLength > existing.textLength) {
byMessageId.set(messageId, candidate);
}
}
// 4. Return longest for this turn
const items = Array.from(byMessageId.values())
.sort((a, b) => b.textLength - a.textLength);
return items[0] || null;
}
// 5. Find last assistant turn (scanning backwards)
let lastAssistantTurn = null;
let bestAssistant = null;
for (let i = turns.length - 1; i >= 0; i--) {
const best = extractBestAssistantFromTurn(turns[i]);
if (best) {
if (!bestAssistant) {
bestAssistant = best;
lastAssistantTurn = turns[i];
}
}
}
// 6. Detect completion state
const stopVisible = Boolean(document.querySelector('[data-testid="stop-button"]'));
const finished = Boolean(lastAssistantTurn?.querySelector(
'[data-testid="copy-turn-action-button"], [data-testid="good-response-turn-action-button"]'
));
return {
text: bestAssistant?.text || '',
stopVisible,
finished,
messageId: bestAssistant?.messageId,
turnCount: turns.length,
foundAssistant: !!bestAssistant,
};
})()`;
```
### 5.3 Completion Detection Heuristics
The provider uses three signals to detect response completion:
1. **Stop button visibility**: `stopVisible = false` indicates generation stopped
2. **Finished markers**: Copy/good response buttons present on the turn
3. **Text stability**: No length change for 6 consecutive polls over 1200ms minimum
**Polling Parameters**:
- Poll interval: 400ms
- Required stable cycles: 6
- Minimum stable duration: 1200ms
- Global timeout: 45 minutes (default, configurable)
### 5.4 File Upload Handling
The provider handles file attachments through CDP's DOM domain:
```javascript
// 1. Locate or trigger file input
const selector = await waitForFileInputSelector(cdp, 12000);
// Returns: [data-surf-file-input-id="surf-upload-<timestamp>-<random>"]
// 2. Extension performs actual upload via native file chooser protocol
resp = await caller.Request(ctx, map[string]any{
"type": "UPLOAD_FILE",
"tabId": tabID,
"selector": selector,
"files": []string{"/path/to/file.txt"},
}, 45*time.Second)
```
The selector is injected as a unique `data-surf-file-input-id` attribute to ensure stable identification across the CDP → Extension → Browser pipeline.
---
## 6. Failure Analysis: The Citation-Fragment Bug
### 6.1 Symptom Description
On April 10, 2026, the interactive `chatgpt` command returned garbled output for a research-heavy prompt:
**Observed Output**:
```
MIT OpenCourseWare
MIT Press
Mathematical Association of America
MIT OpenCourseWare
...
```
**Expected Output**: A substantive 18,933-character response about "The Art of Insight in Science and Engineering" by Sanjoy Mahajan.
### 6.2 Root Cause Analysis
**Original (Buggy) Algorithm**:
```javascript
// INCORRECT: Global node selection
const assistantNodes = document.querySelectorAll('[data-message-author-role="assistant"]');
const lastNode = assistantNodes[assistantNodes.length - 1];
const text = lastNode.innerText;
```
**Problem**: ChatGPT renders citations and sources as separate assistant-marked elements at the end of the document. These fragments have:
- `data-message-author-role="assistant"` (same attribute)
- Short text length (typically 50-200 characters)
- Located after the main response body in DOM order
The global `querySelectorAll` approach selected the citation fragment instead of the main response.
### 6.3 Resolution: Turn-Based Selection
**Corrected Algorithm** (mirrors transcript extractor):
```javascript
// CORRECT: Turn-based selection
const turns = document.querySelectorAll('[data-testid^="conversation-turn-"]');
const lastTurn = turns[turns.length - 1];
const candidates = lastTurn.querySelectorAll('[data-message-author-role="assistant"]');
// Deduplicate and select longest
const byMessageId = new Map();
for (const node of candidates) {
// ... collect by message ID
}
const items = Array.from(byMessageId.values())
.sort((a, b) => b.textLength - a.textLength);
const best = items[0]; // Main response (long text) wins over citation (short text)
```
### 6.4 Validation Methodology
**Ground Truth Establishment**:
1. Identify conversation tab ID: `441390650`
2. Run transcript command on same tab:
```bash
go run ./cmd/surf-go chatgpt-transcript \
--tab-id 441390650 --with-glaze-output --output yaml
```
3. Record transcript output:
- Turn 1: User prompt
- Turn 2: Assistant response, 18,933 characters, model `gpt-5-4-thinking`
**Provider Log Analysis**:
```bash
snap run --shell chromium -c 'tail -n 120 /tmp/surf-host-go.log'
```
**Before Fix**:
```
[chatgpt] waitForResponse poll=1 len=0 stop=true finished=false ... turnCount=0 foundAssistant=false
[chatgpt] waitForResponse poll=47 len=187 stop=true finished=false ... turnCount=0 foundAssistant=false
```
**After Fix**:
```
[chatgpt] waitForResponse poll=1 len=0 stop=true finished=false ... turnCount=2 foundAssistant=false
[chatgpt] waitForResponse poll=523 len=18933 stop=false finished=true ... turnCount=2 foundAssistant=true
```
**Key Indicators**:
- `turnCount=2`: Correctly identifies two conversation turns (user + assistant)
- `foundAssistant=true`: Successfully located assistant content within turn
- `len=18933`: Full response body extracted
### 6.5 Remaining Issue: Completion Gate Timing
After fixing extraction selection, a secondary issue emerged: long-running research responses may remain in polling state because `stopVisible` stays `true` for extended periods during browsing/tool use.
**Status**: Extraction selection bug is resolved. Completion-gate behavior tracked separately.
---
## 7. Implementation: Shared Tab Readiness Helper
### 7.1 Problem Context
Commands that create fresh tabs (Kagi, Gmail) need deterministic readiness before running extraction scripts. Ad hoc approaches produced:
- Race conditions between tab creation and JS execution context availability
- Inconsistent retry logic across commands
- Hard-to-test transport sequences
### 7.2 Implementation
**File**: `go/internal/cli/commands/tab_ready.go`
```go
const (
defaultTabReadyTimeout = 20 * time.Second
tabReadyRetryInterval = 400 * time.Millisecond
tabReadyProbeScript = `return { href: location.href, title: document.title, readyState: document.readyState }`
tabExecutionContextErrMsg = "Cannot find default execution context"
)
func openOwnedTab(ctx context.Context, client *transport.Client, url string, opts tabReadyOptions) (int64, error) {
// 1. Create tab
tabResp, err := ExecuteTool(ctx, client, "tab.new", map[string]any{"url": url}, nil, nil)
tabID, _ := extractTabIDFromResponse(tabResp)
// 2. Wait for readiness
if err := waitForTabReady(ctx, client, tabID, opts); err != nil {
return 0, err
}
return tabID, nil
}
func waitForTabReady(ctx context.Context, client *transport.Client, tabID int64, opts tabReadyOptions) error {
deadline := time.Now().Add(opts.Timeout)
for time.Now().Before(deadline) {
state, err := probeTabReady(ctx, client, tabID)
if err == nil {
if state.ReadyState == "complete" &&
state.Href != "" &&
state.Href != "about:blank" &&
tabURLMatches(state.Href, opts) {
return nil // Ready
}
} else if !strings.Contains(err.Error(), tabExecutionContextErrMsg) {
return err // Non-retryable error
}
time.Sleep(tabReadyRetryInterval)
}
return fmt.Errorf("tab %d not ready before timeout", tabID)
}
```
**Readiness Criteria**:
1. `document.readyState === "complete"`
2. `location.href` is non-empty and not `about:blank`
3. URL matches exact or prefix constraint (if specified)
4. JS execution context is available (no "Cannot find default execution context" error)
### 7.3 Integration Pattern
Commands using the helper follow this sequence:
```
1. tab.new (create)
2. js probe (readiness - shared helper)
3. js extractor (page-specific - command-provided)
4. tab.close (cleanup - unless --keep-tab-open)
```
**Applied To**:
- `kagi_search.go` — replaces ad hoc retries
- `kagi_assistant.go` — replaces ad hoc retries
- Future Gmail commands
**Not Applied To**:
- `chatgpt-transcript` — operates on existing page, not fresh tab
---
## 8. Testing and Validation Framework
### 8.1 Unit Test Coverage
**Transcript Command Tests** (`chatgpt_transcript_test.go`):
| Test | Purpose |
|------|---------|
| `TestBuildChatGPTTranscriptCodeIncludesOptionsAndScript` | Verify SURF_OPTIONS prelude injection |
| `TestChatGPTTranscriptResponseToRowsExpandsTranscript` | Verify row expansion from JSON response |
| `TestWriteChatGPTTranscriptExportMarkdown` | Markdown artifact generation |
| `TestWriteChatGPTTranscriptExportJSON` | JSON artifact generation |
| `TestRenderChatGPTTranscriptMarkdown` | Markdown formatting correctness |
**Provider Tests** (`chatgpt_test.go`):
| Test | Purpose |
|------|---------|
| `TestParseChatGPTRequest` | Request parsing and validation |
| `TestHandleChatGPTToolSuccess` | End-to-end provider flow with fake caller |
| `TestHandleChatGPTToolWithPageContext` | Page context injection (`--with-page`) |
| `TestHandleChatGPTToolWithFileUpload` | File attachment flow |
| `TestHandleChatGPTToolListModels` | Model enumeration flow |
### 8.2 Fake Native Caller Pattern
Provider tests use a `fakeNativeCaller` that simulates CDP responses:
```go
type fakeNativeCaller struct {
handler func(msg map[string]any) (map[string]any, error)
}
func (f *fakeNativeCaller) Request(ctx context.Context, msg map[string]any, timeout time.Duration) (map[string]any, error) {
return f.handler(msg)
}
```
**Response Simulation**:
The handler switches on `msg["type"]` to return appropriate synthetic responses:
- `GET_CHATGPT_COOKIES` → session token present
- `CHATGPT_NEW_TAB` → `{"tabId": 42}`
- `CHATGPT_EVALUATE` → switches on expression content to simulate DOM state
- `CHATGPT_CDP_COMMAND` → `{"ok": true}`
- `CHATGPT_CLOSE_TAB` → `{"success": true}`
### 8.3 Integration Test Patterns
Integration tests in `cmd/surf-go/integration_test.go` verify:
- Command registration in root command
- Flag parsing and tool routing
- Tool response to row conversion
### 8.4 Live Browser Validation Checklist
**Fresh-Tab Commands** (Kagi):
```bash
export SURF_SOCKET_PATH=/home/manuel/snap/chromium/common/surf-cli/surf.sock
go run ./cmd/surf-go kagi-search --query "hello" --keep-tab-open
```
**Existing-Page Commands** (ChatGPT transcript):
```bash
# 1. Identify tab ID
# 2. Extract transcript
go run ./cmd/surf-go chatgpt-transcript \
--tab-id <TAB_ID> \
--with-activity \
--export-file /tmp/transcript.md
# 3. Validate artifact
cat /tmp/transcript.md
```
**Provider Validation** (requires installed host):
```bash
# 1. Rebuild installed host
go build -o /home/manuel/snap/chromium/common/surf-cli/surf-host-go \
./cmd/surf-host-go
# 2. Run query
surf chatgpt "Explain quantum computing"
# 3. Check logs
snap run --shell chromium -c 'tail -n 120 /tmp/surf-host-go.log'
```
---
## 9. Status and Open Problems
### 9.1 Implemented and Validated
| Component | Status | Validation |
|-----------|--------|------------|
| DOM deduplication algorithm | ✅ Production | Unit tests + live extraction |
| Turn-based selection | ✅ Production | Citation-fragment bug resolved |
| Activity flyout extraction | ✅ Production | Tested with o3/o4-mini models |
| Dual-mode Glazed command | ✅ Production | Markdown and row output verified |
| Artifact export (JSON/Markdown) | ✅ Production | File generation tested |
| Shared tab readiness helper | ✅ Production | Kagi commands stabilized |
| Provider turn-based extraction | ✅ Production | Host log confirms `turnCount` |
### 9.2 Known Limitations
| Issue | Severity | Description |
|-------|----------|-------------|
| Activity extraction timing | Medium | 400-7000ms non-deterministic flyout appearance; 2-attempt retry may fail on slow connections |
| Completion gate on long research | Low | `stopVisible` remains true during browsing; stability heuristic may delay return |
| Template literal restrictions | Low | Service worker IIFE wrapper; research scripts must avoid `${...}` interpolation |
| Backend API inaccessibility | Accepted | `/textdocs` endpoint requires token not available to page JS |
### 9.3 Open Research Questions
1. **Token capture**: Can the extension's network layer capture the bearer token required for `/backend-api/conversation/<id>/textdocs`? This would enable backend-based extraction more reliable than DOM scraping.
2. **Streaming extraction**: Currently the provider polls at 400ms intervals. Can CDP's `Runtime.addBinding` or `console` API enable event-driven streaming as tokens arrive?
3. **Pagination**: Very long conversations may have turn virtualization. Does the algorithm correctly handle lazy-loaded historical turns?
4. **Model-specific selectors**: The `data-testid` attributes are stable but not guaranteed. Can we build a selector fallback chain that degrades gracefully if ChatGPT changes DOM structure?
### 9.4 Recommended Future Work
**Immediate**:
- Add `--streaming` flag to `chatgpt` command for real-time partial response delivery
- Implement automatic conversation pagination detection
**Medium-term**:
- Investigate network-layer token extraction for backend API access
- Build DOM mutation observer-based extraction for event-driven updates
**Long-term**:
- Extract common Activity flyout patterns into reusable helper for other providers (Claude, Perplexity)
- Formalize the probe script methodology into a documented reconnaissance playbook (see [[ARTICLE - surf-go Browser Verbs - Using JS Probes to Build Reliable Web Automation|Browser Verbs Playbook]])
- Apply [[PROJ - Improving Minitrace and Transcript Analysis|transcript analysis tooling]] to extract patterns from this system's probe script sessions
---
## 10. Conclusion
The ChatGPT transcript extraction system demonstrates that reliable browser automation can be achieved through systematic DOM analysis without dependence on private APIs. The key technical contributions are:
1. **The turn-based longest-text algorithm**, which correctly handles ChatGPT's redundant DOM structures by selecting content over wrapper nodes through length-based ranking within conversation turn boundaries.
2. **The Activity flyout extraction protocol**, which solves the interactive flyout problem through duration-matching retry logic with empirical timing parameters.
3. **The dual-mode command architecture**, which provides both human-readable (Markdown) and machine-readable (Glazed rows) output through a unified extraction core. The implementation uses Go-native dual-mode commands (implementing `WriterCommand` and `GlazeCommand` interfaces) rather than code-generated definitions.
4. **The systematic probe methodology**, which documented seventeen falsification attempts before converging on the DOM-based approach, preserving the decision trail for future maintainers. This process is formalized in [[ARTICLE - surf-go Browser Verbs - Using JS Probes to Build Reliable Web Automation|the Browser Verbs playbook]].
The citation-fragment bug and its resolution illustrate the importance of ground-truth validation: the transcript command served as the DOM-grounded reference against which the interactive provider was corrected. This pattern—extractor as validator—should be applied to future provider implementations.
All probe scripts, design documents, and implementation code are preserved in the repository under `ttmp/` and `go/internal/cli/commands/` for future reference and replication.
---
## Related Vault Notes
These notes are directly related to the DOM extraction approach, command authoring patterns, or the same Surf CLI project:
- [[PROJ - DOM Scraping Experiment - Web to Markdown via JS DOM Queries]] — Parallel DOM extraction work using jsdom + querySelectorAll pipelines; shares the exploration-then-generation workflow
- [[PROJ - Claude Agent SDK - Teaching an AI to Write Web Scrapers]] — Uses Claude to automate the DOM investigation and scraper generation process
- [[PROJ - go-go-goja jsverbs - JavaScript to Glazed Commands]] — The JavaScript-to-Glazed authoring pattern that informs dual-mode command design
- [[PROJ - Surf CLI - ChatGPT Transcript Extraction]] — The companion project note with the feature overview and architecture summary
- [[ARTICLE - surf-go Browser Verbs - Using JS Probes to Build Reliable Web Automation]] — The broader Surf CLI methodology: probe → freeze → validate
---
## References
### Primary Sources
1. Research Diary: `ttmp/2026/04/08/SURF-20260408-R4--surf-go-non-provider-cli-parity-architecture-and-implementation-guide/reference/02-chatgpt-transcript-download-research-diary.md`
2. Bug Report: `ttmp/2026/04/10/SURF-20260410-R6--shared-tab-readiness-helper-and-chatgpt-extraction-bug/reference/01-chatgpt-extraction-bug-report.md`
3. Design Document: `ttmp/2026/04/10/SURF-20260410-R6--shared-tab-readiness-helper-and-chatgpt-extraction-bug/design-doc/01-shared-tab-readiness-helper-design.md`
4. Implementation Guide: `ttmp/2026/04/10/SURF-20260410-R6--shared-tab-readiness-helper-and-chatgpt-extraction-bug/design-doc/02-implementation-guide.md`
### Code Artifacts
5. Transcript Command: `go/internal/cli/commands/chatgpt_transcript.go`
6. Transcript Script: `go/internal/cli/commands/scripts/chatgpt_transcript.js`
7. Provider Implementation: `go/internal/host/providers/chatgpt.go`
8. Tab Readiness Helper: `go/internal/cli/commands/tab_ready.go`
9. JS Command: `go/internal/cli/commands/js.go`
### Test Artifacts
10. Transcript Tests: `go/internal/cli/commands/chatgpt_transcript_test.go`
11. Provider Tests: `go/internal/host/providers/chatgpt_test.go`
12. Native Client: `native/chatgpt-client.cjs`
### Probe Script Archive
13. Full probe script collection: `ttmp/2026/04/08/SURF-20260408-R4--surf-go-non-provider-cli-parity-architecture-and-implementation-guide/scripts/`
---
**End of Technical Report TR-2026-0411-001**