Documentation

Summary

Senate Hearings is a comprehensive Python package for scraping, extracting, and analyzing congressional hearing data from the govinfo website. It combines web scraping with natural language processing techniques to automatically identify, retrieve, and process hearing transcripts and metadata.

The package provides a complete pipeline for:

Web Scraping and Content Extraction: Dynamically retrieves fully expanded HTML content from govinfo pages using Selenium, handling JavaScript-rendered content and interactive elements.
Transcript Parsing: Extracts and cleans raw congressional hearing transcripts, removing formatting artifacts, metadata, and structural noise to produce readable narrative text.
Metadata Extraction: Automatically extracts key metadata from hearing documents including titles, dates, categories, and session information.
Keyword Extraction and Normalization: Implements advanced keyword extraction using Maximal Marginal Relevance (MMR) to identify semantically relevant and diverse keywords from hearing content. Keywords are filtered, normalized, and consolidated for consistency.
Data Management: Processes and manages large collections of hearing data, supporting CSV I/O operations and structured data organization.

Key Features:

Robust Web Scraping: Handles dynamic content, timeouts, and page interactions gracefully
Intelligent Keyword Filtering: Removes banned terms and person names automatically
Semantic Keyword Selection: Uses embeddings and MMR to select diverse, relevant keywords
Text Normalization: Converts variant forms (abbreviations, capitalization) to canonical representations
Scalable Data Processing: Processes congressional session data efficiently in batch operations

The package is ideal for researchers, data scientists, and analysts working with legislative records, policy analysis, and congressional hearing data.

Functions

is_banned_candidate()

Description:

Determines whether a candidate keyword phrase should be excluded from further consideration.

A candidate is considered banned if it contains:
Any substring from a predefined list of banned terms, or
Any person name from a provided set

All checks are performed in a case-insensitive manner. This function is typically used early in the pipeline to remove inappropriate or unwanted keyword phrases.

Parameters:

c (str):

Candidate keyword phrase to evaluate.

banned_substrings (list[str]):

List of substrings that should not appear in valid keywords.

person_names (set[str]):

Set of person names that should be excluded if found in a candidate.

Returns:

bool

Returns True if the candidate should be banned, otherwise False.

Examples:

from senate_hearings.keybert_funcs import *

is_banned_candidate(
    c="john doe analysis",
    banned_substrings=["spam"],
    person_names={"john doe"}
)
# True
is_banned_candidate(
    c="machine learning",
    banned_substrings=["spam"],
    person_names=set()
)
# False

False

mmr_select()

Description:

Selects a subset of candidate keywords using Maximal Marginal Relevance (MMR).

MMR balances:

Relevance of each candidate to the document, and diversity relative to keywords that have already been selected.

This approach helps avoid redundant keywords while still prioritizing semantic relevance. Similarity calculations are based on vector dot products (cosine similarity when embeddings are normalized).

Parameters:

doc_emb (numpy.ndarray):

Embedding vector representing the document.

cand_embs (numpy.ndarray):

2D array of candidate keyword embeddings with shape (n_candidates, embedding_dim).

candidates (list[str]):

Candidate keyword phrases corresponding to the embedding rows.

top_n (int, default = 10):

Number of keywords to select.

lambda_param (float, default = 0.7):

Trade-off parameter between relevance and diversity:

Values closer to 1 emphasize relevance

Values closer to 0 emphasize diversity

Returns:

list[str]

List of selected keyword phrases in selection order.

Examples:

doc_emb = np.array([1.0, 0.0])
cand_embs = np.array([
    [1.0, 0.0],
    [0.9, 0.1],
    [0.0, 1.0]
])
candidates = ["ai", "machine learning", "biology"]


mmr_select(doc_emb, cand_embs, candidates, top_n=2)
# ['ai', 'machine learning']

['ai', 'machine learning']

normalize_keyword()

Description:

Normalizes a keyword to a canonical representation using a lookup dictionary.

This function is useful for consolidating multiple surface forms (e.g., abbreviations or capitalization variants) into a single standardized keyword.

Parameters:

k (str):

Keyword to normalize.

normalize_map (dict[str, str]):

Dictionary mapping lowercase keyword variants to their canonical forms.

Returns:

str

Normalized keyword if a mapping exists; otherwise, the original keyword.

Examples:

normalize_map = {
    "ai": "artificial intelligence",
    "ml": "machine learning"
}


normalize_keyword("AI", normalize_map)
# 'artificial intelligence'
normalize_keyword("statistics", normalize_map)
# 'statistics'

'statistics'

filter_and_normalize_keywords()

Description:

Filters, normalizes, and formats a list of keywords into a final output structure.

This function performs the following steps:

Removes banned keywords using is_banned_candidate
Normalizes remaining keywords using normalize_keyword
Trims the list to a target size (top_n)
Pads the list with empty strings if fewer than top_n remain
Produces both a list representation and a semicolon-separated string

Parameters:

kws (list[str]):

Raw list of keyword strings.

banned_substrings (list[str]):

Substrings that should be excluded.

person_names (set[str]):

Person names to filter out.

normalize_map (dict[str, str]):

Dictionary mapping keyword variants to canonical forms.

top_n (int, default = 10):

Target number of keywords in the output.

Returns:

tuple

keywords_str (str): Semicolon-separated string of non-empty keywords

keywords_list (list[str]): List of keywords padded with empty strings to length top_n

Examples:

kws = ["AI", "John Doe", "ML", "Data Science"]
banned_substrings = []
person_names = {"john doe"}
normalize_map = {
    "ai": "artificial intelligence",
    "ml": "machine learning"
}


filter_and_normalize_keywords(
    kws,
    banned_substrings,
    person_names,
    normalize_map,
    top_n=3
)
# ('artificial intelligence; machine learning; Data Science',
#  ['artificial intelligence', 'machine learning', 'Data Science'])

('artificial intelligence; machine learning; Data Science',
 ['artificial intelligence', 'machine learning', 'Data Science'])

trim_candidates_by_frequency()

Description:

Trims a list of candidate keyword phrases based on their overall frequency in a corpus.

Candidate phrases are ranked by total frequency across all documents, and only the most frequent phrases are retained. This is commonly used to reduce the candidate set before embedding or MMR selection.

Parameters:

candidates (list[str]):

Candidate keyword phrases.

X (scipy.sparse matrix):

Document-term matrix (e.g., output of CountVectorizer). Columns correspond to candidates.

max_candidates (int):

Maximum number of candidates to retain.

Returns:

list[str]

Trimmed list of candidate phrases sorted by descending frequency.

Examples:

from scipy.sparse import csr_matrix


candidates = ["ai", "ml", "statistics"]
X = csr_matrix([
[3, 1, 0],
[2, 0, 1]
])


trim_candidates_by_frequency(candidates, X, max_candidates=2)
# ['ai', 'ml']

['ai', 'statistics']

safe_click()

Description:

Safely clicks a Selenium web element while handling common interaction failures.

This helper function attempts to click an element by:

Scrolling it into view
Performing a standard click
Falling back to a JavaScript-based click if necessary

It gracefully handles common Selenium exceptions such as intercepted clicks or stale references and returns a success flag instead of raising errors.

Parameters:

driver (selenium.webdriver):

Active Selenium WebDriver instance.

el (selenium.webdriver.remote.webelement.WebElement):

Element to be clicked.

sleep_after_click (float, default = 0.25):

Time (in seconds) to wait after clicking to allow the page to update.

Returns:

bool

Returns True if the click was successful, otherwise False.

click_all_more_buttons()

Description:

Clicks all visible “More” or “Show more” style controls on a page until none remain.

This function is designed for pages with dynamically expanding content. It:

Repeatedly searches for or

Skips elements that would navigate away from the page

Clicks valid expansion controls until no more are found

This is commonly used to fully expand truncated content before scraping HTML.

Parameters:

driver (selenium.webdriver):

Active Selenium WebDriver instance.

timeout (int, default = 10):

Maximum time to wait for page updates after clicking.

sleep_after_click (float, default = 0.25):

Pause after each click to allow content to load.

Returns:

bool

Returns True if at least one button was clicked, otherwise False.

expand_all_toggles()

Description:

Expands common in-page toggle controls such as accordions and collapsible sections.

This function targets typical Bootstrap and ARIA-based toggles and:

Skips already-expanded elements
Avoids real navigation links
Uses safe clicking to prevent Selenium errors

It is useful for revealing hidden content sections before extracting page HTML.

Parameters:

driver (selenium.webdriver):

Active Selenium WebDriver instance.

sleep_after_click (float, default = 0.25):

Pause after each toggle expansion.

Returns:

None

get_fully_expanded_html()

Description:

Retrieves fully expanded HTML content from a dynamic web page using Selenium.

This function:

Loads a page in a headless Chrome browser
Expands all “more” buttons and toggles
Captures the final rendered HTML
Optionally saves the HTML to disk

It is the primary entry point for scraping dynamically loaded govinfo pages.

Parameters:

url (str):

URL of the page to load.

headless (bool, default = True):

Whether to run Chrome in headless mode.

save_to (str | None):

Optional path to save the retrieved HTML.

wait (int, default = 10):

Time to wait for the page body to load.

Returns:

str

Fully expanded page HTML.

extract_hearing_links()

Description:

Extracts hearing detail and transcript links from a govinfo HTML page.

The function searches for hearing identifiers (CHRG-*) and returns normalized URLs for:

Hearing detail pages
Hearing transcript text pages

Duplicate hearings are automatically removed.

Parameters:

html_text (str):

HTML content from a govinfo page.

Returns:

pandas.DataFrame

DataFrame with columns:

details_url
text_url

get_session_from_url()

Description:

Extracts the congressional session number from a hearing URL.

For example, URLs containing CHRG-119… will return “119”.

Parameters:

url (str):

Hearing URL containing a CHRG identifier.

Returns:

str | None

Session number if found, otherwise None.

get_text()

Description:

Downloads and extracts transcript text from a govinfo text endpoint.

This function:

Fetches transcript pages using HTTP requests
Parses
```
blocks containing raw hearing text
```

Retries failed requests using exponential backoff

Parameters:

url (str):

Bracketed metadata
Page numbers and formatting artifacts
Speaker labels and headings
Excessive whitespace and separators

Optionally, it can remove ALL CAPS headings for cleaner narrative text.

Parameters:

raw (str):

Raw transcript text.

remove_all_caps_headings (bool, default = False):

Whether to remove ALL CAPS headings.

Returns:

str

Cleaned, readable transcript text.

get_date()

Description:

Extracts the first full date string from transcript text.

Dates are expected in the format: Month Day, Year

Parameters:

text (str):

Transcript text.

Returns:

str

Extracted date string, or “na” if none found.

extract_hearing_title()