Technical Analysis

Authors

Mikkel Christensen

Corbin Brinkerhoff

Technical Analysis

Executive Summary

This project analyzes U.S. Senate hearing transcripts from 2023-2025 (sessions 118-119) to understand legislative priorities and how they evolve in response to external events. We built a complete data pipeline that scrapes hearing data from govinfo.gov, extracts keywords using a custom KeyBERT-inspired approach with sentence transformers, and performs correlation analysis to reveal topic relationships.

Key Findings: Budget appropriations, agency oversight (EPA, FCC, DOD), and healthcare consistently dominated Senate attention. Temporal analysis revealed both predictable seasonal patterns (budget cycles) and event-driven spikes (natural disasters, international crises). Statistical analysis uncovered distinct thematic clusters—healthcare ecosystems, energy-environment nexus, and defense-security groupings—showing how policy domains interconnect in legislative oversight.

Technical Achievements: We developed reusable Python tools for congressional data analysis, including automated web scraping with Selenium, regex-based text extraction, custom keyword extraction optimized for commodity hardware, and an interactive Streamlit dashboard. Our approach successfully balanced analytical rigor with practical hardware constraints, making large-scale congressional text analysis accessible without requiring specialized computing resources.

Research question and motivation

We set out to answer a clear question: How do U.S. Senate hearing topics shift in response to outside events, which issues dominate the agenda, and which topics tend to travel together? Concretely, we wanted to (1) track which topics receive the most attention, (2) see how attention rises or falls with external events and the legislative calendar, and (3) uncover correlations and clusters of related topics across hearings.

Where did the data come from?

At first we were using a different website but it wouldn’t allow us to scrape from it so we had to pivot. We ended up using https://www.govinfo.gov/browse/committee as our source but not without issues.

Scraping difficulties

One issue we had is this webpage, once you clicked on a specific committee, used JAVA to render html after the page was loaded. Because of this we had to create functions that would use selenium expand all toggles and click buttons so that every drop down on the page would then render the html and allow us to scrape it (get_fully_expanded_html in documentation). This function saves the html into a seperate file to allow us to access it later without having to run the code everytime as it was time consuming.

Another issue we ran into was that after being completely renderd the html files were incredibly large and difficult to work with. We focused on scraping 2 links (a details like and a text link) and then pulling out necessary data from those to symplify that process. We did not end up needing the details link as we were able to get all our info from the text file. After we got those links we created a function that accessed each text link and downloaded the entire text (a very large .csv).

Data size compromise: We initially aimed to process as much historical data as possible, but our laptops couldn’t handle the massive file sizes. Even split into smaller chunks, the data processing crashed repeatedly. To maintain progress, we focused on the two most recent Senate sessions (119 and 118), representing approximately two years of data. This scope proved sufficient for our analysis while keeping hardware demands manageable. If we expand this project in the future, scaling to additional historical sessions is straightforward using our existing pipeline.

For a detailed step-by-step guide to the entire data collection process, including code examples and function explanations, see our Tutorial.

Cleaning the data

To clean the data we created a bunch of functions that cleaned this text chunk by removing formatting and used REGEX to pull dates and titles from the text to be used in our data set. We also spliced the date up into month, day and year to give us freedom more freedom for visualization.

For keyword extraction, we initially planned to use the official KeyBERT package, but after testing it on our laptops with thousands of documents, we found it was far too slow to be practical. Instead, we built our own lightweight KeyBERT-inspired implementation using sentence transformers and Maximal Marginal Relevance (MMR) selection. MMR is an algorithm that balances two competing objectives: (1) relevance to the document content and (2) diversity among selected keywords. This prevents redundant selections like including both “climate change” and “climate crisis” for the same hearing. Instead, diverse terms are chosen that better represent the full scope of discussion. Our custom implementation is significantly faster than KeyBERT on commodity hardware, maintains extraction quality through semantic embeddings, provides full control over filtering and normalization, and reduces dependencies by only requiring sentence-transformers and scikit-learn.

For a complete walkthrough of our keyword extraction implementation, including the MMR algorithm, filtering strategies, and code examples, see the keyword extraction section in our Tutorial.

After getting all the data we needed we deleted both links columns and the text to decrease the size of the files and ease the load on our computers. Below is the column structure we ended up with after cleaning and organizing our data.

session | date | month | day | year | title | keywords | kw_1 | kw_2 | kw_3 | kw_4 | kw_5 | kw_6 | kw_7 | kw_8 | kw_9 | kw_10

EDA and Visualization Process

With our cleaned dataset and extracted keywords in hand, we conducted exploratory data analysis to understand patterns in Senate hearing topics and their relationships. Our EDA process focused on three main areas: keyword frequency analysis, temporal trends, and keyword associations.

Data Preparation for Analysis

Before diving into visualizations, we applied the same filtering and normalization rules from our keyword extraction phase to ensure consistency:

  • Banned terms filtering: Removed remaining boilerplate congressional language that slipped through initial extraction
  • Keyword normalization: Standardized agency acronyms and common variants (e.g., “EPA” vs “epa”, “Veterans Affairs” vs “VA”)
  • Time window selection: Limited analysis to 2023 through present to focus on recent legislative priorities

This additional cleaning step was necessary because even with our careful KeyBERT-inspired extraction, some procedural terms and inconsistent capitalization remained in the data.

Keyword Frequency Analysis

Most common topics: We identified the top 20-25 keywords across all hearings to understand what issues dominate Senate discussions. This revealed key focus areas like budget priorities, specific agencies (EPA, FCC, DOD), and policy initiatives. Bar charts of keyword counts quickly showed which topics received the most attention from Senate committees.

Temporal trends: By aggregating keyword mentions by month, we tracked how specific topics wax and wane over time. For instance, budget-related keywords spike in predictable seasonal patterns, while crisis-related terms (like natural disasters or international events) appear suddenly and intensely. We visualized these patterns using line charts showing the top 10 keywords over time.

Keyword Co-occurrence and Association

Understanding which topics appear together in hearings reveals thematic clusters and policy connections:

Co-occurrence heatmaps: We built a matrix showing how often keyword pairs appear in the same hearing. This raw co-occurrence data highlights which topics are naturally linked. For example, “health care” frequently appears alongside “Medicare” and “Medicaid.”

Correlation analysis (Pearson): We computed Pearson correlation coefficients on binary keyword presence (1 if keyword appears in a hearing, 0 otherwise) across the top 10 keywords. This measures linear relationships between keyword occurrences:

  • Positive correlation: Keywords that tend to appear together
  • Near zero: Keywords that appear independently
  • Negative correlation: Keywords that rarely appear in the same hearings (distinct topic clusters)

However, Pearson correlation can be inflated by highly frequent keywords, so we also computed:

Normalized Pointwise Mutual Information (NPMI): NPMI measures how much more (or less) two keywords co-occur compared to what we’d expect by chance, normalized to a [-1, 1] scale:

  • NPMI near 1: Strong topical association beyond just frequency
  • NPMI near 0: Co-occurrence at chance levels
  • Negative NPMI: Keywords that avoid each other (distinct topics)

NPMI is more robust than Pearson for identifying truly meaningful topic relationships versus just frequent terms appearing everywhere.

Phi coefficient: We also calculated the phi coefficient (Pearson correlation specialized for binary variables) as a validation check, which matched our Pearson results as expected.

Visualization Approach

All analysis and visualizations were developed in the EDA.ipynb notebook using:

  • Pandas for data manipulation and aggregation
  • Matplotlib for basic plotting (bar charts, histograms)
  • Seaborn for heatmaps with color scales and annotations
  • Lower-triangle heatmaps to reduce redundancy (since correlation matrices are symmetric)

Key visualizations include:

  1. Top keyword counts bar chart - horizontal bar chart showing the 25 most frequent keywords
  2. Keyword trends over time - multi-line chart tracking monthly mentions of top 10 keywords
  3. Correlation heatmap - lower-triangle matrix with color-coded cells and numerical annotations
  4. NPMI heatmap - similar layout showing association strength beyond frequency effects
  5. Top keyword pairs tables - ranked lists of strongest associations by both metrics

Interactive Dashboard

To make these insights accessible to non-technical users, we built a Streamlit dashboard (app.py) that provides:

  • Data preview: Displays the full cleaned dataset with filtering options
  • Keyword search: Users can select any of the top 20 keywords and view monthly mention trends with options for exact or partial matching
  • Correlation explorer: Interactive versions of all three heatmaps (Pearson, NPMI, Phi) with explanatory text about interpretation
  • Top pairs tables: Automatically generated rankings of the strongest keyword associations

The dashboard replicates the EDA notebook’s filtering and normalization logic to ensure consistency, and uses the same statistical measures with added interpretive guidance for each visualization. Note that we did not include the raw co-occurrence heatmaps in the Streamlit app, as they were largely redundant with the correlation and NPMI visualizations that provide more interpretable metrics.

Key Findings

Our analysis revealed several compelling patterns in Senate hearing topics and their evolution over the 2023-2025 period:

Dominant Policy Domains: The most frequently discussed topics reflect the Senate’s core legislative responsibilities. Budget-related keywords (“president’s budget”, “appropriations”, “federal funding”) appeared most consistently across hearings, reflecting the annual budget cycle and ongoing appropriations discussions. Federal agencies like the EPA, FCC, DOD, and Veterans Affairs featured prominently, indicating sustained oversight of these departments. Healthcare-related terms (“health plans”, “Medicare”, “Medicaid”) also ranked among the top keywords, reflecting ongoing healthcare policy debates.

Temporal Patterns and Event-Driven Spikes: Keyword trends over time revealed both predictable cycles and reactive spikes. Budget-related discussions showed seasonal patterns, with mentions increasing during budget proposal and appropriations season (typically January-March and September-October). Conversely, crisis-driven topics appeared suddenly: natural disaster keywords spiked during hurricane season and wildfire events, while international relations keywords surged during geopolitical developments. Infrastructure-related terms (“bipartisan infrastructure”, “transportation”) maintained elevated levels following major infrastructure legislation, showing sustained implementation focus.

Thematic Clustering Through Association Analysis: NPMI analysis revealed meaningful topic clusters that go beyond simple co-occurrence:

  • Healthcare ecosystem: Strong positive associations between “health plans”, “Medicare”, “Medicaid”, and “FDA” indicate these topics are discussed together as part of comprehensive healthcare policy deliberations.
  • Energy and environment cluster: Keywords like “EPA”, “natural gas”, “climate” (if present), and energy-related terms showed high NPMI values, reflecting integrated discussions of environmental policy and energy infrastructure.
  • Defense and security nexus: Terms related to “DoD”, “homeland security”, and related security agencies appeared together more than expected by chance, indicating coordinated oversight of national security apparatus.
  • Economic policy grouping: Federal funding, appropriations, and specific agency budget discussions formed another coherent cluster, showing how budget discussions transcend individual departments.

Topic Separation and Independent Domains: Negative or near-zero correlations highlighted distinct policy silos. For example, healthcare policy discussions rarely overlapped with defense/security hearings, and agricultural/rural programs appeared independently from urban infrastructure topics. This separation suggests specialized committee jurisdictions and distinct legislative calendars for different policy areas.

Cross-Cutting Themes: Interestingly, certain keywords acted as bridges across multiple domains. “Federal funding” appeared alongside healthcare, infrastructure, defense, and environmental topics, reflecting its role as a universal lever in policy implementation. Similarly, “tribal programs” appeared across agriculture, healthcare, and energy discussions, showing sustained attention to Native American policy across multiple sectors.

Evolution of Legislative Priorities: Comparing earlier months (2023) to more recent hearings (2024-2025) revealed shifting priorities. While budget and agency oversight remained constant, newer concerns around emerging technologies, supply chain resilience, and workforce issues gained prominence over time. This evolution suggests the Senate adapting its oversight focus to emerging challenges while maintaining core institutional responsibilities.

Limitations and Context: These patterns reflect hearing topics, not necessarily legislative outcomes or the depth of discussion. High-frequency keywords may indicate either sustained important issues or routine procedural oversight. Additionally, our two-year window captures only recent Senate activity; longer-term historical data would be needed to identify multi-year trends or generational shifts in legislative priorities.

Addressing Our Motivating Questions: This analysis successfully addressed all three of our original research goals. First, we identified which topics dominate Senate attention through frequency analysis, revealing budget, agency oversight, and healthcare as persistent priorities. Second, we demonstrated how Senate discussions respond to external events through temporal analysis, showing crisis-driven spikes in disaster and international relations keywords alongside predictable seasonal budget patterns. Third, we found meaningful correlations between topics using NPMI and Pearson correlation, uncovering thematic clusters (healthcare ecosystem, energy-environment nexus, defense-security grouping) that reveal how policy domains interconnect in legislative oversight. Together, these findings established a quantitative foundation for understanding the Senate’s policy attention landscape, revealing both the stable institutional priorities and the dynamic responsiveness to current events that characterize legislative oversight.

Future Work and Recommendations

Expand Historical Coverage: Our current analysis spans only two years (2023-2025). Extending the dataset to include earlier congressional sessions (104-117) would enable longitudinal analysis of how legislative priorities shift over decades, revealing generational changes in policy focus and the impact of major historical events (9/11, 2008 financial crisis, COVID-19).

Enhanced Topic Modeling: While our keyword extraction successfully identified major themes, implementing more sophisticated NLP techniques like Latent Dirichlet Allocation (LDA) or BERTopic could uncover subtler thematic patterns and track how specific policy narratives evolve within broader topic categories.

Cross-Reference External Events: Systematically correlating keyword spikes with dated external events (natural disasters, legislation votes, international incidents) would provide quantitative validation of our observation that Senate discussions respond to real-world developments. This could involve building a timeline database and calculating temporal correlations.

Committee-Level Analysis: Our current analysis aggregates across all Senate committees. Disaggregating by committee (Finance, Armed Services, HELP, etc.) would reveal how different committees specialize in distinct policy domains and whether certain cross-cutting issues (like funding or technology) bridge committee boundaries.

Predictive Modeling: With sufficient historical data, machine learning models could potentially predict which topics are likely to receive increased attention based on current events, seasonal patterns, and political cycles—useful for policymakers, advocacy groups, and researchers.

Sentiment and Tone Analysis: Beyond topic identification, analyzing the sentiment and tone of hearing discussions could reveal whether certain issues receive more contentious debate versus consensus treatment, and how this changes over time.

Practical Recommendations:

  • For researchers: Our senate_hearings package provides a reusable foundation for congressional text analysis; extending it to House hearings or other legislative bodies would broaden its utility
  • For policymakers: The correlation analysis reveals which policy domains naturally interconnect, suggesting opportunities for integrated legislative approaches
  • For the public: The Streamlit dashboard demonstrates how complex legislative data can be made accessible; similar tools could increase civic engagement and government transparency