Breaking News with Data Pipelines: How Airflow and AI Power Investigative Journalism

Investigative journalism often relies on uncovering hidden patterns in vast amounts of unstructured and semi-structured data. At the FT, we leverage Airflow to orchestrate AI-powered pipelines that transform complex, fragmented datasets into structured insights. Our Storyfinding team works closely with journalists to automate tedious data processing, enabling them to tell stories that might otherwise go untold.

This talk will explore how we use Airflow to process and analyze text, documents, and other difficult-to-structure data sources combining AI, machine learning, and advanced computational techniques to extract meaningful entities, relationships, and patterns. We’ll also showcase our connection analysis workflows, which link various datasets to reveal previously hidden chains of people and companies, a crucial capability for investigative reporting.

Attendees will learn:

• How Airflow can orchestrate AI-driven pipelines for handling unstructured and semi-structured data.
• Techniques for automating connection analysis to support investigative journalism.
• Lessons from our experience working with journalists to develop data-driven storytelling and storyfinding capabilities.