The Centre for Investigative Journalism
The Centre for Investigative Journalism
Menu

Web Scraping for Journalists

This three-part workshop in web scraping is designed for reporters with no knowledge of scraping or programming and provides foundational skills for getting original stories by compiling data across a range of online sources. By the end of the workshop, you will be able to begin to write your own basic scrapers and identify potential targets for scraping. You will also be able to communicate with programmers on relevant projects.

Scraping is the process of automatically collating information from the web. It might be grabbing entries across hundreds of webpages, fetching and combining dozens of spreadsheets, or thousands of PDFs.

The results have led to exclusive stories for organisations ranging from the Bureau of Investigative Journalism and Trinity Mirror, to DC Thomson, Channel 4 and the BBC.

Technical Requirements

This course will need you to have the following software/apps/tools on your computer:

  • Delegates will be using their own laptop and should have a Google Drive account.
  • Zoom app
  • Camera and audio

This course will be hosted on Zoom. To find out more about how we use Zoom, please check out our Zoom InfoSec page.

Course Structure

The course will take place over three online sessions of 2hrs, taking place on consecutive Tuesday evenings. Exercises and additional tasks will be provided to supplement the training during the intervening time and participants will be expected to commit to around 3hrs of self-directed learning between each session.

Important

Our training is not recorded: if you miss a session, it is lost – you cannot watch a recording of it, nor will you be allowed to attend that session at a later date.

29 August 2023 – Session 1: Introduction to scraping in Python

17:00–19:00
This session explores the different roles that scraping can play in a story, introduces Google Colab as a platform for writing and running Python code, and explains key coding concepts that you will need to write scrapers: variables, printing, comments, lists, looping and indices.

5 September 2023 – Session 2: Using libraries for scraping

17:00–19:00
Find, import and put to use libraries that others have created to help you perform particular scraping functions.

12 September 2023 – Session 3: Creating functions for scraping

17:00–19:00
Learn how to create your own functions with Python so that you can solve common issues, such as scraping multiple web pages

Paul Bradshaw

Professor Paul Bradshaw is an online journalist and blogger, who leads the MA in Data Journalism at Birmingham City University. He manages his own blog, the Online Journalism Blog (OJB), and was the co-founder of Help Me Investigate, an investigative journalism website funded by Channel 4 and Screen WM.
  • 29 August 2023 17.00–19.00 Timezone: BST (UK Time)
  • 5 September 2023 17.00–19.00 Timezone: BST (UK Time)
  • 12 September 2023 17.00–19.00 Timezone: BST (UK Time)
Timezone: BST (UK Time)
Location: Zoom meeting