Introduction
Ever felt like you are drowning in data scattered across the web? Web scraping can help. It is a powerful way to extract information from web pages and turn it into actionable insights. In this guide, we will walk through the core idea behind building a web scraper with Python.
The Challenge
Imagine gathering job postings from a large job board. Manually copying each listing is slow, repetitive, and difficult to scale. Web scraping automates that process by collecting the fields you care about, such as job title, location, and link, then organizing them into a reusable dataset.
The Tools
This example uses three practical Python libraries:
Building the Web Scraper
The example below shows the basic structure of a scraper: prepare a list, request each page, parse the response, find listing cards, extract useful fields, and save the result.
import requests
from bs4 import BeautifulSoup
import pandas as pd
jobs = []
for i in range(1, 10):
url = (
"https://www.linkedin.com/jobs/search/"
"?keywords=computer%20science"
f"&start={i}"
)
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
job_listings = soup.find_all(
"div",
class_="base-card relative w-full hover:no-underline "
"focus:no-underline base-card--link "
"base-search-card base-search-card--link job-search-card",
)
for job_listing in job_listings:
title_element = job_listing.find("h3", class_="base-search-card__title")
location_element = job_listing.find("span", class_="job-search-card__location")
link_element = job_listing.find("a", class_="base-card__full-link")
if title_element and location_element and link_element:
title = title_element.text.strip()
location = location_element.text.strip()
link = link_element["href"]
jobs.append([title, location, link])
df = pd.DataFrame(jobs, columns=["Job Title", "Location", "Link"])
df.to_csv("jobs.csv", index=False)
print(jobs)Example Output
After the scraper runs, the records can be shaped into a table like this:
Job Title Location Link
Software Engineer, AI/ML London, United Kingdom https://www.linkedin.com/jobs/view/...
Software Engineer, AI/ML London, United Kingdom https://www.linkedin.com/jobs/view/...
Senior Machine Learning Engineer (Remote-USA) United States https://www.linkedin.com/jobs/view/...Decoding the Code
- Imports: The script starts by importing the libraries needed for fetching pages, parsing HTML, and structuring data.
jobs = []: An empty list stores each extracted job record before it becomes a table.for i in range(...): The loop moves through multiple result pages so the scraper can collect more than one page of data.url = f"... ": The URL is constructed for each page. This structure is specific to the target website and can change over time.requests.get(url): Python fetches the page HTML.BeautifulSoup(...): The HTML is parsed so the scraper can search for elements by tag and class name.find_all(...): The scraper locates repeated job-card containers on the page.find(...): Each card is searched for the title, location, and job link.DataFrameandto_csv: The extracted data is converted into a table and saved as a CSV file.
Important Notes for the LinkedIn Example
Large platforms often load content with JavaScript. A simple requests scraper may not capture every listing. For dynamic pages, browser automation tools such as Selenium or Playwright may be needed because they can render JavaScript before extraction.
The search URL used in an example is tied to a specific query and website structure. When websites update their HTML or routing, scrapers often need maintenance.
Ethics and Reliability
Web scraping is useful, but it should be done carefully. Always check a website's terms, respect robots.txt where applicable, avoid excessive requests, and do not collect private or protected information. A good scraper is not only functional; it is respectful, transparent, and maintainable.
Summary
Web scraping can save time and create structured datasets from scattered public information. The advantages of automation and efficiency are significant, but real-world scraping also brings challenges: site changes, dynamic content, ethical boundaries, and increasing complexity. The example above is a solid starting point, but production-grade scraping requires careful planning and responsible practice.