Ayushi Rawat - The official blog of Ayushi Rawat

Web Scraping Coronavirus Data into MS Excel

UpdatedOctober 21, 2020

•7 min read•View as Markdown

Web Scraping Coronavirus Data into MS Excel

Ayushi Rawat

Software Engineer | Technical Blogger | YouTuber | Exploring and Refactoring! | connect with me on Twitter

Comments (28)

Join the discussion

Sara Cron3y ago

This is an incredibly useful resource for anyone looking to keep up to date on the ever-changing coronavirus situation. The ability to easily scrape the data into an Excel spreadsheet makes it much easier to track changes over time and compare the data from multiple sources. By: https://businessleadsworld.com/pricing

Ethan Steven3y ago

This is an awesome and helpful article for me. I appreciate your work providing such useful information; thank you so much! Discountagent

registrationkeys3y ago

TubeMate Downloader Crack YouTube Downloader is a handy tool that does just what its name suggests: downloads YouTube videos at a click of a button and is for the Android Operating System.

Mia Rose3y ago

Hi everyone, I am Mia Rose from London, UK.

Mia Rose3y ago

Hi everyone, I am Mia Rose from London, UK. I am here to offer the best assignment writing help to the students of the UK as well as in surrounding areas.

Nick Nicks4y ago

Very useful and well worded article.

Thomas Murphy4y ago

https://proctoredexamonline.com/ boost grades and provides reliable online exam help for students looking up professionals to take my online exam for me or do my online exam.

Adam Wong4y ago

All our work is ready from scratch at Nursing Assignments Help and will be delivered fresh to you. Due to the above, the value of our job writing service should not be stressed. Our talented Assignment Writer's analytical and problem-solving skills ensure that uniqueness is never compromised. Our clients are obligated to an endless number of additional adjustments for ultimate satisfaction. Our customer support team are 24/7 here to satisfy your need.

Thomas Snow4y ago

Take Online Exam Help Now offers proctor online exam help. Just say take my online exam help or do my online exam for me for your proctored online exam help.

James Brukz4y ago

Thanks, very useful! For my scraper I also had to use https://webscraping.ai/ to render JS content before parsing it.

StreaMaestro4y ago

A very brief and detailed article. StreaMaestro is a top online streaming platform that keeps you up to date on information that other competitors seem to find difficult to obtain.

Technical Tutorial - Python Projects

Part 3 of 36

The series content will mostly cover Technical Python tutorials and my recent learnings and experience in Python.

Up next

How to create your own captcha with python

We perform daily activities on the internet. You must often encounter CAPTCHA and reCAPTCHA tests on the Internet. CAPTCHAs are used by any website that wishes to restrict usage by bots. Do you wish to know how to create one? You can also follow alon...

More from this blog

Beginner's guide to Hacktoberfest 2021

All you need to know about Hacktoberfest 2021

Sep 30, 20216 min read

What is Hacktoberfest? Hacktoberfest 101

Hello reader! Well, Hacktoberfest is something, you are gonna hear a lot about in the coming month. You can refer to my YouTube video Tutorial to see a working tutorial for better understanding and a step-by-step guide of the same. https://www.yo...

Sep 27, 20214 min read

What is Hacktoberfest? Hacktoberfest 101

12 Python Tips & Tricks You must know

Hello reader! In this Blog post, I will share some useful code snippets and functions in Python. Give it a read. Let's get started! 1. Memory The method getsizeof can be used to retrieve the size of any object. Here's an example of the same. import...

Jul 25, 20214 min read

HTTP Status Codes that You must know

Hello reader! Whenever a client sends a request to the server, the response always contains a status code. You might not see it always but it's returned at every client-server interaction. Well, even if you are not a programmer, you would still know...

Jul 14, 20216 min read

GitHub Copilot: All you need to know

Hello reader! Microsoft subsidiary GitHub launched Copilot to power pair programming with AI. In this blog post, I will share all you need to know about it. You can refer to my YouTube video Tutorial to see a working tutorial for better understandin...

Jul 9, 20216 min read

Ayushi Rawat - The official blog of Ayushi Rawat

72 posts

Software Engineer | Technical Blogger | YouTuber | Exploring and Refactoring! | Connect with me on Twitter

You can refer to my video tutorial for the same at YouTube

https://www.youtube.com/watch?v=CTRYYz1u7Y8

Repository for Ultimate Resource in python. Drop a star if you find it useful! Got anything to add? Open a PR on the same!

Coronavirus cases are increasing rapidly worldwide. This tutorial will guide you on how to web scrape Coronavirus data and into Ms-excel.

What will be covered in this blog

Introduction to Web Scrapping
Understanding HTML basics
How to scrape a website
How to export the data into excel file

Pre Requisites

- python
- Beautiful soup
- pandas
- HTML
- CSS

Alt Text

What is Web Scrapping?

Introduction

If you’ve ever copy and pasted information from a website, you’ve performed the same function as any web scraper, only on a microscopic, manual scale.

Alt Text

Web scraping, also known as web data extraction, is the process of retrieving or “scraping” data from a website. This information is collected and then exported into a format that is more useful for the user. Be it a spreadsheet or an API.

Two important points to be taken into consideration here:

Always be respectful and try to get permission to scrape, do not bombard a website with scraping requests, otherwise, your IP address may get blocked!
Be aware that websites change often, meaning your code could go from working to totally broken from one day to the next.

The process: three simple steps

Request for a response from the webpage
Parse and extract with the help of Beautiful soup and lxml
Download and export the data with pandas into excel

Alt Text

Its Uses

It can serve several purposes, most popular ones are Investment Decision Making, Competitor Monitoring, News Monitoring, Market Trend Analysis, Appraising Property Value, Estimating Rental Yields, Politics and Campaigns and many more.

If you wish to know about it further. I am attaching the Wikipedia link here. You can have a look.

What is Coronavirus?

I do not think Coronavirus needs an introduction, but just in case if someone does not know, Coronavirus disease (COVID-19) is an infectious disease caused by a newly discovered coronavirus.

The COVID-19 virus spreads primarily through droplets of saliva or discharge from the nose when an infected person coughs or sneezes

If you wish to know about it further. I am attaching the Wikipedia link here. You can have a look.

The data source

We need a webpage to fetch the coronavirus data from. So I am using the Worldometer website here.

You can use the attached link to navigate to the website. You can also refer WHO website Worldometer's webpage will look something like this

Alt Text

We are interested in the data contained in a table at Worldometer’s website, where it lists all the countries together with their current reported coronavirus cases, new cases for the day, total deaths, new deaths for the day, etc

Alt Text

Now we are ready to start the coding.

Time to Code

You can find the code at my Github Repository

Alt Text

Let's Understand it

There are a few libraries you will need, so first, you need to install them. Go to your command line and install them.

pip install requests
pip install lxml
pip install bs4

Now let's see what we can do with these libraries.

Understanding HTML basics

This is the basic syntax of an HTML webpage. Every serves a block inside the webpage:

Alt Text

<!DOCTYPE html>: HTML documents must start with a type declaration.
The HTML document is contained between <html> and </html> .
The meta and script declaration of the HTML document is between <head> and </head> .
The visible part of the HTML document is between <body> and </body> tags.
Title headings are defined with the <h1> through <h6> tags.
Paragraphs are defined with the <p> tag.
Other useful tags include <a> for hyperlinks, <table> for tables, <tr> for table rows, and <td> for table columns.

requests

Use the requests library to grab the page.
This may fail if you have a firewall blocking Python/Jupyter.
Sometimes you need to run this twice if it fails the first time.

import requests

#make requests from webpage
result = requests.get('https://www.worldometers.info/coronavirus/country/india/')

print(result.text)

The request library that we downloaded goes and gets a response, to get a request from the webpage, we use requests.get(website URL) method.

If the request is successful, it will be stored as a giant python string. We will be able to fetch the complete webpage source code when we run result.text .

But the code will not be structured.

Beautiful soup

BeautifulSoup library already has lots of built-in tools and methods to grab information from a string of this nature (basically an HTML file). It is a Python library for pulling data out of HTML and XML files.

Using BeautifulSoup we can create a "soup" object that contains all the "ingredients" of the webpage.

Alt Text

If u wish to know more about BeautifulSoup, I am attaching the Documentation link here. You can have a look.

import bs4

soup = bs4.BeautifulSoup(result.text,'lxml')
print(soup)

And now import B.S. for the next step is to actually create the soup variable. And we're gonna pass on two things here, result.text string and lxml as a string.

Lxml goes through this HTML document and then figure out what is a CSS class, What is a CSS I.D. , Which are the different HTML elements and tags etc.

Extracting the data

Find the div To find the element you need to right-click and hit inspect on the number of cases. Refer the attached snapshot below.

Alt Text

we need to find the right class. class_= 'maincounter-number' serves our purpose. Refer the attached snapshot below.

Alt Text

The Beautiful Soup object has been created in our Python script and the HTML data of the website has been scraped off of the page. Next, we need to get the data that we are interested in, out of the HTML code.

cases = soup.find_all('div' ,class_= 'maincounter-number')
print(cases)

Alt Text

There is still a lot of HTML code that we do not want. Our desired data entries is wrapped in the HTML div element and inside class_= 'maincounter-number' . We can use this knowledge to further clean up the scraped data.

Storing the data

we need to save the scraped data in some form that can be used effectively. For this project, all the data will be saved in a Python List. To achieve this we will:

data = []

#find the span and get data from it
for i in cases:
    span = i.find('span')
    data.append(span.string)

print(data)

we will use span to fetch data from div.

Alt Text

We need the numbers, we do not want us to be dealing with the tags. So we will use span.string to get the numbers.

We will update the numbers into the data list

Alt Text

Now that we have the numbers we are ready to export our data into an excel file.

Exporting the data

Our last step is to export the data to Ms-excel. To fulfil the purpose, I am making use of Pandas

Alt Text

To load the pandas package and start working with it, import the package. The community agreed alias for pandas is pd, so loading pandas as pd is assumed standard practice.

If u wish to know more about Pandas, I am attaching the Documentation link here. You can have a look.

import pandas as pd

df = pd.DataFrame({"CoronaData": data})

#naming the coloumns
df.index = ['TotalCases', ' Deaths', 'Recovered']

#file-name
df.to_csv('Corona_Data.csv')

DataFrame is a 2-dimensional labelled data structure, potentially heterogeneous tabular data structure with labelled axes (rows and columns).

df = pd.DataFrame({"CoronaData": data}) is used to create a DataFrame and give it a name and map it to the data list that we created earlier.

Next, we will give column names with df.index . It will look something like this.

Alt Text

Final step, We are ready to export the data into Ms-excel. We will use df.to_csv for the same

Here's our result

Alt Text

NOTE: The output depends on the current statistics

You can find the code at my Github Repository

You can also connect with me on Twitter

I hope this helped you in understanding how to Web scrape coronavirus data into excel. Also, have a look at my other Blogs:

If you have any Queries or Suggestions, feel free to reach out to me in the Comments Section below.

Resources:

https://www.crummy.com/software/BeautifulSoup/bs4/doc/
https://pandas.pydata.org/docs/
https://en.wikipedia.org/wiki/Web_scraping
https://en.wikipedia.org/wiki/Coronavirus
https://www.worldometers.info/coronavirus/#countries

Command Palette

Comments (28)

Technical Tutorial - Python Projects

How to create your own captcha with python

More from this blog

What will be covered in this blog

Pre Requisites

What is Web Scrapping?

Introduction

The process: three simple steps

Its Uses

What is Coronavirus?

The data source

Time to Code

Understanding HTML basics

requests

Beautiful soup

Extracting the data

Storing the data

Exporting the data