Web scraping with R using rvest: List of countries by proven oil reserves

In this post, let's see how to scrape information from the web using the rvest R package. Specifically, we will scrape the "List of countries by proven oil reserves" table from wikipedia.

library(rvest)
library(dplyr)
library(calibrate)
library(stringi)

wiki= read_html("https://en.wikipedia.org/wiki/List_of_countries_by_proven_oil_reserves")

oil=wiki %>%
  html_nodes("table") %>%
  .[[1]]%>%
  html_table()

oil[1:10,]

		Country	Reserves (MMbbl)
1	—	 OPEC	1,112,448 - 1,199,707
2	1	 Venezuela (see: Oil reserves in Venezuela)	297,740[2]
3	2	 Saudi Arabia (see: Oil reserves in Saudi Arabia)	268,350[2]
4	3	 Canada (see: Oil reserves in Canada)	173,625 - 175,200
5	4	 Iran (see: Oil reserves in Iran)	157,300[3]
6	5	 Iraq (see: Oil reserves in Iraq)	140,300[3]
7	6	 Kuwait (see: Oil reserves in Kuwait)	104,000[2]
8	7	 UAE (see: Oil reserves in the United Arab Emirates)	97,800
9	8	 Russia (see: Oil reserves in Russia)	80,000[2]
10	9	 Libya (see: Oil reserves in Libya)	48,014

The table above has some minor problems and let's fix those problems step by step. Let's remove the second column.

oil=oil[,2:3]

Renaming columns:

names(oil)=c("country","reserves")

Let's remove OPEC and World data, to just work with data on country level.

oil[1:10,]
oil[103,]

	country	reserves
1	 OPEC	1,112,448 - 1,199,707
2	 Venezuela (see: Oil reserves in Venezuela)	297,740[2]
3	 Saudi Arabia (see: Oil reserves in Saudi Arabia)	268,350[2]
4	 Canada (see: Oil reserves in Canada)	173,625 - 175,200
5	 Iran (see: Oil reserves in Iran)	157,300[3]
6	 Iraq (see: Oil reserves in Iraq)	140,300[3]
7	 Kuwait (see: Oil reserves in Kuwait)	104,000[2]
8	 UAE (see: Oil reserves in the United Arab Emirates)	97,800
9	 Russia (see: Oil reserves in Russia)	80,000[2]
10	 Libya (see: Oil reserves in Libya)	48,014

	country	reserves
103	Total World (2011)[11]	1,481,526

oil =slice(oil,2:102)
oil[1:10,]

	country	reserves
1	 Venezuela (see: Oil reserves in Venezuela)	297,740[2]
2	 Saudi Arabia (see: Oil reserves in Saudi Arabia)	268,350[2]
3	 Canada (see: Oil reserves in Canada)	173,625 - 175,200
4	 Iran (see: Oil reserves in Iran)	157,300[3]
5	 Iraq (see: Oil reserves in Iraq)	140,300[3]
6	 Kuwait (see: Oil reserves in Kuwait)	104,000[2]
7	 UAE (see: Oil reserves in the United Arab Emirates)	97,800
8	 Russia (see: Oil reserves in Russia)	80,000[2]
9	 Libya (see: Oil reserves in Libya)	48,014
10	 Nigeria (see: Oil reserves in Nigeria)	37,200

Now, let's apply some regular expressions to make the data tidy.

We just want country names and the corresponding oil reserves.

oil[,1] = gsub("\\((.*)","",oil[,1])

oil[1:10,]

	country	reserves
1	 Venezuela	297,740[2]
2	 Saudi Arabia	268,350[2]
3	 Canada	173,625 - 175,200
4	 Iran	157,300[3]
5	 Iraq	140,300[3]
6	 Kuwait	104,000[2]
7	 UAE	97,800
8	 Russia	80,000[2]
9	 Libya	48,014
10	 Nigeria	37,200

Now, we can remove the first character in the coutry names using the stringi package.

oil[,1]=stri_sub(oil[,1],3)

oil[1:10,]

	country	reserves
1	Venezuela	297,740[2]
2	Saudi Arabia	268,350[2]
3	Canada	173,625 - 175,200
4	Iran	157,300[3]
5	Iraq	140,300[3]
6	Kuwait	104,000[2]
7	UAE	97,800
8	Russia	80,000[2]
9	Libya	48,014
10	Nigeria	37,200

Now, let's work with the "reserves" column and remove the square brackets and the commas.

oil[,2] = gsub("\\[(.*)","",oil[,2])  # removing square brackets
oil[1:10,]

	country	reserves
1	Venezuela	297740
2	Saudi Arabia	268350
3	Canada	173625
4	Iran	157300
5	Iraq	140300
6	Kuwait	104000
7	UAE	97800
8	Russia	80000
9	Libya	48014
10	Nigeria	37200

oil[,2] = gsub(",","",oil[,2]) # removing commas

oil[1:10,]

	country	reserves
1	Venezuela	297740
2	Saudi Arabia	268350
3	Canada	173625 - 175200
4	Iran	157300
5	Iraq	140300
6	Kuwait	104000
7	UAE	97800
8	Russia	80000
9	Libya	48014
10	Nigeria	37200

In the 'reserves' column where the values are ranges, we can, for simplicity, take the first value.

oil[,2] = gsub("-(.*)","",oil[,2])

oil

	country	reserves
1	Venezuela	297740
2	Saudi Arabia	268350
3	Canada	173625
4	Iran	157300
5	Iraq	140300
6	Kuwait	104000
7	UAE	97800
8	Russia	80000
9	Libya	48014
10	Nigeria	37200
11	United States	36420
12	Kazakhstan	30002
13	China	25585
14	Qatar	25382
15	Brazil	13986
16	Angola	10470
17	Mexico	10364
18	Algeria	9940
19	Azerbaijan	7000
20	Ecuador	7000
21	Norway	6900
22	United Kingdom	6900
23	European Union	6700
24	Malaysia	5800
25	India	5650
26	Oman	5500
27	Ghana	5000
28	Egypt	4500
29	Vietnam	4400
30	Australia	4158
31	Indonesia	3990
32	Gabon	3700
33	Yemen	3000
34	Sudan	2800
35	Syria	2500
36	Mongolia	2493
37	Colombia	2377
38	Congo, Republic of the	1940
39	Equatorial Guinea	1705
40	Chad	1500
41	Peru	1240
42	Brunei	1200
43	Uganda	1000
44	Denmark	900
45	Trinidad and Tobago	830
46	Romania	650
47	Turkmenistan	600
48	Uzbekistan	594
49	East Timor	554
50	Argentina	2330
51	Thailand	442
52	Tunisia	425
53	Italy	400
54	Ukraine	395
55	Pakistan	313
56	Netherlands	310
57	Germany	276
58	Turkey	262
59	Cameroon	200
60	Bolivia	200
61	Albania	199
62	Belarus	198
63	Congo, Democratic Republic of the	180
64	Cuba	124
65	Papua New Guinea	170
66	Philippines	168
67	New Zealand	166
68	Chile	150
69	Spain	150
70	Bahrain	125
71	France	101
72	Ivory Coast	100
73	Mauritania	100
74	Poland	96
75	Austria	89
76	Guatemala	83
77	Afghanistan	80
78	Suriname	79
79	Serbia	77
80	Croatia	66
81	Burma	50
82	Japan	44
83	Kyrgyzstan	40
84	Georgia	35
85	Hungary	26
86	Bangladesh	28
87	Bulgaria	15
88	South Africa	15
89	Czech Republic	15
90	Lithuania	12
91	Tajikistan	12
92	Greece	10
93	Slovakia	9
94	Benin	8
95	Belize	7
96	Taiwan	2
97	Israel	2
98	Barbados	2
99	Jordan	1
100	Morocco	0.7
101	Ethiopia	0.4

Now, we can work with this data frame. We can produce world map that shows oil reserves using ggplot2 or do other types of analysis.

One can refer to my previous post to see how to produce global map similar to what is shown below

Wikipedia