Top searches associated with each nation

In this post, we will get top searches associated with each nation. In doing so, first, we will scrape the list of world countries from wikipedia. We are including all member states of the United Nations. We will use the gtrendsR package to get the data from the Google Trends API.

Load all required packages

library(rvest)
library(dplyr)
library(calibrate)
library(stringi)
library(ggplot2)
library(maps)
library(ggmap)
library(stringr)
library(gtrendsR)
library(dplyr)

Scrape UN member states from Wikipedia

wiki= read_html("https://en.wikipedia.org/wiki/Member_states_of_the_United_Nations")

countries=wiki %>%
  html_nodes("table") %>%
    .[[2]]%>%
  html_table(fill=T)

Apply text extraction and regular expressions to remove unnecessary characters

countries[,1]=stri_sub(countries[,1],3)
countries[,1]= gsub("\\(.*)","",countries[,1])
countries[,1]= gsub("\\[.*]","",countries[,1])
countries= countries[,1]

How many countries are members of the UN?

length(countries)

193

So, there are 193 countries.

Let's see the first 10 countries

countries[1:10]

1. "Afghanistan"
2. "Albania"
3. "Algeria"
4. "Andorra"
5. "Angola"
6. "Antigua and Barbuda"
7. "Argentina"
8. "Armenia"
9. "Australia"
10. "Austria"

Getting Google Trends Data

We will use the gtrendsR package to get data from the Google Trends API. We need to give it our gmail user name and password.

gconnect(username, password)

Top searches for India

Let's first visualize the top searched terms associated with a single country. Let's take India.

data=gtrends('india')
z=as.data.frame(data$searches)
names(z)=c('searches','hits')
z=filter(z,hits>10)

 z$searches <- factor(z$searches, levels = z$searches[order(z$hits,decreasing =F)])  # re-order the terms

ggplot(z, aes(searches,hits))+ 
  geom_bar(stat='identity',fill="skyblue",color='black')+ylab('Hits')+
  theme(axis.title.y=element_blank(), axis.text.y = element_text(colour="black",size=10),
        axis.title.x = element_text(colour="blue",size=14),
        axis.text.x = element_text(colour="black",size=14,angle=90,hjust=.5,
                                   vjust=.5))+coord_flip()+
  ggtitle('Related top searches for India')+theme(plot.title = element_text(size = 14,colour="blue"))

Spatial Distribution of the Searches

We can also see the spatial distribution of the searches. We can make animations that show how the searches vary over space and time.

regions = as.data.frame(data$regions)

names(regions)=c('region','hits')

regions$region[regions$region=="United States"] = "USA"

world_map = map_data("world")

world_map =merge(world_map, regions, by="region",all.x = TRUE)

world_map = world_map[order(world_map$group, world_map$order),]

g=ggplot(world_map, aes(x=long, y=lat, group=group))+
  geom_polygon(aes(fill=hits), color="gray70") 

g+theme(axis.text.y   = element_blank(),
        axis.text.x   = element_blank(),
        axis.title.y  = element_blank(),
        axis.title.x  = element_blank(),
        panel.background = element_blank(),
        panel.grid.major = element_blank(), 
        panel.grid.minor = element_blank())+
scale_fill_gradient(low = "skyblue", high = "blue", guide = "colorbar",na.value="white")+
 theme(legend.key.size = unit(1, "cm"),
         legend.title = element_text(size = 12, colour = "blue"),
         legend.title.align=0.3,legend.text = element_text(size = 10))+
         theme(panel.border = element_rect(colour = "gray70", fill=NA, size=0.5))

Top searched terms associated with each nation

searches=c()

for(i in 1:dim(countries)[1]){
    data=gtrends(as.character(countries[i,1]))
    z=data$searches
    top_1=paste(as.data.frame(z)[,1][1],collapse=", ")
    searches=c(searches,top_1)
    
}

Data=as.data.frame(list(Country=countries, Top_searches=searches))

The data frame Data above cotains, the top most searched term associated with each country.

Top search terms by catagory

Now, let's group the countries based on the top most searched term.

Countries with top most search terms having "war" in them

Data[sapply(searches, function(x) "war"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
1	Afghanistan	afghanistan war
79	Iraq	iraq war
189	Vietnam	vietnam war

Countries with top most search terms referring to an airline

Data[sapply(searches, function(x) "air"%in%unlist(str_split(x,' '))| 
    "airways"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
13	Bahrain	bahrain air
32	Canada	air canada
89	Kuwait	kuwait airways
109	Mauritius	air mauritius
122	New Zealand	new zealand air
127	Oman	oman air
137	Qatar	qatar airways
151	Serbia	serbia air
186	Uzbekistan	uzbekistan airways

Countries with top most search terms having "hotel" in them

Data[sapply(searches, function(x) "hotel"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
4	Andorra	hotel andorra
106	Malta	hotel malta
142	Rwanda	hotel rwanda
147	San Marino	san marino hotel
156	Slovenia	hotel slovenia

Countries with top most search terms referring to thier past history

Data[sapply(searches, function(x) "ancient"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
52	Egypt	ancient egypt
66	Greece	ancient greece

Countries where weather is the top search

Data[sapply(searches, function(x) "weather"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
15	Barbados	barbados weather
43	Cyprus	weather cyprus

Getting the map of the country is most important

Data[sapply(searches, function(x) "map"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
6	Antigua and Barbuda	antigua map
20	Bhutan	bhutan map
82	Italy	italy map
86	Kazakhstan	kazakhstan map
90	Kyrgyzstan	map kyrgyzstan
115	Morocco	map morocco
131	Papua New Guinea	new guinea map
161	Spain	map spain

What about Africa?

Many people consider Africa as a single country. Others, on the other hand, when they search for information related to an African country, they add the word "Africa" to the country name. Let's see countries, whose top most search term is the name of the county plus Africa.

Data[sapply(searches, function(x) "africa"%in%unlist(str_split(x,' '))),]

	Country	Top_searches
23	Botswana	botswana africa
33	Central African Republic	central africa republic
48	Djibouti	africa djibouti
54	Equatorial Guinea	equatorial guinea africa
94	Lesotho	lesotho africa
108	Mauritania	mauritania africa
116	Mozambique	mozambique africa
118	Namibia	namibia africa
159	South Africa	india south africa
165	Swaziland	swaziland africa
183	Tanzania	tanzania africa
191	Zambia	zambia africa

As shown in the table above, the top search for many African countries is the country name plus Africa. Of course, central africa republic is a country and I do not know why "india south africa" is the top searched term associated with South Africa.

Summary

In this post, we searched what the top search associated with each country is. We scraped the list of the UN member states from wikipedia. Further, we classified the top searches based on the common terms they have. Countries linked with war are Iraq, Vietnam and Afghanistan. For many countries, the most searched term is an airline such as air Canada, qatar airways, etc. It is expected that Egypt and Greece would be searched most for thier interesting and intriguing history and our results show the same. A dozen of countries (Italy, Spain, Morocco, etc) are searched most for thier maps.