How to scrape Google Trends data without an API!

Google Trends is a great tool for market research, finding out what people are searching for can be super beneficial to you, so let’s go ahead and scrape some data from Google Trends!

Basic introduction you could probably skip that I copied from my other article

Let’s get started!

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

As I stated in my previous articles, we are using the Google Chrome browser as our GUI, but you can use other browsers within Selenium, if you’de like to use a different browser go for it! Make sure to have the specific browser installed on your machine.

Within Selenium we need to define our web browser, so let’s do so by using the following line of code:

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
driver = webdriver.Chrome(ChromeDriverManager().install())

I would recommend running all of the code you just typed out and see if a blank Google Chrome window opens up, if so, you’re doing great 👍 !

Next up, we want to get the URL of the specific Google Trends search page, in this case we’ll use the search term “cars”. let’s go ahead and make our way to Google Trends and search cars:

As you can see theres tons of data on this page, let’s say we wanted to take the most popular related search term on Google Trends:

In order to do this let’s make our way back to your Python script and copy and paste this code:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('COPY AND PASTE YOUR URL HERE')

We want to replace this code with the URL from our Google Trends search page:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

Next up, we want to get the HTML attribute of the top related query from the web browser. First off, research how to enable your developer settings from your web browser and enable those settings. Next, we want to right click on the actual top search query (as shown below), and click on inspect, you should get a pop up like this:

Next up copy and paste this code into your Python script:

POSTS = driver.find_element_by_xpath(‘DELETE THESE WORDS AND REPLACE WITH XPATH’).text

Now, let’s go back to the Google Chrome window and copy the xpath. To do this, on the inspector tab you have to right click on the line where the xpath came up, then right click > copy > copy full xpath

Next, go back to your Python script and paste this in between the 2 quotes:

POSTS = driver.find_element_by_xpath(‘/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span’).text
print(POSTS)

That’s it! We will now be able to see the the top related query for that search term printed in our console:

Here is the final code of this project:

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
driver = webdriver.Chrome(ChromeDriverManager().install())

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

POSTS = driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span').text
print(POSTS)

Awesome! You have just scraped some data from Google Trends without an API! This is a pretty basic tutorial, but some ways to build on this project would be: How could you get all of the related queries? Could you add the search term within the code? Maybe make a front end that asks for multiple words to be searched for? Those are ways to make this tool way more functional! Enjoy scraping data from Google Trends!

Data Scientist / Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store