Saiba como fazer Web Scrape de especificações de produtos do Google Shopping com Node.js

Andrei Ogiolan on Feb 23 2023

In this article, we will explore how to use our SERP API with Node.js in order to web scrape product specifications from Google Shopping. We will cover everything from setting up the development environment to extracting relevant data and discussing potential issues. By the end of this article, you will have the knowledge and tools you need to scrape Google Shopping product specifications on your own. But before diving into the technical details, let's first understand what Google Shopping is.

Google Shopping is a platform that allows users to search for and compare products from various online retailers. It displays a wide range of products and their prices, making it a convenient tool for consumers to find the best deal. By using web scraping techniques, we can extract valuable data from Google Shopping, such as product specifications, to gain insights and make informed decisions.

Why should you use a professional scraper instead of building your own?

Many people are tempted to build their own scraping solution when it comes to web scraping. However, using a professional scraper is often a better option. Professional scrapers are built and maintained by experienced developers who understand the complexities of web scraping. They are designed to handle the challenges that come with scraping, such as CAPTCHAs, IP blocks, and website changes. They also come with built-in features like scheduling, data export, and error handling.

Outro aspeto importante é o facto de os fornecedores de scrapers profissionais cumprirem as políticas de recolha de dados dos sítios Web que recolhem e poderem fornecer uma utilização legal dos dados, o que é importante ter em conta quando se recolhem dados.

Using a professional scraper can save time, effort, and money. Moreover, Professional scrapers are equipped with a lot of functionalities that allow you to scrape at scale, schedule your scraping tasks, and even scrape behind a proxy and VPN to avoid IP blocks. In summary, using a professional scraper can save you time and effort, and provide you with more accurate and reliable data, making it a wise choice for any web scraping needs.

Finally, in our particular case, Google Shopping often changes its CSS classes which requires you to do a lot of manual updates which can become time consuming especially when you build a more complex scraper. Luckily, a professional scraper like ours can take care of this issue and you are not required to do these updates anymore in order to correctly receive the data back.

What are Google Shopping Product specs?

Google Shopping Product specs refer to the technical details and information about a product that is listed on Google Shopping. This includes details such as product name, price, brand, image, description, and more. These specs can provide valuable insights into a product's features and characteristics, which can be used to make informed decisions about purchasing or selling that product. For example, by comparing the specs of similar products, consumers can make an informed choice about which product is the best fit for their needs.

Scraping product specifications from Google Shopping can be useful for a variety of use cases. For example, e-commerce businesses can use this data to analyze the competition and make strategic decisions on pricing, product offerings, and marketing campaigns. Retailers can also use this information to identify gaps in their product line and make informed decisions on which products to stock. Additionally, researchers and analysts can use this data to study consumer behavior and market trends.

Como é que é o nosso objetivo?

Configuração

Before starting to scrape Google Shopping product specifications using our API, it's crucial to have the right tools set up. The main requirement is Node.js, a JavaScript runtime that allows the execution of JavaScript on the server side, which can be downloaded from their official website.

Additionally, an API KEY is needed, which can be obtained by creating an account here and activating the SERP service.

Depois de configurar o Node.js e obter uma API KEY, o próximo passo é criar um ficheiro de script Node.js. Isso pode ser feito executando o seguinte comando:

$ touch scraper.js 

E agora cole a seguinte linha no seu ficheiro:

consola.log("Hello World!")

E, em seguida, execute o seguinte comando:

$ node scraper.js

Se vir a mensagem "Hello World!" apresentada no terminal, significa que o Node.js foi instalado com êxito e que está pronto para avançar para a secção de raspagem propriamente dita.

Let’s start scraping Google Shopping Product Specs

With the environment set up, you are ready to begin scraping Google Shopping Product Specs using our API. This is a straightforward process and apart from what was discussed above, all you need to do is to get the product ID of the product you are interested in.

Sugestão: É assim que pode obter o ID do produto de um produto do Google Shopping:

Having now set up Node.js , an API key and a product ID ,you are now ready to start scraping. In order to get started now, create a js file, or use the one you created for the above section and import Node.js built in `https` module which enables you to send requests to our API. This can be done as follows:

const https = require("https");

Em segundo lugar, tem de especificar a sua chave API e a propriedade product_id do produto em que está interessado:

const API_KEY = "YOUR-API-KEY-HERE" // You can obtain one by registering here

const PRODUCT_ID = "11607214845071611155"

Next, you need to pass this information in an options object in order to let our API know what is product you are looking to scrape:

const options = {

 "method": "GET",

 "hostname": "serpapi.webscrapingapi.com",

 "port": null,

 "path": `/v1?engine=google_product&api_key=${API_KEY}&product_id=${PRODUCT_ID}`,

 "headers": {}

};

And lastly, you need to set up a call to our API with all this information:

const req = http.request(options, function (res) {

 const chunks = [];

 res.on("data", function (chunk) {

   chunks.push(chunk);

 });

 res.on("end", function () {

   const body = Buffer.concat(chunks);

   const results = JSON.parse(body.toString());

   const product_specs = results.specs_results;

   console.log(product_specs)

 });

});

req.end();

All that is left for you to do now is to execute the script you have created and wait for the results:

$ node scraper.js

E deve agora receber os seus resultados:

{

  display: {

    native_aspect_ratio: '16:9',

    screen_shape: 'Flat',

    led_backlighting_type: 'Direct-LED',

    display_technology: 'LCD',

    display_resolution: '1920 x 1080 pixels'

  }

}

And that's it! You have successfully scraped Google Shopping product specifications using our API, and you can now utilize the obtained data for various purposes such as price comparison, market research, SEO optimization, and more. For further reference and code samples in the other six programming languages, you can check out our Google Product API documentation.

Limitations of Google Product specification

One of the limitations of using Google Shopping product specifications is that the data is not always complete or up-to-date. Not all retailers and manufacturers may have their products listed on Google Shopping, or they may not have provided all of the necessary information. Additionally, some of the data may be inaccurate or out-of-date. This can lead to inconsistencies and inaccuracies in the scraped data, which can negatively impact the results of any analysis or research that is conducted using the data. Moreover, Google is constantly working on detecting and blocking scraping attempts, so the scraping process may fail or become more difficult over time, but using a professional scrape like ours can help you overcome this issue.

Conclusão

In conclusion, web scraping Google Shopping product specifications with Node.js can be a powerful tool for businesses and researchers looking to gain insight into the products and trends in a particular market. By using Node.js, an API key, and a product id it's possible to easily and quickly gather a large amount of data on product specifications from Google Shopping.