How to Extract Product Data from Alibaba.com Using Open Source Framework Scrapy

3 Reasons to Scrape Alibaba.com

Data extracted from e-commerce websites is a potential help to businesses that are in e-commerce and not only. Keep reading to learn three main reasons why you need to scrape data from Alibaba.

How to Create an Alibaba Crawler

Written in Python, Scrapy is one of the most efficient free frameworks for web scraping that enables the users to extract, manage, and store information in a structured data format. It is perfectly adapted for web crawlers extracting details from various pages. Let’s move forward to learn how to scrape data from the leading marketplace.

Getting started

To create an Alibaba crawler you need to have Python 3 and PIP. Follow the links to download them:

To install the necessary packages, the following command is used:

Creating Alibaba Scrapy project

The next step is to create a Scrapy project for Alibaba with the scrapy_alibaba folder name containing all necessary files. The command is the following:

Creating the crawler

There is a built-in command in Scrapy called genspider that is responsible for generating the primary crawling template.

To generate our crawler that will create spiders/scrapy_alibaba.py file it should be:

The complete code should look like:

Extracting Product Data from Alibaba

In this example, we’re going to extract the following fields for the earphones: https://www.alibaba.com/trade/search?fsb=y&IndexArea=product_en&CatId=&SearchText=earphones&viewtype=G

To extract the required data from Alibaba we’re going through the following 3 steps:

1. Create a Selectorlib library
2. Create a keyword file
3. Export data in the required format

Creating a Selectorlib pattern for Alibaba

Selectorlib is a Chrome extension enabling users to point out the required data and create CSS Selectors or XPaths to extract that data. To learn more about Selectorlib go to the following link https://selectorlib.com/getting-started.html

Below you may find how we point out the fields in the code for the required data we need to extract from Alibaba using Selectorlib

When you marked all the required data, click on the Export button to download the YAML file and save it as search_results.yml in the folder named /resources

Reading keywords

Now we’re going to set up the Alibaba crawler to read specific keywords from a certain file placed in the folder /resources. Let’s create there a CSV file named keywords.csv and use Python’s CSV module to read our keywords file.

Exporting data into CSV or JSON

With Scrapy you can have in-built JSON and CSV formats. To save the extracted data in the desired format just use the appropriate command line

The output will be saved in the same folder as the script. Here is an example of extracted data in CSV

The full code for our Alibaba crawler is given below: (code source: https://github.com/scrapehero/alibaba-scraper/blob/master/scrapy_alibaba/spiders/alibaba_crawler.py)