How to Collect Job Postings Data from Indeed Using Selenium



You’ve probably heard about Indeed, which is known as one of the most widely used job websites nowadays. So, if you are planning to scrape job sites, do not skip it. Indeed job posting sites are used in about 60 countries and provide data about job posts, hiring firms, and career pages from various countries. But what if you do not have any scraping tools but still need to use Indeed job scraping to get and analyze job-related data? Why not build a web scraper by yourself to collect data from Indeed web? If you have some coding skills, let’s try to scrape Indeed using Selenium. Let’s do it together!


Indeed job posting web page is a popular job aggregator where job seekers can find their dream job all over the world. It is a very convenient platform for recruiters as well, as it is free to post job advertisements, though there are some paid features as well, especially if you want to promote your job post. On top of this, Indeed enables users to get valuable insights about competing salaries and companies seeking the same candidates. Having such kind of data to create a competitive and attractive job ad is a decisive advantage. Screenshot from Indeed by DataOx

Why Scrape Indeed Job Posting

Do you know that job-related data stands out as one of the most required information? By Scraping you can get the most actual job data, analyze trends of the job market, investigate Indeed resume dataset, or even gather data about IT job listings with salaries based on location.

What are the benefits of Indeed job scraping

Check out how else businesses can benefit from extracting job data. They can: Benefits from scraping Indeed by DataOx
  • Track competitors’ job vacancies and benefits.
  • Collect data about the labor market.
  • Generate leads by offering services to companies that are looking for the same.
  • Keep job databases up-to-date.

What data can you get by scraping Indeed

Let’s find out what data you can extract by scraping Indeed, though this is a tight list.
  • Job postings
  • Job positions
  • Job descriptions
  • Job locations
  • Employee profiles
  • Company profiles
  • Ratings
  • Reviews
Scraping data from Indeed by DataOx

Indeed Job Scraping using Selenium: How to Start

Now, that you know how to take advantage by scraping Indeed, let’s get down to business. We’re going to use Selenium API, which is very handy and recommended particularly for web automation. Besides, it is simple to install using the following code line: Scraping Indeed using Selenium by DataOx 1

Importing Selenium

Before importing Selenium make sure you have a driver to interface along with the web browser required by Selenium. Drivers can be downloaded from here. Just note that it should be saved in the same directory as your browser app. Scraping Indeed using Selenium by DataOx 2

Navigating through Indeed

But how does indeed scrape jobs? To understand this, let’s start with navigation. The driver.get method is navigating to a page by using the given URL. Scraping Indeed using Selenium by DataOx 3 Once you run the above code, you can see a notification that your browser is being controlled. Screenshot from by DataOx 1

Performing a Search

When you are using Selenium, you can take advantage of identifying the required item or button by name, ID, or Xpath. Let’s make an advanced job search by specifying the needed search items and numbers of jobs displayed per page. We can see that “Advanced Job Search” is taken in a <a> tag from the HTML structure. We can use “contains” to identify the Xpath by text. Scraping Indeed using Selenium by DataOx 4 Then we need to add search values. Here is a piece of code where position, display number, and results by date are specified. Scraping Indeed using Selenium by DataOx 5

Extracting Job Card Data at Once

Let’s say that you would like to collect the complete information related to one job card
  • Position
  • Company name
  • Company rating
  • City
  • Salary
Then go through all the jobs on the current page, and move to the next page. Below is a code to loop to go through job cards on every page and extract relevant data. Scraping Indeed using Selenium by DataOx 6

Getting job descriptions from different URLs

There may be a case when you would like to get a job description from different URLs, then you need to use the following piece of code: Scraping Indeed using Selenium by DataOx 7 And to put them in a one data frame, add: Scraping Indeed using Selenium by DataOx 8

Common Methods to Extract Data from Indeed

But what to do if you have no coding skills? There are at least three common methods to get data from any web source on any scale:
  1. Buy a scraping tool.
  2. Hire a freelance web scraper developer.
  3. Outsource your scraping job to a professional team.
Common methods to scrape Indeed by DataOx

Indeed Web Scraper Questions and Answers

How to scrape Indeed?

Use Indeed API, which is free. You can scrap Indeed using Selenium WebDriver using our helpful guide.

How to pull resumes from Indeed?

The easiest way to pull your resume from Indeed is by downloading it from your account. Indeed allows PDF downloads. If you are an employer and want to download resumes of all potential employees, go to the jobs page, selector, and choose the needed candidates with checkboxes. After that, click on the Export button.

How does Indeed scrape jobs?

Indeed uses XML job feeds to scrape jobs. However, company websites must be eligible for Indeed guidelines to be listed.


So, let’s recap. Now you have some idea of how to scrape data from Indeed if you are ready to play with coding. But if you are not but still need to extract information from Indeed, you can always outsource this job to a web scraping company like DataOx. Schedule a free consultation with our expert to check the complete list of our web scraping services and learn how DataOx can help you scrape Indeed data according to your business goals.
Popular posts
surface web vs deep web vs dark web

Importance of Understanding the Differences Between Surface Web vs Deep Web vs Dark Web

Scrape Zillow: A Detailed Guide to Extracting Real Estate Listings with Python

Sports Betting Arbitrage – a Modern Way to Supplement Your Profits

Python PDF scraping

Python PDF Scraping – How to Extract PDF Files from Websites

Basics of web scraping DataOx's article

Web Scraping Basics, Challenges & Technologies for Startups and Entrepreneurs

Our site uses cookies and other technologies to tailor your experience and understand how you and other visitors use our site. Visit our Cookie Policy and our Privacy Policy for more information on our datd collection practices. By clicking Accept, you agree to our use of cookies for the purposes listed in our Cookie Policy.