Web crawling of links from websites is used for the following purposes:
Before website content is scraped, all URLs need to be extracted. It also helps to make sure that no piece of content is missed.
Crawling of URLs can be used to analyze the cross-links and keywords of a particular website. There are a lot of ready-to-use tools for such tasks.
Even without scraping all of the content from a website, you can understand a lot about your competitors or market players by scraping URLs. For example, scraped and parsed website URLs can help you understand the number of goods, their names, and other parameters. It could be very helpful for quick market research or competitor analysis.