Scrapy

Scrapy is an open source programming library for web crawling and web page text extraction, written in Python. You can make calls to Scrapy code from within your own scripts and applications to automate the task of extracting data from websites.

You would typically use Scrapy to automate the task of visiting one or more web pages, on a website to which you have access. You could alternately use it to invoke web-based Application Programming Interfaces (APIs).

Scrapy can download everything it encounters or selected information. One of its capabilities is being able to extract structured data from web pages, such as one or more individual pieces of text from specific locations on the page, or all of the data in a table.