Web scraping is a very essential tool for programmers to learn to gather data from websites. Specifically, for Data Scientists web scraping is goto tool to gather data from websites. We can use bs4.BeautifulSoup or selenium in Python to scrape any website.

You can see some of my projects where I scraped websites like 99acres.com, flipkart.com, housing.com and gather useful data for my Data Science projects like arv-anshul/campusx-real-estate.

I have learned Web Scraping from YouTube only.

YouTube Videos

• Corey Schafer
  • Working with JSON Data using the json Module
  • Request Web Pages, Download Images, POST Data, Read JSON, and More
  • Web Scraping with BeautifulSoup and Requests
  • Web Scraping with Requests-HTML

YouTube Playlists

• John Watson Rooney

  • Modern Web Scraping with Python
  • Best Web Scraping Methods

• Indian Pythonista

  • Discovering Hidden APIs
  • Python for web

If you follow/learn these resources then you will understand how do Web Scraping works and how to do it.

My Python Package for Web Scraping

I have done lots of project on Web Scraping but while doing those web scraping projects I doesn’t found a good python package to handle/parse cURL command. But I found a package called @spulec/uncurl on GitHub but it is managed badly so that I cloned that project and refactor it well and republished as curler on PyPI.

• curler on PyPI
• curler on GitHub

My Projects Related to Web Scraping

• campusx-dsmp
• 99acres-scrape
• ecommerce-scrapper-api
• pw-api