How to hide (almost) perfectly selenium to avoid Captcha and master web scraping.

Introduction

Mastering web scraping will open a lot of gates to you. That is why I learned, worked, tried, failed (many and many times!), and finally, succeeded! To avoid losing time, I’m writing this tutorial to help you achieve your web scraping dreams. Sometimes, when you want to perform your web scraping script, you can be interrupted by a captcha. Selenium can help you to avoid it by simulating a browser. But you have to configure it correctly if you want it to use all its capabilities. This article intends to people exhausted by captcha (like I was!). We will not go further into scraping, because I think there is already a lot of wonderful articles about it.

Disclaimer

Be careful, if you perform web scraping, do not use the recovered data for commercial purposes without asking the website owner. Follow robots.txt and be respectful!

This tutorial intends for people who already know Selenium and want to go further on it.

Prerequisites

• Python
• Google chrome
• Chromedriver (available here)

Here is the classic Selenium’s call, with the result below: