Article content
Pavlo Zinkovski
4 min read
Even though everyone is talking about protecting the privacy of users, we almost don’t have one. Various services and apps have more access to our personal information than ever. And we can only wonder what they do with this data. Moreover, we tell the world a lot of stuff voluntarily on different social media platforms - we even add geotags to our posts to let our friends know where we are. How easy it is to violate our privacy? A piece of cake! Because we give away most of the data.
And also, there are browser fingerprints. They contain a lot of data about the user - here is another threat to our privacy. But additionally, these fingerprints are a threat to the success of the data gathering process. Why and how can they impact web scraping? Let’s figure this out.
Browser fingerprints: what are they?
Just like a real fingerprint of a human, one that belongs to a browser is unique. It exists because the destination server wants to identify the user. Such a fingerprint contains the information about the browser itself, operating system, plugins, languages, fonts, hardware - you name it. If you’re wondering what your fingerprint looks like, you can check it here.
This information might seem insignificant, but it shows that there is a real user visiting a website. Only one browser out of almost 300 others will have the same fingerprint as yours. So, as you can see, it’s quite unique. And that’s the issue fingerprints create for web scraping.
Main data in fingerprints
While many users can have the same operating system and fonts, some of the data that fingerprints contain is quite unique. That’s the information that allows servers to identify users. And this is exactly the data that will spoil your web scraping.
Most scrapers don’t send fingerprints, or the latter are empty. So you should set up your tool so that it mimics the real browser and sends some kind of a fingerprint to a destination server.
IP address
Of course, a fingerprint contains the IP address of a user. That’s the main piece of information that gives you away when you’re collecting data on the internet. So it’s quite logical that the first thing you should do is to change your IP.
How can you do that? With proxies, of course. Residential proxies are perfect for web scraping because they belong to real devices and, therefore, are hard to detect. Infatica owns an enormous pool of such IP addresses, so it won’t be an issue for you to rotate them properly and remain anonymous when you’re gathering the data.
Headers
Requests sent by a browser have a header, but the ones a scraper sends don’t. Most servers will suspect that they’re dealing with a bot if there will be no header data. By simply making sure that your scraper sends a header in the request, and that it contains the necessary information, you will minimize your chances of getting caught. You can find special libraries on the internet that will provide your scraper with headers.
User behavior
This is the hardest issue to overcome. Fingerprints pass the data about the activity a user had within one browser session. This can be information about the movement of a cursor, or the way the pages are scrolled. If this activity doesn’t look human-like - the destination server understands it’s dealing with a bot.
Robots don’t scan the page the way a human does. They quickly gather the information they need and leave the page. While this activity won’t get you blocked in most cases, it will quite likely make your scraper face a CAPTCHA.
There is no solid solution for this problem. You could use some virtual machine that would mimic the real browser environment, but that won’t make your scraper behave like a human. So all you can do is to use CAPTCHA-solving methods. There are Tesseract OCR tools that can detect the text on the CAPTCHA. Also, you can find instruments that involve humans to help scrapers with this issue.
Perhaps, in the nearest future developers will come up with scrapers that would mimic human behavior. But so far, there is no such solution, and fingerprints remain a real obstacle on the way to successful web scraping.
Tips that will help you avoid blocks
Data collection is a cat-and-mouse game at the moment. As data scientists come up with new ways to gather information from the web, owners of data sources create new anti-scraping techniques to keep crawlers away from their websites. It becomes significantly harder for bots to remain unnoticed and gather information from the Internet.
Of course, we could debate for a long time about who’s right and who’s wrong - after all, the Internet was supposed to be a free place - but it will be better if we focus on the solutions. In the future, we should expect advanced scrapers to emerge. These bots will need to have some footprint libraries they rotate to pretend to be a human. Also, they would have to behave like a real user by showing curves of a cursor and scrolling pages.
As for now, all we can do is to perfect our approach to web scraping. To do that we should follow the set of rules:
- Use residential proxies. They will cover up our authentic IP address and help us pretend to be someone else to keep the website’s guards down.
- Rotate proxies properly. Special tools for the proxy management will change IP addresses for the scraper frequently enough for the site to not suspect anything.
- Work on headers. The scraper should send some information in a header to keep a destination server assured that everything is good. Use libraries of headers to supply your scraper with this data.
- Keep pauses between requests. Even a 2-second pause will help you not spam the destination server so that the latter doesn’t notice suspicious behavior.
- Set up protocols. Make sure that protocol versions your scraper is using match headers it sends for requests to look realistic.
- Use CAPTCHA-solving tools. They will help your scraper deal with CAPTCHAS and avoid blocks.
