Data Gathering Issues: How to Deal with CAPTCHAs?

CAPTCHA is a powerful tool for distinguishing between human and bot traffic. How does it work — and is it possible to circumvent it? Let's find out.

Data Gathering Issues: How to Deal with CAPTCHAs?
John Garfield
John Garfield 8 min read
Article content
  1. What is CAPTCHA?
  2. CAPTCHA implementations
  3. CAPTCHA types
  4. How to avoid CAPTCHAs with proxies?
  5. How To Remove ReCAPTCHAs With AI?
  6. Frequently Asked Questions

CAPTCHAs are annoying: They make us doubt our ability to read symbols or find a bus on the picture every time they ask us to do that. Trying to catch all the robots, CAPTCHAs turned into full-blown riddles we often have to bend our mind over to solve. Yet, while it’s still possible for us, humans, to deal with CAPTCHAs, bots have difficulties cracking them. And it becomes a significant issue when we’re trying to gather data from the internet.

That’s where artificial intelligence and machine learning come to the rescue. We can add handy plugins to our scrapers to teach them to recognize and solve CAPTCHAs. In this article, we will learn about different CAPTCHA types, variations, and how to stop reCAPTCHA.

What is CAPTCHA?

Here's what the word "CAPTCHA" stands for:

Completely Automated Public Turing Test to Tell Computers and Humans Apart

And you thought the abbreviation is too long, huh. The purpose of this tool is obvious — it’s meant to help web services distinguish between humans and robots. The roots of CAPTCHAs go back to the 1950s when Alan Turing created the Turing Test that was supposed to determine if computers had the same thought patterns as humans.

The test was quite straightforward: An interrogator asked two participants questions without knowing who is who. One of the participants was a human, and the other one — a computer. An interrogator had to play a game of guess trying to determine who is the robot. If they couldn’t, the computer was considered a winner.

Communication between the test participants

Later, developers created CAPTCHAs based on this test by Alan Turing. This tool became necessary because people started using bots for harmful purposes — to perform a DDoS attack or send spam, for example. Or to buy tickets for a hot show in bulk to resell them at a higher price.

In most cases, CAPTCHAs look like distorted letters and numbers that are somewhat easy to read for a human but impossible to understand for a robot. Also, there are tests that make us find certain things in photos. But we will talk about the kinds of CAPTCHAs a bit later.

The logic of this tool is simple: Humans can generalize things. This means that, if we know that we can sit in the chair, we don’t get confused about what to do with a couch when we see it. We know we can sit on it, too. Computers, however, struggle with this. Another human ability CAPTCHAs use is that we can detect patterns where they don’t really exist. For example, we can see familiar shapes in fairly shapeless things such as clouds and large spots — but robots are unable to do that.

CAPTCHA implementations

The technology behind CAPTCHAs is constantly improved by CAPTCHA providers, which adapt it to local markets, platforms, and services. Here are the most popular ones:

Google reCAPTCHA

reCAPTCHA logo

This is a service provided by Google that uses advanced risk analysis techniques to distinguish humans from bots. It offers three versions: reCAPTCHA v2 (which displays a checkbox or an image grid), reCAPTCHA v3 (which returns a score based on the visitor's behavior), and reCAPTCHA Enterprise (which provides enhanced security and customization options).

hCAPTCHA

hCAPTCHA logo

This is a service similar to reCAPTCHA that also uses image-based challenges, but with a twist: it pays website owners for using it and uses the data collected to train machine learning models. It also claims to respect user privacy and offer better bot detection than reCAPTCHA.

FunCaptcha

FunCaptcha logo

This is a service that uses interactive mini-games, such as rotating an animal or matching shapes, to verify human users. It aims to provide a fun and engaging experience for users while preventing bots from cheating.

Confident CAPTCHA

Confident CAPTCHA logo

This is a service that uses 3D images that users have to rotate and identify. It claims to be more secure and user-friendly than traditional CAPTCHAs, as well as accessible for people with disabilities.

Sweet CAPTCHA

Sweet CAPTCHA logo

This is a service that uses cute graphics and simple puzzles, such as dragging an item to a matching slot or connecting two dots, to verify human users. It claims to be more enjoyable and creative than other CAPTCHAs, as well as customizable for different themes and languages.

CAPTCHA types

There are many different types of CAPTCHAs, each with its own advantages and disadvantages. Some of the most common ones are:

Image-based CAPTCHAs

CAPTCHA asks to select certain objects

These CAPTCHAs display a grid of images and ask the visitor to click on the ones that contain a certain object, such as a car, a traffic sign, or an animal. They are more user-friendly and harder to break than text-based ones, but they require more bandwidth and processing power.

Text-based CAPTCHAs

An example of a CAPTCHA image

These CAPTCHAs display a bunch of distorted letters and numbers and ask the visitor to type them in correctly. They are easy to implement and widely used, but they can be hard to read for some people and easy to break for some bots.

Audio-based CAPTCHAs

These CAPTCHAs play a sound clip and ask the visitor to type what they hear. They are useful for people who have visual impairments or low-quality screens, but they can be noisy and annoying for some users and vulnerable to speech recognition software.

Invisible CAPTCHAs

These CAPTCHAs do not require any user interaction, but instead analyze the behavior and characteristics of the visitor, such as mouse movements, browser settings, or IP address. They are seamless and convenient for users, but they may raise privacy concerns and false positives.

Biometric CAPTCHAs

These CAPTCHAs use biometric data, such as fingerprints, face recognition, or voice recognition, to verify the identity of the visitor. They are very secure and accurate, but they also raise privacy and ethical issues and may not be compatible with all devices.

Checkbox CAPTCHAs

CAPTCHA asks to check the box

This must be our favorite one: When a CAPTCHA asks us to just check a box to prove we are humans, it’s an easy thing to do. Well, for us, humans. Robots can’t pass this test. But why? It’s so simple!

Well, the real test doesn’t boil down to just reading a text and ticking a box. It’s in the way we move our mouse. A human never can move a cursor creating a straight line because our hands are always a bit shaky. Robots can’t mimic this pattern. In addition, these CAPTCHAs might take a look at HTTP cookies that become available for a destination server as you try to enter a website.

How does reCAPTCHA understand that it needs an additional test? So if this tool became so smart, why do we still need to solve CAPTCHAs sometimes? It’s unclear what exactly triggers it, but the potential causes are:

  • Cookies,
  • Browser history, and/or
  • Mouse movements.

How to avoid CAPTCHAs with proxies?

Proxies are intermediaries that act as a bridge between your device and the website you want to access. They hide your real IP address and location and assign you a different one from their pool of IPs. This way, you can access websites that are blocked or restricted in your region, or mask your identity and activity from the website owners.

Proxy CAPTCHAs can also help you bypass or avoid CAPTCHAs, which can be annoying and time-consuming, especially if you need to access or scrape multiple websites or pages. They can also block your access if they detect suspicious or bot-like behavior from your IP address. There are two main ways that CAPTCHA solver proxy can help you:

Rotating proxies: These are proxies that change your IP address for every request or after a certain period of time. They make it harder for websites to track and flag your activity, as you appear as a different user each time. Rotating proxies can help you avoid triggering CAPTCHAs in the first place, or bypass them if they are not tied to a specific session or cookie.

CAPTCHA proxies (e.g. reCAPTCHA  proxies): They specialize in solving CAPTCHAs for you. They use various methods, such as human workers, artificial intelligence, or third-party services, to automatically fill in the CAPTCHA forms for you. A CAPTCHA proxy can help you bypass CAPTCHAs quickly and efficiently, without interrupting your workflow or compromising your data quality.

How to remove reCAPTCHAs with AI?

First of all, we should note that modern CAPTCHAs are based on artificial intelligence themselves. They learn correct answers by analyzing user responses. So machine learning is at the core of all those riddles.

That’s why it’s rather ironic yet logical that the very same artificial intelligence is used to let bots solve CAPTCHAs. Image recognition helps robots see the same patterns we do. Therefore, they become able to solve CAPTCHAs without much effort.

Does it mean you can bypass CAPTCHAs?

Yes, by adding an image recognition tool to your scraper you can avoid it getting blocked by CAPTCHAs and enjoy smooth data gathering. But remember that it’s not just the puzzles. We’ve mentioned that reCAPTCHAs detect movement patterns and cookies to understand if they deal with a bot or a human. That’s why there are different libraries that let you provide a scraper with data needed to mimic a real request. Also, modern scrapers can at least somehow mimic the behavior of a real user.

Yet, if you let your scraper run without proxies, even all the fancy add-ons won’t do much because all requests will come from the same IP address. And that’s a clear indication of robot activity. To address this problem, use residential Infatica proxies for effortless data gathering to automate IP switching and make sure websites don’t become suspicious towards your scraper.

👷‍♂️ Further reading: How Residential Proxies Simplify Data Gathering for Price Aggregators

Frequently Asked Questions

To stop CAPTCHAs, you need to stop triggering them: CAPTCHAs are typically fired when the anti-bot system, reCAPTCHA, considers your activity suspcious – for example, when your current IP address is observed to be shared with other users.

The best way of bypassing CAPTCHAs is residential proxies, which provide exclusive access to IP addresses. Technically, proxies don't bypass CAPTCHAs – rather, they prevent them from appearing in the first place.

Yes: Web scraping enthusiasts have created specialized software which use image recognition libraries to solve CAPTCHAs; online services employ human workers to solve CAPTCHAs, paying $0.50 for 1-2 hours of work.

Google's reCAPTCHA is a more advanced CAPTCHA system: In addition to distorted images, it uses advanced risk analysis techniques, considering the user’s entire engagement with the CAPTCHA, and evaluates a broad range of cues that distinguish humans from bots.

You can also learn more about:

Proxy service demand doubled as businesses recognize the importance of cybersecurity due to new SEC regulations
Proxies and business
Proxy service demand doubled as businesses recognize the importance of cybersecurity due to new SEC regulations

How did new SEC regulations influence proxy service demand? Which search terms are rising in popularity on search engines? Read the latest Infatica's report to learn more.

What Are Proxies for Bots? Everything You Want to Know
Proxies and business
What Are Proxies for Bots? Everything You Want to Know

Learn why proxies are essential for botting sneakers and other online products. Find out the best types of proxies for bots and where to get them.

How Many Proxies Per Task Do I Need?
Proxies and business
How Many Proxies Per Task Do I Need?

Discover how many proxies you need to perform data collection, SEO monitoring, sneaker copping, ad verification, and more.

Get In Touch
Have a question about Infatica? Get in touch with our experts to learn how we can help.