An Empirical Study & Evaluation of Modern CAPTCHAs
Developed 20 years ago to prevent hackers from stealing content, inserting malicious posts, engaging in fraudulent transactions, or slowing down websites to the point of complete inoperability, the acronym of this ubiquitous defensive line clearly defines its mission: Completely Automated Public Turing test to tell Computers and Humans Apart – CAPTCHA.
For almost two decades, CAPTCHAs have been widely used as a means of protection against bots. Over time, as their usage spread, techniques for bypassing or deceiving CAPTCHAs continued to improve. However, CAPTCHAs have also evolved in terms of complexity and diversity, becoming increasingly challenging to solve for both bots (machines) and humans. Given this long-term and still relevant technological competition, it is critically important to investigate how much time legitimate users require to solve modern CAPTCHAs and how they are perceived by these users.
Today, CAPTCHAs still remain one of the primary concerns for users.
However, researchers from the University of California, Irvine, have concluded that bots seem to solve them better than humans.
In their work, scientists study CAPTCHAs in natural conditions, assessing users' performance in solving them and their perception of the currently used CAPTCHAs. They collect these data through manual checks of popular websites and user studies in which 1 400 participants collectively solved 14 000 CAPTCHAs. The results show significant differences between the most popular types of CAPTCHAs: surprisingly, the time it takes to solve and users' perception do not always correlate. A comparative study was conducted to examine the impact of the experimental context – specifically, the difference between solving CAPTCHAs directly and solving them as part of a more natural task, such as creating an account. Despite several potentially confounding factors, the results indicated that the experimental context can influence this task and should be considered in future CAPTCHA research. The researchers also explore users' refusal to complete tasks caused by CAPTCHAs, analyzing participants who start a task and do not finish it.
They found that bots not only solve various forms of CAPTCHAs, such as image recognition, slider puzzles, and distorted text, better but also faster.
As CAPTCHAs evolve in terms of complexity and diversity, they become increasingly difficult to solve for both bots (machines) and humans. However, with the development of computer vision and machine learning, bots' abilities in recognizing distorted text have significantly increased, reaching an accuracy of over 99%. Bots can successfully overcome CAPTCHAs with distorted text in almost 100% of cases. Human accuracy in solving CAPTCHAs varies from 50% to 84%. Furthermore, it takes humans up to 15 seconds to solve these tasks, while bots can do it in less than a second.
Based on this study, researchers have come to an obvious conclusion: there is no longer a simple way, based on small images or other features, to definitively distinguish humans from bots. Instead, they recommend using artificial intelligence advancements to develop "intelligent algorithms" that can more effectively differentiate bot actions from human actions.
You can read the full research paper at the following link.