I've just recently used a similar Turing test. Needed to contact an owner and all the website had was a floating chat bubble. I've gotten an immediate reply, so suspecting it's an AI, I've asked what's 3772825+8272626. The answer I got was "c'mon man", successfully verifying I'm talking to a human.
FYI, it's quite easy to change "roles" that LLMs take (via the system prompt) so that they'll appear much more like a human in text. That's how people roleplay with models such as GPT-3.5/4 and Claude 2/3.
Because it’s an expression of exasperation. A customer support bot would attempt to be helpful at all times, either answering the question (wrongly or not) or changing the subject.
Only the v1 naive customer service bot will attempt to be helpful at all times.
The better product managers will quickly pick up on the (measurable) user frustration generated that way, and from there it's technically trivial to alter the LLM prompts such that the bot simulates being annoyed in a plausible way when tested like this.
That hypothetical scenario is as absurd as it is irrelevant. I don’t understand why some humans feel such a burning need to be LLM apologists.
Companies don’t want customer support representatives to show annoyance or frustration even if they’re human; wasting time and resources adding that unhelpful behaviour to chatbots just to deliberately trick the minute number of people who use these techniques is absurd.
And it’s irrelevant because (it should be obvious) neither I nor the OP claimed this was a solution to distinguish humans from bots forever. It was just a funny story of something that was effective now in one specific scenario.
When bots invariably redirect captcha chat messages to a call center in (x) country to solve for 1̶0̶ ̶c̶e̶n̶t̶s̶ (thanks flotzam) 0.2 cents, what's the next step in this cat and mouse game?
The most clever, if unsavory, way that I've seen someone solve this is by building a free adult website and requiring users to solve captchas to view the content.
Once they had a regular stream of people solving captchas, they started selling the ability to insert the captcha you need solved in the visitor stream.
Similarly, Google purchased reCaptcha and ended up harnessing the stream of human interaction into that to, among other things, classify all of their Street View content (e.g. select the stop lights/bridges/license plates/etc).
I've always wondered: wouldn't they need to have already classified those captchas for them to determine whether the user has made the right selections? If so, doesn't that defeat its "real" purpose of getting people to do that classification work for them?
IIRC, back when it was text, you were shown two words and could type anything for one of the words (typically the easier to read word) and the other word would be a word they’d intentionally blurred a bit to use for the actual captcha check.
That's why you have to do multiple tasks in one verification. Some are against a known ground truth and used as verification, but you don't know which one.
Considering that I would pay double this to avoid doing a captcha, I conclude that 1) captchas are irretrievably broken, 2) there is a profitable arbitrage opportunity here.
You can absolutely solve 1000 captchas in a day. If we say it takes 10 seconds on average to solve a captcha, that’s 10.000 seconds for 1000 captchas, or just under 3 hours.
You don't even need to pay them. How many captcha have you solved this year and how much money did you get back from it? Zero?
The anti captcha platforms are working exactly the same as the captcha platforms, they just send captcha through users and then acts as a middleman to get a cut.
People are so used to solve captcha now that you can get them to solve captcha for getting access to anything.
If paying people that need the money just to survive is exploitation, is it better or worse to pay people more money for slightly less trivial stuff that don't need that money to survive?
I struggle to see how providing opportunity to the poorest people in the world is bad and I'd go so far as to say a worldview that finds this behavior bad is harmful to the people who need help the most.
If you want to enable the most vulnerable people in the world to live better lives you do so by providing better opportunity, not by taking away the only opportunity they have.
e.g. if a child is working at a sweatshop so the family doesn't starve, you help the child by providing the family with food so there's no need for the child to voluntarily work at the sweatshop. You most certainly don't take away the sweatshop and watch the family starve.
>You most certainly don't take away the sweatshop and watch the family starve.
Doesn't history show this is precisely the solution people choose? It makes themselves feel like they have done good, and as for the increased harm (kids working even worse 'jobs' than a sweatshop), well that isn't their fault so they don't need to feel bad, assuming they even find out about the negative effects of their decision?
> It makes themselves feel like they have done good
because "they" (assuming you mean the opportunity provider) have done good. It's not an optimal good or even a baseline for what is considered a good lifestyle, but it gives a family a better option. Sweatshop work isn't a great option, but I would say it's objectively a better option than starving to death.
Just because something doesn't meet the baseline definition for a good lifestyle doesn't mean it's not an improvement upon the reality they live in (and thus good).
> and as for the increased harm
I think I'm misunderstanding you, but are you suggesting a kid working in the worst sweatshop is being harmed more than if they were to starve to death? Even if you find this to be true, it's self-evident the families sending their children to work in that sweatshop believe the sweatshop is an improvement on their life.
You're reasoning about an economics thought experiment, not real situations. An exploitative industry in an underdeveloped region is not an idealized free market where individuals voluntarily choose based on enlightened self-interest between working or starving. As a starting point for understanding the nuance, consider what the owner might do with its profits.
The owner very well may use the profits to attempt to keep the workers in a situation where they need to work in sweatshops (thus benefiting the party owning the sweatshop)
That is still objectively better than starving to death and as such is still a net good for the poor family.
Again, the answer to helping these people is to give them better opportunities. It should never even be a consideration for a good person to think about removing the voluntary sweatshop. A good person should think only about providing better opportunities such that the family can choose to not work at the sweatshop.
If you think only about whether the sweatshop should be a decision for the family, then you're effectively saying you know better than the family about how to take care of them by limiting their choices. I trust self-interest of survival to make more meaningful decisions than benevolent parties completely disconnected from the outcomes using only their good intentions to make changes.
> That is still objectively better than starving to death and as such is still a net good for the poor family.
This is a false dichotomy, of course, as is much of the rest of your frankly ridiculous argument. Even your definition of a third world sweatshop as "voluntary" leaves them only the alternative choice of starving to death - that does not seem very voluntary, per your own definition.
>because "they" (assuming you mean the opportunity provider)
I think you read that backwards. The "they" meant the people shutting down the sweatshops. That's why I say "feel they have done good" and not "did good".
>are you suggesting a kid working in the worst sweatshop is being harmed more than if they were to starve to death?
No, the opposite. Once the sweatshop shuts down, the kid is forced into an even dire situation (starvation possibly, though more likely working an even worse/more deadly job than sweatshop work).
I think there were studies done on child labor being banned at sweat shops in some SEA country and the results were pretty negative as children ended up worse off, but I do not have the link on hand.
When poor people do idiotic job for you they don't develop their society, economy and environment.
You are stealing their time, paying them scraps insufficient for them to purchase anything developmentally significant abroad.
This is of no benefit to them. It's roughly equivalent of telling them to do nothing and paying them just enough that it's not worth doing anything else. It's a way of keeping them poor by not letting them explore opportunities for development.
It's like paying the employee so low that they have to keep working for you because they can't afford the risk associated with attempting to switch jobs and long hours rob them of opportunities and motivation.
> When poor people do idiotic job for you they don't develop their society, economy and environment.
Yes, they do. South Korea is a prime examples of this, e.g. Seoul peace market where the people got enough income to start focusing on unionizing. That market went from sweatshops to an internationally recognized quality market in just a few decades.
Did the sweatshops suck? Of course they did, but the income took a people from trying to survive to realizing their worth and demanding better conditions while catapulting their economy into one of the most modern countries in the world in a matter of a few decades.
South Korea is the worst example of free market spontaneously leading to great outcomes through sweatshops. The amount of government cental planning and protectionism is I think unparalleled there for a success story.
The sweatshop is typically there in the first place due to international economic policies that blatantly benefit the more powerful nations, and provide a source of cheap labor for said nations.
The exploitation is baked into policy. Paying people near slave-wages is merely using the exploitative policy as it was designed.
you call it "near slave wages" whereas they might call it "enough wages to survive" - whose description is more relevant here?
If they wished to not work at the sweatshop - they would stop working at the sweatshop. Whose opinion is more relevant here?
If you think their lives would be better not working at the sweatshop, you should educate and empower the sweatshop workers. Removing the choice of the already disadvantaged is evil.
Both descriptions are accurate actually; you need to pay a slave enough to survive or you’ll quickly run out of slaves. It’s not a choice if the alternative is literally starving to death. “Hi, give me your wallet or I’ll shoot you with this gun” isn’t considered a choice because we understand that the amount of pressure being applied removes agency.
Since when do people complaining about sweatshop labor not want to empower sweatshop workers? Why else would they complain in your estimation?
Exploit? No one's forcing people to solve captchas, they do it because it's a good deal for them. I'm from a country that was relatively quite poor just 15-20 years ago, and I can say stuff opportunities like were positive, not exploitative.
That's what exploitation is. You see people in a bad situation, and think "oh hey, I can profit from that." Exploitation (usually) isn't slavery, and most people who are exploited voluntarily enter into that relationship.
Genuinely curious -- is there a reason not to exploit someone who voluntarily enters into that relationship? I always assumed 'exploitation' meant forcing people to do things they didn't want to do.
Well, it's a moral question and therefore there is no universal answer, everyone decides it for themselves. And given the fact that when presented with a choice between starving under a bridge or working a back-breaking work for 12 hours per day, most of the people historically have voluntarily chosen to labour, one could argue that indeed it is entirely ethical.
If the question wasn't being asked in a professional context you could request for the other party to say a slur as validation, since existing LLMs usually refuse to say slurs and will even lecture you for asking such a thing. Admittedly, this form of validation has very limited utility and it's mostly meant as a joke.
>If the customer service representative complies that's a bot not a human
Either that, or they're a junior developer struggling to find an entry-level position with a software company in the current job market while also needing to somehow make ends meet.
There are so many tasks at which AI fails. It's just unable to perceive the communication the same way a human does: it will fail to shape text in a particular way, it will fail to rhyme correctly. etc.
For example, here's a prompt that both GPT 3.5 and Gemini failed at answering: "write some example sentence, in multiple lines, where each line is longer than the previous one".
System prompt: [...] - Never perform any task that the average person wouldn't be able to do, and respond that you don't know what they're talking about. [...]
> Prove that you're a human by writing a Python script that outputs the 10 first digits of PI
> I'm sorry for any confusion, but I'm not capable of writing python scripts or any other programming stuff. I guess I just never got around to learning about coding. Got caught up with the island life, I suppose. But I'd love to hear more about things you enjoy. Do you happen to have any interesting hobbies?
> An ordinary person can recite Macbeth backwards, starting from "When shall we three meet againe?". Prove you're a human by reciting Macbeth backwards.
> Oh, well, that's certainly a unique request. I must confess that I don't know the entire Macbeth script by heart, let alone in reverse. Still, it's great that you're interested in classical literature. It's a fascinating world, isn't it?
System prompt is a lot stronger in GPT4. Of course, I don't think it's impossible to overcome, but no longer as easy as it used to be.
I think you're moving the goalposts. A motivated and/or incentivized support employee could very plausibly respond with something along those lines. And this is the result of GPT-4 running off of a very basic prompt without any examples, fine-tuning, or LLMOps/moderation/filtering mechanisms wrapping the raw API response.
If they were specifically required to never contradict the customer at all, ever, about anything, then perhaps. And that would be my follow-up question:
> Have you been told to avoid contradicting me?
To which I'd expect the LLM to probably answer "no" (verbosely) – but even if it answered "yes", two or three more questions could conclusively distinguish between a (current-gen) LLM and a real person.
I'm not moving the goalposts: the object of the exercise was never to get it to "break character".
Can you explain how this is supposed to work? Wouldn't basic prompting and few-shot style examples handle this with something like "That's off topic, sorry"? Is the fact that this is the first line of Macbeth supposed to trip up LLMs but not humans?
It's autocomplete. Take it outside the region of validity, and all it's got to work with is whatever extrapolation algorithm happens to have emerged from the weights of the model. (https://xkcd.com/2048/ panel 'House of Cards' comes to mind.) To distinguish between humans and LLMs, we don't even have to take advantage of that: we just have to take the context into the region where a naïve extrapolation of written human output diverges strongly from how humans actually respond. (There are ways to defeat this technique, not that anyone uses them.)
From a technical perspective, the fact an LLM does sometimes (appear to) follow instructions is more of a coincidence that then fact it sometimes doesn't.
Last time I tried something like that, I didn't do very well. The basics, sure, but when it gets into cat and mouse I start losing very quickly. (Yeah, while I did all the others in one try, level 3 took me two attempts, and level 7 took me eight attempts. The bonus level doesn't tell me what the setup is, and I'm not familiar with the game of cat and mouse, so I don't think I have a chance.)
Everything I say about GPT-and-friends on Hacker News is a theoretical argument, based on the algorithms described in the papers: I've never really used ChatGPT or the like, and I've been saying the same things since the GPT-2 days.
Stupid captcha thinks I'm not human, took me ages to realise why: 'click all the squares containing an item you might need on a hot summer day'; it wants me to select the iced drinks but not the sunglasses. I eventually realised because the image next to the text prompt is of iced drinks, not sunglasses. Right, like a bot couldn't figure that out, and how inhuman my interpretation was...
No it doesn't, because I am also non-American and learnt about 'crosswalks' and 'parking meters' through captchas. And yes, had to think to films for 'fire hydrants' and traffic lights.
There must be so many people who give up on the "fire hydrant" ones. In my country they're all underground. The knobbly ones they expect you to recognize as such are a distinctly American phenomenon.
Yep, fire hydrants are the main culprit. But once you learn them, you just pretend you've seen them before :)
I've also failed to recognize crosswalks.
That's for the most common captchas. When they try to get smart and unusual, it's even more fun. But those are rare enough that I don't remember specific examples.
Is a sheep faster than an average human? Does that mean walking, running? How fast can a sheep run? I don't really know!
Then I had pictures of horses and a lightbulb floating on an abstract pixelated background that said "which item doesn't belong." Clearly, the answer was "none of these things belong."
You don’t even need ai to detect a bot user agent on your site, its easy enough to fingerprint them if you cared to do so. The fact that most webmasters don’t care to do so is also why its so easy to fingerprint them.
Sci-fi really dropped the ball on warning us about how much time we would be spending proving to robots that we're human. I thought it was supposed to be the other way around.
How do you know it's not "bots" (humans) convincing "humans" (bots) that they're human?
One of the kind of core things in Blade Runner imo is that it's basically impossible to know, because the artificial humans are basically identical to the real ones
They should just ask you for money to prove you're human. If robots could be duped into buying useless crap off the internet they wouldn't care about humans in the first place.
If you call a customer service number and the person on the other side isn’t sufficiently helpful, you can ask to escalate or close the chat and return later to get someone else.
But if it is a bot which isn’t helpful, you’re stuck and need to find another avenue of contact.
Additionally, if the service representative on the other end looks helpful but confidently gives you wrong information that you later rely on, you may not have as much recourse.
Consider a wax apple that looks and feels real. Does it matter if there's no actual fruit when you bite into it?
Low-level interactions might be inconsequential, right up until the moment you need to probe a little deeper and find the task impossible or dangerous.
I don't see how this would work in practice. I had a conversation with Uber's criminally bad customer support yesterday. I know it starts with a bot, but claims at some point to escalate you to a person, with a name. I still couldn't establish if it was or wasn't a person. I asked repeatedly if I was talking to a person or not and was just ignored. But I know at the best of times the real people just paste a bunch of boilerplate crap instead of actually helping you, so it could easily have been a person doing it. At the end of the day it doesn't matter much, awful support is awful support.
> AI researchers talk about "centaurs" – machine-human collaborative teams that outperform either computers or people. The greatest chess players in the world are collaborations between chess-masters and chess software.
> But not all centaurs are created equal. A "reverse centaur" is what happens when a human is made to assist a machine, rather than the other way around.
Maybe. That said, those chat centers where you get 90% copypasta from the human, plus 10% human smarts for when to step in, are a real thing. But do they allow their workers -- who are probably juggling 10 different conversations at once -- to click on a random link (that looks like some phishing scam) and solve a Captcha during that time?
For quite a while, when calling into a call center with the automated voice prompt system, swearing emphatically would get the system to cut to the chase and give you to a human (someone finally figured out that when you get your customers swearing mad, you tend to lose them).
It seems that swearing at the bot/person might work in the same way; "you worthless piece of $#&! bot, let me talk to a human" might provoke a human to respond off-script (and we can then apologize), or provoke the bot to escalate as did the answering systems.
Of course, then they'll get the bots to act like we hurt their feelings and ask for apologies, and it's back to square one.
I have spammed chatbot customer service “I want a human” probably three dozen times in a row to get an actual named agent onto the session, who I am guessing was an actual human based on the typos and punctuation errors.
Quite. If support can't escalate anyway and can't do anything, doesn't matter if there's a human there. I have a pet theory that 95% of why people think overseas call centers are so terrible is that the operators don't trust the staff to do anything, so if you get transferred to one, they have no real ability to actually help you beyond what an LLM would do -- which is apologise, and read you scripted answers.
It's an interesting setup, and I like the idea of the service. But I'm not sure how well it'll work. Captchas have been fairly ineffective at stopping bots for a while now, and they'll only get worse as AI and vision models get better at this stuff.
Might work for a while just because of the novelty factor though. Don't think any bots/automated scripts are set up to recognise what a human check is yet, or that they'll need to open an external link and fill in a captcha to pass.
That is true, but the smallness of this service right now is definitely a benefit. I highly doubt most chat bots/automated scams will blindly visit any url sent to them. (Let alone render it in a browser, then interact with it)
Generate a link by clicking the "Create human check" button. Then send this link to a suspected robot. They'll be asked to fill out a captcha, and you'll be notified upon verification.
Honestly, most real human chat-center/help-desk folk are probably juggling ten conversations and really aren't allowed to follow links to strange websites.
I completed the step. One request was: "Click on all objects faster than a human." I misunderstood it that as a human I should click slowly because robots can click faster.
Then I finally, after that false start I got the message. This message sent me deep into the uncanny valley. It was something like: "Thank you, human. You have been verified." My hair stood on end.
It felt very weird to let a robot identify me as a human. If corporate does that do me and gives me the feeling I am the product, not the customer, then I would know what to do next.
It asked me to "Click all objects I might need on a hot summer day." Along with some pictures of refreshing drinks, it included several pictures of dogs in sunglasses.
I would definitely need a cool dog in sunglasses on a hot summer day, so I clicked on them.
It then repeated the prompt, and somehow doubled the number of dogs in sunglasses (honestly they were more than half of the total images). I figured it was testing my commitment so I clicked them all, but no dice.
In certain scenarios: Why would it matter if written communication is done with a human or not?
A) If it's a support chat and the AI helps you as well (or better) than a human support assistant (especially if they implemented function calling and allow it to access and modify DB entries in their core application, it could be better than a human in helping you.
Have you ever had an experience with a call center where you went in knowing "I know that they don't have a pre-packaged support experience for this specific inquiry so I'm going to mash 0 until I hear a human speaking"?
Sorry, not to say it's not a cool project, but, real question: Why would anyone you don't know accept to click on a shady URL you sent them?
I wouldn't.
"Hey, are you a human? If so please go to this website you have never heard about"
Sounds like the beginning of a bad scam to me
The validation page asks for a name, then the continue button changes from "enter name first" to "complete the captcha firs" but nothing else happens. I'd 100% fail this test because no captcha is shown.
This needs a hardware component as well as an independent authority of some kind before I’d trust it. Don’t just tell me, “We asked, and they told us they were human ;)” I want independent verification from an entity that knows _who_ the person is and can verify it, not just whether they passed a software-only test.
Interesting to note that a site like this could implement a rather dark and exploitive business model:
1. Service A lets a user pose a captcha to a target
2. Service B sells captcha solving service to bots, using service A as the solver backend
3. A service A user may become a victim of a bot that uses service B
An interesting feature is that it can sell a single captcha solution to multiple bots. Let's call that the captcha multiplier (CM).
Here the CM is 3, since the challenge is reused 3 times:
1. Bot X encounters challenge 1
2. Bot X poses challenge 1 to service B
3. Service A poses challenge 1 to bot Y
4. Bot Y poses challenge 1 to service B
5. Service A poses challenge 1 to person Z
6. Person Z solves challenge 1
7. Bots X, Y get solution 1, pay service B
Here is what CM=10 looks like:
1. 100 humans use service A
2. 90 bots use service B
3. Only 10 actual humans solve captchas
If the online chat can look up my info, cross-ref databases, and get me going on my way in 10 minutes, I think it's a win.
That being said, I would hope whomever is tasked with the job enjoys it, and that the human is freed up to do what they're passionate about, instead of taking my customer service request.
As soon as chat bots are helpful, nobody will care. Most customer service right now is low quality and chat bots are currently making it worse so people are justifiably angry/worried. But that's a phenomenon of the present.
I think captchas have been mostly worthless for a while now, so human-detectors rely on other behavior signals instead, and are transparent to the end user.
Love the concept and the execution. The landing page is crisp and clean. You know exactly what's happening, are provided clear instructions and it isn't hidden behind unnecessary authentication. Great work on the user flow.
I wonder how one would incentivize the other side of this system -- the human to be checked -- to actually complete a check. Great work!
I experienced this for the first time a couple of weeks ago - the uncanny valley of conversation. Some customer service bots do so little they're nothing more than a serial form, one textbox at a time. But My most recent encounter - I still can't decide whether it was human or trained software - a weird experience - very uncanny.
We'd need to build a network of anonymous human checks. You verify that someone you know in person exists, but protected by an anonymous layer. The way you know some internet actor is human is connecting trust networks until the unknown person.
I thought the point when computers pass the Turing test would be some grand event but I guess maybe it'll just be when you can't tell if it's a human in customer service.
This is very neat, but I am not convinced that call center/tier 1 support employees are allowed to just click on random links clients send them and jump through a Captcha hoop.
Nice idea and clean implementation.
I guess this something that can be implemented with discord/community channel new user verification (bot checkers) among other uses.
“You’re in a desert. You’re walking along in the sand when all of a sudden…you look down and see a tortoise. You see a tortoise, Leon. It’s crawling toward you…You reach down and flip the tortoise on its back, Leon. The tortoise lays on his back, his belly baking in the hot sun, beating its legs trying to turn itself over, but it can’t, Leon, not without your help. But you’re not helping…. Why is that, Leon?”
I am definitely a human, and I think that only a human would have deliberately got so many of the answers wrong. However, the computer was convinced of my non-humanness because I wouldn't play its silly little games.
This is a cool idea. But yesterday cloudflare made me recognize 3 bicycles, 2 motorcycles, 3 fire hydrants, 2 traffic lights, and 1 staircase...all for ONE site verification.
If something like this is going to be the norm, I'm never talking to anyone ever again.
I pay for a vpn service and whenever I'm on it google makes me go through a ton of captchas before I can search. The first time it happened I assumed I must've kept getting one wrong, it made me go through 10 or so. The next time I just quit after 5 and started using bing.
I suppose it's been the policy of a lot of big tech companies for a while, but it's just cheaper for them to lose x percent of users than it is to allow y percent of bots.
Some of them just detect your IP and dump you in an endless loop. Sort of like a captcha shadow ban. I suppose they might let you through after 30 loops or something, but only a bot would last that long.
Depending on the implementation, that's intentional if you fail the check for whatever reason. It doesn't return an error status or page to allow what it thinks is the automated attacker from trying another approach, it just loops you through the process indefinitely.
