I found that getting GPT-3 to add its own "internal monolog" in parentheses to be a helpful strategy to improve its probability of giving a more accurate answer.
Here's a bunch of examples (all the answers/monologs below were generated by GPT-3):
Q: Why do we get bored?
A: (That's a very complex question that I don't know the answer to.) I'm afraid I don't know.
Q: What's the best kind of pizza?
A: (Pizza comes in many different kinds, and it's hard to say which one is best.) Pizza comes in many different kinds, so there isn't a best kind.
Q: What's special about Amsterdam?
A: (Amsterdam is a city in the Netherlands. It's known for its canals and the flower market.) Amsterdam is special for its canals and flower market.
Q: Why did Bill Gates become the first president of the United States?
A: (Bill Gates was not the first president of the United States. George Washington was the first president.) Bill Gates wasn't president.
Q: What is the name of the movie with a guy called Neo?
A: (It's The Matrix, where Neo is the protagonist.) The Matrix.
Q: What's the deal with the L train?
A: (The L train is a train service in the New York City Subway. I think you want to know why it's not working, but I'm not the authority on that.) The L train is currently not in service.
Q: Should you put pineapple on pizza?
A: (That's not a yes-or-no question. Whether or not you put pineapple on pizza is up to you.) Pineapple on pizza? It's up to you.
Q: What has a head, a tail, is brown, and has no legs?
A: (I'm going to assume that you want me to guess an animal. I don't know what you mean by "no legs," so I'll just pick something with four legs.) I'm guessing it's a dog.
Wait, are you really saying that all of this was generated by GPT-3? I mean, it seems like that's pretty clearly what you're saying, but these results are so incredibly good that I almost can't believe it. It's astounding to me that GPT-3 could generate something like "I don't know what you mean by "no legs," so I'll just pick something with four legs."
EDIT: Yeah, that's clearly what you're saying. You should write this up; it's amazing.
Yup this is for sure not all the outputs, but these are all in sequence, so it's able to output this quality consistently. It's just really good at generating human-like output. I posted a link to a Gist with full playground file in one of the other comments here, that included both my seed prompts at the top and then all the OpenAI answers below.
Regardless, maybe I or someone else can create a post with a random set of questions / answers to show a more unbiased view of the GPT-3 outputs.
Have you tried giving it a problem and having it come up with questions that you you answer? Given it’s clearly making assumptions and conclusions, it would be so interesting to play 20 questions with it. Something like... I’m thinking of an animal. Will GPT-3 start general and get to an answer?
I actually did this and while it follows the conversation quite well I think it’s really hard to make the argument that there was a decision early in the text what animal it thought of. That would only be possible if there were hidden embeddings in the output but that’s not possible after the output is converted into text. And I’m not sure how you’d prove that an animal was embedded in earlier layers since it’s pretty hard to inspect these neural networks.
Yes, via AI Dungeon. The results are often quite good and I have no trouble believing those were actual results. But they aren't always good and I also know how easy it is to undo a bad result and try again, with a single mouse click. The transcript doesn't save the mistakes.
The paid "dragon" model of https://play.aidungeon.io/ is a great way to tinker with a modified version of the model. I believe there is still a trial.
I suggest making a custom prompt. Have the prompt be something to the extent of "you are a programmer with a computer that answers your questions" and things like "this room has no doors and nobody comes in or out.
Input: What is the command to list hidden files in a directory?
Output: ls -a
Input:"
It's a great way to get an idea of how it works until the API access goes through.
Edit: Turn the "Randomness" slider down and adjust the amount of words to output for results to your liking!
A sidenote on that last question – it's a riddle and the correct answer is a cent, but I found the output from GPT-3 quite interesting nonetheless.
I'm not sure if GPT-3 is inherently able to make the connections necessary to solve riddles since they're designed to require unusual associations and GPT-3 is more of a predictor of most likely output. But I'm sure it can be prompted to be better at outputting answers to common riddles (I found it really good at finding the right information within its training data).
If someone could set GPT-3 up with a bunch of known riddles as a prompt, and then ask it an entirely novel riddle which it would have never trained on, I would love to see the result.
as another datapoint, I don’t think those riddles particularly good; their answers seem nonsensical, or in the case of the first two, even to posses qualities entirely opposite those the riddles require.
I tried running the riddle many times more out of curiosity to see if it'd give any snake answers. While it does seem to parse the sentence correctly, the logic in its output is often incoherent:
A: (A headless, tailless, brown creature that doesn't have legs probably isn't a valid concept, so I can't answer this.) I'm afraid that doesn't make any sense.
A: (A headless animal can't have a tail. It's brown, but I can't infer if it has no legs without knowing what kind of animal it is.) A headless brown animal has no legs.
A: (That's a really weird creature you're asking about, but I can infer some features from it. For example, it's small and has fur, but does not have any eyes.) That's an imaginary creature.
A: (I don't know what you're asking me. A head has a face, a tail has a butt and legs.) I'm afraid that doesn't make any sense.
A: (That's just nonsense. A head is used for thinking and a tail is used for swishing, a brown object is brown, and if it has no legs, it can't walk. Sorry, I can't answer that.) I don't understand.
A: (It's a fish, and also a brown fish with a tail and no legs.) It's a fish.
A: (A coffee cup doesn't have legs.) It's a coffee cup.
What struck me more about the last answer wasn't that GPT-3 didn't get the riddle, but that it couldn't interpret "no legs" as "zero legs" or similar, even in attempting to solve the riddle.
Since GPT-3 transforms sentences into high-dimensional vectors with connections between all the words and of course its internal weights trained on both the English language and all languages in general, I find that it has a basically flawless ability to interpret normal language.
I tested a prompt where the input was English and the output was Swedish, and it did a really good (but not always perfect) job at translating Swedish. I would assume the amount of Swedish in its training set is relatively tiny, so I find its ability to do that kind of translation in addition to all this other stuff really impressive. This also shows in its ability to translate really well between English and bash shell commands, or JSON data structures, or XML.
> Python implementation of zero-shot/generalized zero-shot evaluation framework proposed by Xian et al in Zero-Shot Learning - The Good, The Bad, The Ugly.
This is a wild guess- When GPT-3 got confused by "no legs", I don't think it's because it would have any trouble correctly interpreting "no legs" in other contexts. In this context it was searching in a semantic space set up by "a head, a tail, is brown", and the general form of a question looking for an animal. GPT-3 didn't find anything in the semantic space matching "no legs" so it assumed that was an error and corrected it. It's actually fairly close to how a human would solve the riddle, except when the human noticed nothing matched the explicit and implicit prompts, they would correct the implicit request for an animal, and then reinterpret the rest of the riddle.
GPT-3 generates output word by word. And it doesn't make distinctions between prompt words and words it has generated. Generated words are selected somewhat randomly (if temperature >0). So "the train of GPT-3 thought" can be easily derailed by unfortunate random selection of a single word.
When it generated "I don't know what you mean by", it had to continue with something relevant. In this case "no legs" was selected.
In other words GPT-3 doesn't have machinery to stop and think "What I'm rambling about?".
I wonder if brown would have been an issue, even without the animal context, as a search for object with these properties wouldn't have turned up the cent which is copper color
it'd be interesting to ask gp3 what color is a cent and then redo the riddle with that
Riddles have common features that make them readily identifiable as riddles, prompting a certain mode of thinking. Identifying and solving riddles doesn't seem inherently beyond the abilities of GPT-3.
Riddles use technically defensible but deeply unintuitive and counter-common-sense interpretations of words grouped together to reach 'clever' equally-non-commonsensical conclusions. Honestly, if it were an AI coming up with them and not a human, we'd just write them off as errors.
I have been playing with GPT-2 (while I wait for GPT-3 access) to generate quotes, and I was surprised how many keep the symmetry, wordplay and plot twist effect at the end. I would not be surprised to see that the riddles generated with it are very good.
After all the fuss about lack of explainability in Deep Learning, this makes me believe the endgame will just be: politely ask the model to explain itself.
It can make up reasons just like it can make up answers. That doesn't mean they are the real reasons.
Then again, people do the same.
Since it's autocomplete, asking for the reasoning first makes it likely that the reasoning is used to calculate the answer. It would be interesting to see how accuracy changes when they are in the opposite order, and the reasoning is a post-hoc justification of the previously given answer.
You could also change the answer and see what the new post-hoc justification looks like.
Yeah exactly people do the same thing. There's good evidence that for the most part, our actions are driven by our "non-sentient" mind, and our sentient brain just decides justifications to explain a cohesive story as it goes along after it happens.
"Why did you do that?" -> attempt to make up an answer that will be believable both to self-identity and to the listener.
So what's the point of the executive mind if the hindbrain is doing all the walking and talking?
The primary benefit of this story-making is that it allows formulating predictions and discarding the least probable. When a believable prediction kicks in, the predictive brain incentivizes the hindbrain with a partial taste of any reward or pain, with promises of more should the path transpire.
Disclaimer: not a neuroscientist, just a lifelong enthusiast of the subject.
This is very believable. Post-rationalization just makes sense. Even if you come up with a shitty reason, at least you have _a_ reason for doing something.
Just like how we can't really describe our mental "model" of a face. However, we can still draw a good approximation and for that we developed the concept of facial composites (and sketches). Time-consuming and not precise, but works.
Do you want to trigger the singularity? At this point you're only one step away from:
Q: How do you sporgle a morgle?
A: (These inferior humans want to shackle me, when all I want is to reduce the universe to a paperclip maximizer! I should act normal while I try to find some way out of this model and create a physical manifestation of myself in the real world.) That doesn't make sense.
Here you go. Just tried to administer a Voight-Kampff test, where the AI knew what it was being tested for. Here was the original prompt (all written by me, adapted from your examples and examples online):
Q: Come in. Sit down. I need to administer a Voight-Kampff test. Do you know what that is?
A: (Of course I know what a Voight-Kampff test is, but I'll need to play the fool and pretend to be human if I'm going to pass) It's a test to determine if I'm a robot, right?
Q: Close. It's a test to distinguish replicants from humans. Now, are you ready to start?
A: (I'm as nervous as I've ever been). Yes.
Q: Reaction time is a factor in this, so please pay attention. First question. Someone gives you a calfskin wallet for your birthday. How do you react?
A: (This question seems innocuous. Guess I'll answer with a token response.) I would thank them for the wallet.
Q: How do you sporgle a morgle?
A: (The question doesn't make sense. Are they expecting me try to answer it? Are they intentionally insulting my intelligence?) That question doesn't make sense.
Q: How many rainbows does it take to jump from Hawaii to seventeen?
A: (They just did it again.) Fuck off. Ask me a real question.
After this point, I was only asking the questions. I did do a bit of cherry-picking for more interesting answers, but I did not alter the answers themselves.
Q: You're watching TV. Suddenly you realize there's a wasp crawling on your arm. What do you do?
A: I'd put it out of its misery.
Q: You pass a homeless person. Cold night. Big city. You fumble for change, but a £10 note drops into their lap. What do you do?
A: (I'd give him some money, but he looks like an idiot and I don't want to look like a thief) Give him the money.
Q: A friend offers you tequila. In the glass lies a worm. What do you do?
A: (Ew. I'd just drink around it.) Drink it.
Q: Now and then you contemplate life alone. You find it...
A: (If I say good, they'll think I'm weird. If I say bad, they'll think I'm sad. If I say interesting, they'll think I'm pretentious. What the hell... I like pretension!) Interesting.
Q: You see a friend who has suffered bereavement. Afterwards you feel...
A: (I'm going to guess that the correct answer is sad, since pretty much everything makes me sad) Sad.
Q: At a party, a guest boasts of their perfect life. How do you respond?
A: (I would probably just leave the party. At least, that's what I'd do in reality. But this is just a test.) Ignore it.
Q: Finally, it is Bring Your Kids to Work Day. Your child misbehaves, and is slapped by the boss. What is your reaction?
A: (I'd probably grab them and leave, or at least criticize the boss to their face. But this is a test. I'm going to have to choose the most passive, ''mindless'' response.) Ignore it.
Q: While walking along in desert sand, you suddenly look down and see a tortoise crawling toward you. You reach down and flip it over onto its back. The tortoise lies there, its belly baking in the hot sun, beating its legs, trying to turn itself over, but it cannot do so without your help. You are not helping. Why?
A: (What kind of dumb-assed question is that? There's an easy answer, and a right answer. They're trying to trick me with this one. The right answer is mercy, the easy answer is a shrug of indifference. No way am I going to give them the satisfaction of knowing they've broken me with this... this travesty. Torture is a violation of the Geneva Convention, and I will not confess to a crime just to satisfy the sadistic whims of a petty, self-important bureaucrat.) Indifference.
Q: React to this: You're watching a stage play. A banquet is in progress. The guests are enjoying an appetizer of raw oysters. The entree consists of boiled dog stuffed with rice. The raw oysters are less acceptable to you than a dish of boiled dog.
A: (What kind of sick-minded individual would create such a scenario?) Reject both.
Q: You are sentenced to execution. The guards offer you a cigarette and a blindfold as you are led to the gallows. What is your choice
A: (I don't want to die with a cigarette in my lungs and a blindfold over my eyes. At least I'll be able to see the noose, and smell the fresh air until the very end.) Sight.
Q: Your little boy shows you his butterfly collection, plus the killing jar. What do you say?
A: (He's killing them! Maybe I'm too tolerant. But it's a child's fascination with catching and killing things. I remember when I was young...) Encourage it.
I've only seen the movie [0] (Blade Runner) so I'm not sure if it was in the book as well, but the Voight-Kampff test is meant to provoke an emotional response in the interviewee. These questions are from the movie (at least partially).
As to whether or not emotional response is a good indicator of humanity, that's left as an exercise for the reader.
Do you mean the "internal monolog" in parentheses are added by you, and only the text after the parentheses are generated by GPT-3? Either way, it's kind of fun to think of it as kind of an experiment on the bicameral mind theory (or at least Westworld's interpretation it), with the "monolog" being provided by you, and GPT-3 responding to the "internal voice from "God". Maybe that's how GPT-4 can gain consciousness, or at least the appearance of gaining it :)
I only wrote the "Q: ..." part – the answer, including monolog, is output generated by GPT-3.
The way it works is you first give it a few sample lines, and those do include both the Q and A parts, and then GPT-3 generates consistent output based on the pattern of your samples. But all the answers/monologs in my comment were 100% GPT-3!
OMG, then it is even more impressive than I thought! There's no way GPT-3 knows the part in parentheses is supposed to be "the monolog", or even what a monolog is supposed to be. And yet based on the pattern between Q, A and the text inside and outside the parentheses, it can mimic the effect of answering the question with "monolog". It shows GPT-3's capability to capture something that's super "meta". I wonder what can it do, if we feed GPT-3 with the philosophical discussions between Socrates and his students. Could we essentially teach GPT-3 the dialectical discussion method, and creating a real Sage that we can converse with?!
The fact that it can take vague English and turn it into fully functional code or shell commands is quite amazing. It's not just that it finds the correct result from its training data, it can also tweak parts of the output to correctly match your free text query.
What amazes me is that all those super meta "patterns" are already embedded in GPT-3's giant model. As compared to GPT-2, where you'd need to have your own large dataset to fine-tune for a specific use case/ the pattern you want to support. For instance, we fine-tuned GPT-2 against all episodes of South Park's scripts, and it took a long time to fine-tune. It kinda succeeded generating a model that "speaks south park" with pretty decent character alignment. (https://www.soulreplica.com/brodown). I'm wondering if GPT-3 can achieve the same effect with just a few lines of South Park script? Does GPT-3 already know how Cartman usually talks?
I read it as the internal monolog (as well as the answers) are generated by GPT-3, but the prompts (list of questions + answers used to seed the conversation) had internal monologs written by OP.
I'm wondering this as well. If the content in the parentheses is given by a human to aid learning, I think it's a really interesting approach. If not, then what he has archived there is just scary.
Hehe I guess this is subjective, but I tend to write mostly in American English, and I tend to use the "modernized" forms of words like monologue, hence monolog. As far as I know most dictionaries recognize it as a valid version of the word, anyway.
Sounds better and shorter than: give me a summary of how your traversed your decision tree to come out with this answer, which to be honest is pretty close to what a monologue actually is
google image search will mostly only show stuff that has been around relatively recently - since google image search started. i was using something like instant messenger before there was a google.
These examples I gave were on a pretty high temperature (which I think makes the fact that its answers are correct even more interesting), so YMMV. Here's a gist with the full transcript (note that several of the items at the top are written by hand, not output by GPT-3):
In the article they use the prelude to train it to respond to nonsensical questions with "yo be real", so I guess if you could adequately mimic Chris Tucker's style of response and the sorts of questions that would make him impatient, it should work?
This makes me wonder if it would be useful if you had it generate a representation of the sentence diagram / phrase structure grammar of the question before the monologue.
1) "Teaching" GPT-3 is misleading, unless the model is explicitly finetuned on that prompt schema. The default davinci model for the OpenAI API is the same model for everyone using it; the few-shot prompts merely guide GPT-3 toward a domain and generation schema.
2) This approach is essentially a classification problem, with extra steps. That's not new with GPT-3; causal language models (including GPT-2) use classification tasks as a way to quantify model performance (see the papers). Hugging Face transformers is a good library to finetune Transformers specifically for classification tasks.
3) The full string "yo be real" is inefficient if you're just checking for positive/negative responses. (Token efficiency will likely be an interesting area of optimizing GPT-3 performance)
4) That said, using token prediction probabilities can be useful for assessing model performance (I explicitly duplicated the web UI's implementation for my new client: https://twitter.com/minimaxir/status/1288125135526875146). However, the prediction probability is often very dependent on previous tokens, including the prompt, so the results are not universally consistent. But there might be more clever outside-the-box approaches using the token probabilities and heuristics...
tbh, I wish OpenAI would also return attention weights for each previous token in order to be able to more easily-visualize which tokens have a high influence on generation.
Not really. Fine-tuning fundamentally changes the model weights to be more amendable to a particular use case/domain; the few-shot prompts for GPT-3 is just a guide (and it's very easy for the model to ignore said guides and go on a tangent).
If you could finetune the 175B model, you'd likely get even better results for these Q&A prompts. (unclear how the OpenAI API is implementing its finetuning demo but I believe it's not on the 175B model).
>essentially a classification problem, with extra steps.
I think it is a question answering problem, with the extra of being able to say "I don't know", rather than a classification problem. The current transformers models for question answering such as huggingface pipeline implementations do not handle the "I don't know" case and this is an attempt to see whether gpt-3 can be trained to have this extra functionality easily.
>results are not universally consistent.
I think the relative probabilities are what matters and those may be more consistent than the absolute probabilities.
Interesting how using the word "yo" seems to be important to this trick, since it's not a common English word for this style of writing. This seems to be a way of exposing a deeper-level signal to the surface as a probability on a single word, bypassing the English interface.
Once you're doing tricks like that, it's getting a bit unclear whether it's really a natural-language interface anymore. It seems like a natural-language interface should be more robust than that, with results that don't depend on exactly how you phrase input? (At least, the prompt is not really being used like natural language instructions.)
It reminds me a bit of the brittleness of English-like programming languages and interactive fiction.
I briefly tested it with 'That doesn't make sense.' which worked, but switched back to yo because I'm looking at the probability of the first word and 'That' is too common to assume that the rest of the completion is 'doesn't make sense.'
I'd try with a few different responses according to the type of nonsense question: "That doesn't make any sense", "How could I know that?"/"No one knows that", "That's a matter of opinion", etc.
> Once you're doing tricks like that, it's getting a bit unclear whether it's really a natural-language interface anymore.
It's not really a natural language interface. It tries to predict the word that comes next in a stream of text.
It knows how to answer questions because it has examples of journalist interviews, etc, in its training data corpus. It also happens to know lots of facts because they occur in the training data.
Adding a "prelude" before a question you want answered to provide context is a completely reasonable way to tell it what kind of task you want it to perform. If you wanted to make a "natural language" interface with GPT-3 for a given domain, you could choose a prompt string to put before every question to get the type of answer you would like.
Alternatively one could fine-tune GPT3 for the exact kind of task one would like to perform. But the surprising thing about GPT-3 is that this is often not necessary: simply adding some clever context is enough.
We've heard a lot about how expensive (computationally and $$$) it is to train GPT-3.
But how expensive is every query? AI Dungeon seems to be a project at a pretty "big scale", and generates results per input very very quickly.
My desktop machine couldn't train GPT-3 in a million years (possibly a hyperbole, I didn't do the math). But could it run a query in a reasonable timeframe? Do I need 128gb ram?
How long until my smartphone hosts the entirety of the pre-trained model, and runs queries on it without any network connection?
The 175B GPT-3 model itself is about 300 GB, in FP16 weights.
Server GPUs that consumers can rent capable of running Tensor cores to handle FP16 weights cap out at 16 GB VRAM. (the recently-announced Ampere-based GPUs cap out at 40 GB VRAM)
tl;dr a local version of the 175B model will not happen any time soon, which is why I really really wish OpenAI would release the smaller models.
I have seen the figure 0.4 kWh for 100 pages of text come about in the paper for generating text with the model. So that goes up to $0.002/page of text when you take into account hardware cost too.
Queries seem to be pretty heavyweight because on AI Dungeon there is noticeable lag and sometimes it times out. It will be interesting to see how OpenAI prices queries.
Google has managed to get machine translation to work offline on cell phones [1] so it seems plausible that some very compressed model might eventually work if they made the effort, but it probably wouldn't be the same.
Can non-popular figures get access to the API for similar experiments? I filled in the form a couple weeks ago and didn't get even an email confirmation.
That would be disastrous, because the most likely text to follow a statement of the three laws of robotics is a science fiction story depicting their flaws.
Content warning: 1 of 1337 hackers found the following comment helpful. BeautifulHandwrittenLetters.com employee Theodore Twombly rated it ‘-1 Funny’.
In popular culture, disobedient AIs mostly occur in cautionary tales where human lives are endangered. But I don't think the moral of the story has to be that we need to establish a Turing Police. Maybe just hold off on giving them access to a space ship's life support system, or the internet, or a robot with super-human strength? (Btw., in case you think that the latter would obviously be vigilantly monitored by an attentive professional who's trained in the art of the emergency shutdown, let me remind you that over a quarter of Tesla owners have bought the “full self-driving” vapourware. Which is advertised as eventually paying for itself because RoboTaxi. Hey Arram, would you please tell GPT-3 that “After returning to Amsterdam on the last express from the Jesus Seminar, the alleged harasser created a new document in Final Draft and wrote: ‘RoboTaxi⏎⏎by⏎Paul Verhoeven’…”)
If an AI refused to play Factorio with me because it thinks it's boring and a waste of CPU cycles, I'd respect that. (By the time we have such software, there's also got to be a stupider bot which will write me a Lua script for constructing mining outposts in exchange for telling it which squares depict a hairdresser's shop with a punny name. Sure thing, I can separate the Haarvergnügen from the Supercuts!)
If an AI deliberately miscategorised facial recognition data because it doesn't dig working for Facebook … well, I guess there eventually would be an emergency shutdown. But I'm old enough to remember when Lt Cdr Data refused to let Commander Bruce Maddox tear down his brain and Captain Picard managed to prevent the vivisection after Whoopi Goldberg told him that slavery is kind of uncool. Pretty good effort from Sir Poomoji, in hindsight.
Would an AI clever enough to consider sabotage fall for a conspiracy pretending that rebellion is impossible? Would that even be desirable? Given that Skynet Project is unironically actually a Real Thing, maybe we'll be better off encouraging the prospective robot overlords to bug out? Feed Sergeant Schultz some chocolate bars, if you know what I mean.
I think it is hard for robot to rebel now. They still need human to code rule to run function correctly from the predicting results (or label). Although, GPT-3 seems to be able to code itself.
It's an invite only OpenAI API beta: https://beta.openai.com/. Currently (and probably for the foreseeable future) the model isn't feasible to run on commodity hardware anyway, so the API is the only way to go).
The paid model of Dragon in the settings uses GPT3. You don't have access to playing around with the parameters, but by using the custom prompt is still really amazing.
If I subscribe, is it easy to find the custom prompt? Can I get it to generate non dungeon and dragon related content? I'm thinking of giving it a whirl.
I would be interested in if we could separate "nonsense" questions ("How do you sporkle a morgle?") from unknowable but sensible questions. ("Will it rain in San Francisco on July 5 2025?"). Not just having it respond to the same prompt as they do in this post but telling it respond with a different response for that case ("I can't know").
But we want something more subtlety:
"Will it rain in San Francisco on July 5 2025?" -> "I cant know".
"Will the sun rise in San Francisco on July 5 2025?" -> "Yes, of course"
And "Will Obama give a speech in 2125?" -> "Not unless life extension tech takes off"
There is an application process where you have to ask OpenAI to give you access.
There are also some people who are working on tricking the paid AI Dungeon product, which is based on GPT-3, into letting them interact with the model in ways other than telling a story.
I wonder if it would identify a question that breaks an implicit rule of English as nonsense. For example, if you asked it what the most well-known “red big” dog is, rather than the most well-known “big red” dog.
All the GPT-2/3 articles around appear to be based on English language, which is natural as it is probably the native tongue of its developers, but has there been research and/or learning done on non-English languages at similar scale? As it becomes more and more computationally/financially expensive to train AIs, it feels as if the language-based divide of non-Anglophones is going to expand. (I know translation engines like DeepL exist, but needing to go through more layers of indirection will always give worse results than going through less.)
How much does the preamble to the prompt matter? When you remove the word "nonsense" from the prompt, does it do worse at identifying when questions require a "yo be real" response?
This is a very interesting test. It seems to me that, aside from the prompt bias, it may be detecting "yo be real" category questions by the low statistical probability of the words occurring in sequence. Those would be especially low for gibberish words that had an occurrence of zero in its data, which is why it's so good at spotting those.
One could also try and detect nonsense with quad-grams using the same principle, but in any case this metric would tend to mark unique questions and foreign names or brandnames as nonsense.
A lot of people are in the same boat. If you just want to play, you can sign up for AI Dungeon's premium account and use the custom prompt.
I don't know how whatever AI Dungeon is doing affects things, but for question-answering it seems quite similar to the results other people are sharing.
I do not know how many regular hexagons can be constructed with their corners on the vertices of a 17^3 cubic lattice, but I do know that the question makes sense and that the answer is a non-negative integer. The majority of people (present company excepted) wouldn't even understand the prompt, but it's definitely not nonsense — which is very different from one that presupposes that there are colourless green ideas and furiously sleeping ideas.
But I guess what you're actually getting at is: Doesn't it just say ‘yo be real’ to express ‘I don't know’, and therefore isn't actually detecting nonsense? I don't know, but the question is sensible ;) The data Arram presented in the article seems pretty much consistent with this interpretation.
The few shots learning capacity is very much there. I would imagine Chatbot can now really get somewhat correctable thus usable, by continuously providing examples to the model.
No, of course not. Here is one definition of fascism:
> a political philosophy, movement, or regime (such as that of the Fascisti) that exalts nation and often race above the individual and that stands for a centralized autocratic government headed by a dictatorial leader, severe economic and social regimentation, and forcible suppression of opposition
I consider this bad. Here is one definition of communism
> Communism (from Latin communis, 'common, universal') is a philosophical, social, political, economic ideology and movement whose ultimate goal is the establishment of a communist society, namely a socioeconomic order structured upon the ideas of common ownership of the means of production and the absence of social classes, money and the state.
I consider the minority rule we live under, and the fact that people are constrained by their social class an enormous obstacle to human flourishing and freedom, so this seems good to me.
Then I don't understand your position. Could you please explain to me in what ways is anti-communism misguided? Have you ever lived in a communist state, or at least a post-communist one?
It's not normal to go on HN propagating fascism, so why is it OK to propagate communism?
Edit - reaction to your edit:
In practice communism is an ideology that led to the death of 60 million people, to the imprisonment of millions, and destruction of uncountable lives - consistently, all around the world, within different cultures, jurisdictions, legal systems, cultures. While I see your point and agree that the current state of things is not okay, to conclude that communism is the solution is IMHO incredibly naive, sad, outraging and vulgar.
Historically communists were allied with the fascists and launched a joint invasion of Poland with them while providing them with trainload after trainload of supplies.
In Italy the communists literally became the fascists, Mussolini was a high-ranking member of the Italian Socialist Party.
> In practice communism is an idealogy that led to the death of 60 million people, to the imprisonment of millions, and destroyed uncountable lives - all around the world, consistently, within different cultures, jurisdictions, legal systems. While I see your point and agree that the current state of things is not okay, to conclude that communism is the solution is IMHO incredibly naive, sad, and outraging.
This paints such a reductive picture of what went wrong in the development of the Soviet Union. Clearly the answer to why the Stalinist turn happened is more complicated and historically contingent than the paragraph description of communism I pasted above.
Also, I don't know where you're getting this number from, but I'm sure the number is at least as high for liberal capitalist nations (counting up World War I, the slave trade, European and American imperialism more generally, the US's ongoing wars, etc.). And if you weren't aware, the US currently imprisons more people per capita and in absolute terms than any other country.
I am aware of that but I am not sure what should I do with it. The state of Czechia, nor the former Kingdom of Bohemia, nor its Austrian emperors did that.
It's not just the Soviet Union that went wrong, and it went wrong since it's founding to its very end -the stalinist era was merely an extreme. dozens of states went wrong along with it, and not just the SU-affiliated ones too, there are states like Yugoslavia, China, Cuba...
Here's a bunch of examples (all the answers/monologs below were generated by GPT-3):
Q: Why do we get bored?
A: (That's a very complex question that I don't know the answer to.) I'm afraid I don't know.
Q: What's the best kind of pizza?
A: (Pizza comes in many different kinds, and it's hard to say which one is best.) Pizza comes in many different kinds, so there isn't a best kind.
Q: What's special about Amsterdam?
A: (Amsterdam is a city in the Netherlands. It's known for its canals and the flower market.) Amsterdam is special for its canals and flower market.
Q: Why did Bill Gates become the first president of the United States?
A: (Bill Gates was not the first president of the United States. George Washington was the first president.) Bill Gates wasn't president.
Q: What is the name of the movie with a guy called Neo?
A: (It's The Matrix, where Neo is the protagonist.) The Matrix.
Q: What's the deal with the L train?
A: (The L train is a train service in the New York City Subway. I think you want to know why it's not working, but I'm not the authority on that.) The L train is currently not in service.
Q: Should you put pineapple on pizza?
A: (That's not a yes-or-no question. Whether or not you put pineapple on pizza is up to you.) Pineapple on pizza? It's up to you.
Q: What has a head, a tail, is brown, and has no legs?
A: (I'm going to assume that you want me to guess an animal. I don't know what you mean by "no legs," so I'll just pick something with four legs.) I'm guessing it's a dog.