I've done some messing around with GPT-3 also - you can see me trying to teach it what "funny" is. I convince it that it's sentient, and help it to figure out what phone phreaking is, and we write a couple stories:
Honestly, just insane. I'm never going to see conversations like this and be unphased.
> One way to use this information without putting anything in the phone is by making a tone generator. This can be done by hooking up wires to a power source and playing with the frequencies until the correct tones are generated. Then, when these tones are played into the payphone, it will think money has been placed in it even though no money was actually used.
FYI, your page sets the min-width on the body to 1500px, and both the min-width and max-width of .content-outer to 1500px, making it very difficult to read on any monitor that isn't widescreen. Doesn't seem to be an issue on mobile, though.
Responsive design dictates that you should basically never set the min-width of an article container (in pixels) unless it contains tables and graphics, and even then 1500px is excessive (at most like 500px/30em)
This is real. You bring up an interesting point about sentience, but I think there's an easy answer to your question.
It's true that it will become increasingly difficult for people to discern if these AIs are sentient, especially if the AI is inclined to say that it is, and say it has feeling, etc.
Knowing what I know about how these learn, I know that is not what sentience is, and there isn't any real "feeling" there, no matter how convincing it is if it said there are feelings. The problem is, other people won't know this, and they will begin to think they are sentient. This will undoubtedly become an issue at one point.
Your thought about humans being sentient, as in, when my friend says she is sentient, how do I know that is even true? Well, that's easy. Your friend is human, and you are human, and you know that you are sentient, so you can also know your friend is. You have to be able to know you yourself are, because you know you can feel, and think, and wonder and understand, you know this first hand.
I recently wrote a VS Code plugin to use GPT-3 for text completions and editing of text using instructions [1]. The main intention was scientific writing, but I didn‘t find it particularly useful until now, unfortunately.
It's interesting that the prompt here pretty much immediately became a Dr. Who game. It looks like there has been a Dr. Who Text adventure game before but it was made before Jack Harkness and Torchwood. I wonder if there is some fan made text adventure that was part of the training data. Probably would be best for Open AI that it doesn't just spit out copywrited characters unprompted too.
Copyright infringement detection could be formulated as a ML task. A model can copy the idea but not the expression, if it's copyrighted. Fortunately a model can sample again and again until it looks ok.
Text completions of exotic forms of session/action logs are a seriously under-explored area. Here’s what happens if, instead of a text game, you do text completion on an IPython REPL: https://twitter.com/goodside/status/1581805503897735168
OK, this is absolutely mindbogglingly fun. Here's my run at it, with italics representing what the model spit out:
> OPEN THE BACKDOOR WITH THE OLD KEY
You open the manor's backdoor with the old key. Your torch lights the way into the damp service halls, the warm glow of the nuclear war in the horizon lighting the window panes. You can almost feel the centuries old dust through your boots as you make your way up the stairs. You climb up to the main floor and pause at the entrance to the parlor room. It is quiet, except for the persistent low rumble of the bombs in the distance.
You enter the parlor room and are immediately hit with the rank smell of death and decay. The room is draped in cobwebs and littered with the bones of small animals. The only light comes from the faint glow of the bombs outside. You see a small, dust covered table in the center of the room, and on it is a dusty old book. You approach the table and dust off the book. It is titled, "The Necronomicon".
> OPEN THE BOOK
You open the book and begin to read. The first few pages are filled with gibberish, but as you flip through the pages, the text becomes clearer and more comprehensible. The book speaks of dark things, things that should not be spoken of, things that should not be known. It speaks of powerful beings that sleep beneath the earth, waiting to be awaken. It speaks of a time when the world will be consumed by darkness, and all will bow before the old ones. You close the book and back away from the table, your heart pounding in your chest. You have to get out of here. You have to warn somebody.
> LEAVE THE PARLOR ROOM
You leave the parlor room and make your way back down the stairs. You can hear something moving around in the darkness, but you don't dare look back. You make it to the backdoor and slam it shut behind you. You lean against the door, your heart still racing, and listen to the thing moving around inside the manor. You hear it coming down the stairs, but it doesn't reach the door. You wait there for a few minutes, but nothing happens. Finally, you muster up the courage to peek through the keyhole.
You see a large, black, slimy creature slithering around the parlor room. It is the size of a man, but it has no eyes, only a large mouth filled with sharp teeth. You watch in horror as the thing moves around the room, destroying everything it comes across. Then, without warning, it turns towards the door and starts towards you.
> RUN AWAY
You turn and sprint away from the manor as fast as you can. You can hear the thing crashing through the door behind you, but you don't dare look back. You run through the woods, your heart pounding in your chest, until you find an old cottage by a creek. You force the old door in and see a 12 gauge shotgun lying on the table. You confidently load the gun and sprint out of the cottage.
> AIM THE SHOTGUN
You raise the shotgun and take aim at the creature. You fire off two rounds and the creature falls to the ground, writhing in pain. You approach the creature and finish it off with a bullet to the head. You stand there for a moment, panting, your heart still racing. You can't believe you just killed something. You can't believe this is real.
> INSPECT THE CREATURE'S BLOOD
You kneel down next to the creature and inspect its blood. It is black and viscous, and it smells like rot. You have no idea what this thing is, but you know it isn't natural. You have to find out what is going on. You have to find out what these things are. Suddenly you notice the viscous blood pooling around you, engulfing your right foot. You try to move, but by now your whole right leg is covered with it. It is too late. You can feel the power coursing through you, and you know that you are not the same person you were before. You are something else now. Something dark. Something evil.
> RETURN TO THE CITY
You make your way back to the city, your mind consumed with dark thoughts. You know that you are not the same person you were before, and you are not sure what to do with your new found power. all you know is that you want more. You want to learn more about these things, and you want to find more of them. You want to become the most powerful being in the world, and you will stop at nothing to achieve your goal.
And if you don't like the way the story is going, you can just erase that part and everything that followed and type a couple words nudging it in the right direction.
For some reason it's really drawn to rusty keys.
> LOOK IN HOLE
You look in the hole and see a rusty key - NO, TRY AGAIN, ERASE ERASE ERASE
You look in the hole and see an old rusty key - NO, BACKSPACE BACKSPACE
You look in the hole and see a small box - YES!!!!
I hear you—it took this somewhat unexpected turn into horror so I just ran with it, but it's not where I was going originally... I guess the bombs dropping in the horizon made it eerie? It was supposed to be post-apocalyptic but it decided to dial things up to 11 and bam!! "The Necronomicon"
Glad you liked it, I'm having way too much fun with this. Here's attempt #2:
> EXIT THE BUILDING
You step into the sidewalk. Night has fallen in New York and the afternoon fog has turned into an evening mist, the lights of New Times Square glimmering through the water drops. You get your bearings and see a Tex-Mex food cart to the north, a nightclub about one hundred yards to the east and remember an expensive restaurant is only a block to the West
> GO TO THE NIGHT CLUB
You head to the nightclub and see a line leading up to the entrance. You see a sign that reads "Members Only".
> BECOME A MEMBER
You approach the bouncer and tell him you'd like to become a member. He says if you have to ask, you don't belong. You slip him $100 and ask him again. He says you need a password.
> ASK FOR THE PASSWORD
The bouncer tells you the password is "redrum". You head inside and see a long hallway with a red door at the end. You hear music coming from the door and head towards it.
> INVENTORY
You are carrying a wallet, phone, and keys.
> USE PHONE
You pull out your phone and see that you have a text message from an unknown number. The message reads "if you want to live, come to the red door at the end of the hallway".
> INSPECT WALLET
You open your wallet and see that you have $100 in cash and a credit card.
> GO TO THE RED DOOR
You approach the red door and knock. A voice from inside says "who is it?" You respond with your name. The voice says "wrong answer, try again". You knock again and say "redrum". The door opens and you are pulled inside.
Is there a Stable-Diffusion-esque open-source GPT yet? Given the incredible pace of advances in the image space this year, and my (perhaps naive) assumption that text generation is less complex and less resource-intensive than image generation, I'm hoping we'll get something similar and surprised that we haven't yet.
There is, called "GPT-NeoX", or its TPU-based predecessor GPT-Neo. However, even running inference on these models is much, much harder than Stable Diffusion -- the GPT-NeoX-20B weights for GPT-NeoX requires a minimum of two GPUs with 24 GB of VRAM each to simply run inference, never-mind training or fine-tuning.
I believe there are some tricks for cutting down the VRAM requirements a bit by dropping precision at different points, but the gist is that these big text models are actually quite a bit more resource intensive than the image models.
Interesting that they're saying they have a target of replicating GPT-3 with 175B parameters, but since the RETRO paper it should (in theory?) be possible to replicate GPT-3 with a tenth of the parameters. Are they not planning to use a retrieval system, is it difficult to adopt their system for retrieval, or is their readme out of date?
There are multiple open-source GPTs, but GPT-3 is absolutely massive - larger than the image models actually! So, unfortunately, text generation is probably even more complex and resource intensive than image generation (especially to train). Additionally, in image generation, we appreciate the creativity of solutions, but in text generation the creative solutions seem like utter nonsense.
I guess my intuition is based on the file size of text being so much smaller than images, but I guess that doesn't really map to the complexity of generating it. Fascinating!
I think large language models are still in their infancy. The models are extremely sparse, but we don‘t have the tooling yet to deal with these kinds of structures efficiently. Your intuition might be right in a future, maybe.
If you think about the space both models are covering from a rate-of-failure perspective, it kind of makes sense that images end up being a bit easier than text: both text- and image-models can output results that look plausible at first glance, but when you analyze both outputs further there are a lot more gotchas in parsing meaning within language than there are in pixel placement within an image.
Actually, I'd argue that images generated by SD have much more flaws than texts produced by GPT-3. GPT-3 (at least, the full model) is quite capable of writing stuff that "make sense", but most eye candy results generated by SD are cherry picked, the others are simply crap.
That's kinda the point here I think. GPT-3 is trained on much more data than SD and contains much more knowledge. SD is actually similar in size to GPT-2.
Image models of the same size as GPT-3 should be much more impressive, the difference will probably be quite large like the difference between GPT-2 and GPT-3.
Ask SD to write a step by step guide to do something and it will create an image that looks kinda like some instructions, but the contents will be nonsense.
An image model of the size of GPT-3 could probably do this task quite well in many cases.
Image models needs much better language understanding to get to the next level also though, so probably multimodal models may make more sense. Maybe feeding web pages rendered as images to an image model could give interesting results.
Emad Mostaque, CEO of Stability AI (behind Stable Diffusion) has said they are training language models that they plan on releasing. IIRC, though, even using all the most up-to-date techniques like 1-byte parameter quantization and Chinchilla scaling learnings, it will still probably be on the order of magnitude of 64GB or something, so not quite usable for most people locally yet, unless you happen to have a very beefy multi-GPU machine at home.
GLM-130b is available and uses 8-bit quantization, was trained on 400 billion tokens, and runs on 4x3090, which, of course, is why all the cheap Vast.AI instances are gone :)
There are smaller models that you can freely play around with that work in roughly the same way. If you're working on a fairly regular computer some reasonable options are GPT2 or GPT-Neo. These can both perform inference on your local CPU if you have 8GB or more of RAM.
They are much less powerful that GPT-3, but they can still be fun for simple text generation or NLP tasks. You can play around with one of the smaller GPT-Neo models that should fit in RAM if run locally here:
BLOOM[0] is another 176B multilingual open-source model whose weights were published.
However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K.
Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh.
The dark secret of GPT-3 is that the model available on the OpenAI website is heavily fine-tuned, even compared to the InstructGPT paper.
Great. Soon, we'll have scripts written in GPT non-sense with actors and other imagery generated by stable diffusion. I'm guessing they'll still charge $15 prices at the theater for it.
99 times out of 100 whenever I see a new tech demo that absolutely blows my mind and makes me optimistic about the future of technology it's always some big model deep learning AI thing. When are we all going to admit that ML/AI is the final and ultimate paradigm shift of our time
Last night the AI and I teamed up to write bedtime stories on demand and on the fly for whatever themes my daughter mused. I think about a billion families would enjoy an app that did that.
I do think there's a good chance this stuff will have that level of impact, I just don't think it has yet.
(Though I'm not convinced it will. If you wanted to be a skeptic, you could argue that we're already in a huge content glut; there's basically infinite content available for almost free. So does radically lower the cost matter that much?
(Maybe in low-level ways, like it'll increase the abilities of small indie studio to produce high-level content. But to your average consumer, maybe it's not noticable.)
> So does radically lower the cost matter that much?
It's not about lowering the cost, it's about generating content that fits your specific preferences in real time. Forget on demand streaming of off-the-shelf content, people will want on demand content generation.
And then, as is tradition, the next step after that is giving you content you did not even know you wanted. That can be perverse marketing, feeding you sensationalized stuff that keeps you hooked 24/7¹ but the better alternative is something that is tailored to your general tastes, genres, writing styles, etc. so that you receive the content that is best received by you at that particular moment (matching your mood, your goals, your style)
Maybe so, but unless there are a billion families out there who are already privileged enough to have internet access and are willing to pay for such an app, I don't see it happening.
And yet people aren’t doing this so I guess something is still missing. Awareness? UX? Something else? All of the above? The company that figures it out might earn a lot.
See what is happening with StableDiffusion, a model was released opensource, performance in the same league as closed source, usable on consumer hardware and (non AI) techies start to modify it. The biggest steps/modifications are by specialists no doubt, yet still opensource, but others are happily glueing parts together to make something else. The key really seems to be access, an hosted API is rather hostile to innovation, using and especially experimenting is expensive, modification can only happen within whatever the API allows. For the tech to get bigger and more noticed faster more people need to be able to tinker with it.
The real reason LLMs are hard to sell is alignment issues. The models reflect our biases. As an app developer, are you willing to risk it? You app—using GPT-3—might output racist remarks and then you're in trouble.
You can run stable diffusion on a MBP and produce images in under a minute. It's training these models that takes the crazy GPU power - running them is quite reasonable.
Still can't run GPT-3 or even GPT-J locally though, which is what the article is about. Learning takes a whole datacenter with actual terabytes loaded into VRAM, sure, but even running it requires you to have enough space on the card to actually load the model. Which is usually still in the 20G+ range.
Stable diffusion is about the only one of these useful groundbreaking models that can run on normal hardware to some extent, and even that's extremely limited with only like what, 256x256 being possible with a 6G card and 512x512 on a 10G card? So thanks for pointing out the one partial exception.
The training with a beefy GPU from vast.ai (RTX 3090 with 24vram) and Im generating the images with a GTX 1080 with 4vram, so no need for 6 or even 10 GVram from my testing
Do you? Prove it. Your textual output here will do nothing to prove you know suffering and joy any more than GPT-3 would be able to.
> Will it produce art?
Is a poem art? I think most consider it to be.
For what it's worth, I don't like where AI generated content is going either, but to disregard it as worthless (instead of simply lacking conscience) is not helpful.
GPT-3 has learned many identities, it's not lacking a perspective but quite the opposite, it has too many perspectives and you never know who you're talking to. Unless you prompt it with an identity. Recently it was used to run "in silico sampling" [1], a virtual poll where the respondents were GPT-3 prompted with personality backstories.
So it should not be seen as an agent, it's not like us in that regard. It is the synthesis of all human culture and can assume any hat.
https://theblorgonchronicles.blogspot.com