The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases.
This feels a lot like screaming at a child for imperfect grammar.
You're misunderstanding my point. Nobody's screaming at anything. Whether this thing is impressive isn't at issue. It's utterly astonishing.
I'm trying to figure out whether copilot in its current form is a tool that will be useful to me in my job. (I'd be able to do this evaluation properly if they'd just let me on the damned beta.)
Nearly right isn't good enough for this afaics. In fact, I expect there to be a slightly paradoxical effect where nearly-right is worse than obviously-wrong. An analysis of a piece of code like I did above is time consuming and cognitively taxing. An obviously wrong solution I can just reject immediately. An almost-right (or at least vaguely plausible) one like these takes thought to reject. Much more thought, in this case (for me, at least) than just writing the thing myself in the first place.
Edit: BTW, I don't get what you're saying with
"The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together."
The first one is completely (if subtly) wrong. It's supposed to swap two bits but it sets them to the same value. There's no interpretation of the description in which that's correct.
The second one is definitely not "the right idea". It tries to do it with string manipulations, which (regardless of the fact that it does so incorrectly) is completely the wrong approach. This one is actually "better" than the other in the paradoxical sense I mentioned above, because I could reject it the moment I saw it convert the number to a string.
> The second one is definitely not "the right idea". It tries to do it with string manipulations, which (regardless of the fact that it does so incorrectly) is completely the wrong approach. This one is actually "better" than the other in the paradoxical sense I mentioned above, because I could reject it the moment I saw it convert the number to a string.
In this case string ops are a worse idea, but as I said before, this is not generally true of Python, at least when using CPython. Eg. the string method is significantly the faster in this example:
# https://stackoverflow.com/a/20918545/1763356
def reverse_mask(x):
x = ((x & 0x55555555) << 1) | ((x & 0xAAAAAAAA) >> 1)
x = ((x & 0x33333333) << 2) | ((x & 0xCCCCCCCC) >> 2)
x = ((x & 0x0F0F0F0F) << 4) | ((x & 0xF0F0F0F0) >> 4)
x = ((x & 0x00FF00FF) << 8) | ((x & 0xFF00FF00) >> 8)
x = ((x & 0x0000FFFF) << 16) | ((x & 0xFFFF0000) >> 16)
return x
# My ver
def reverse_format(x):
return int(f"{x:032b}"[::-1], 2)
Python's dynamic object overhead (and to a lesser extent, interpreter overhead) makes a lot of seemingly-expensive operations not matter very much.
Well, that also seems like the wrong question to ask. Whether it's currently useful to you for writing short algorithms, rather than as the non-programmer's API interface it's primarily marketed as, seems about the least interesting take-away for it. We'll get to smoothing over the cracks later, once it's not a capability we literally just discovered exists. Heck, Codex is already not SOTA for that, AlphaCode is.
It may not be the question that interests you but who are you to say it's the "wrong question" for me to ask? I want to know if I'm on the verge of having access to a tool that is going to transform the way I do my job, as people keep claiming.
It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code.
That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion.
Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.
>It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code.
Funny, today I was just thinking of people's tendencies to dismiss AI advances with this very pattern of reasoning: take a reductive description of the system and then dismiss it as obviously insufficient for understanding or whatever the target is. The assumption is that understanding is fundamentally non-reductive, or that there is insufficient complexity contained within the reductive description. But this is a mistake.
The fallacy is that the reductive description is glossing over the source of the complexity, and hence where the capabilities of the model reside. "Generating maximum likelihood token strings" doesn't capture the complexity of the process that generates the token strings, and so an argument that is premised on this reductive description cannot prove the model deficient. For example, the best way to generate maximum likelihood human text is just to simulate a human mind. Genuine understanding is within the solution-space of the problem definition in terms of maximum likelihood strings, thus you cannot dismiss the model based on this reductive description.
The difference between me and you is that I implement neural nets professionally. Here is one of my (non-professional) open source projects: https://NN-512.com
I'm sure if you understood what the transformer was doing, you would be less impressed.
This is the wrong context to go with an appeal to authority. I know what the transformer is doing, I've also developed neural networks before (though not professionally). Your experience is working against you in developing your intuition. There's another common fallacy that because we're somehow "inside" the system, that we understand exactly what is going on, or in this case what isn't going on. Language models are composed of variations of matrix multiplications, but that isn't a complete description of their behavior. It's like saying because we've looked inside the brain and there's just electrical and chemical signals, the mind must reside somewhere else. It's just a specious argument.
It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.
If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it.
It's a transformer. Do you understand what that means? It's just matrix multiplication.
It generates maximum likelihood token strings, based on its training data.
It doesn't "understand" what those token string mean.
You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training.
This feels a lot like screaming at a child for imperfect grammar.