Why? That simply depends on how the code was written. The magic constant may be chosen randomly, or based on previous guesses, or generated by another program, etc.
Humans are good at figuring out why something works then creating a higher level abstraction that can be applied in lots of different situations. This saves us time because we don't have to re-derive the rules or constants each time we encounter a similar problem. We figure out "why" something works so that we can understand when to apply our abstraction to other problems that might or might not be similar in a meaningful way.
For a fast enough computer, there is little reason for it to develop any kind of high-level abstraction. It can re-derive the optimal solution whenever it needs it and the benefit is that it might be able to avoid local maxima that we would get stuck on because it doesn't try to apply an abstraction that only partially fits.
So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".
> So when we look at something like this and say "we can't know why the computer did it this way" it almost doesn't make sense. The computer did what it did because it figured out that was the best solution - it doesn't understand the concept of "why".
This idea is what I don't agree with.
At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions. This implies that, given enough resources, a human equipped with a debugger and an understanding of the instructions e.g. algorithm could reverse exactly why and how a program arrived at a particular result.
> it [the program] doesn't understand the concept of "why"
Oh yeah, it doesn't "understand" the concept, but that doesn't mean it doesn't make decisions (branches) based on some parameters (inputs or intermediate values). In other words, there is in fact a "why"! So if we can step through the sequence of branches executed and their results, we can understand "why" even if the program itself can't do that.
Once the number of instructions approaches a certain point, it will increase the cost of human analysis drastically.
From my previous work performing reverse engineering, I could take comfort that I was reconstructing algorithms designed by humans, compiled and assembled by programs written by humans, with really pleasantly un-optimized properties (frame pointers in some assemblies, loading and saving registers at the start & end of functions, logically-separated functions). My job would be much harder if I had to deobfuscate code that was optimized without a strict ruleset.
After enough spaghetti assembly, it would be too time-consuming to reverse engineer code except for the most profitable enterprises (interoperability, vulnerability research for a very important bug).
I'm using the following mental model of what machine learning-generated code would look like: https://news.ycombinator.com/item?id=8092359 (an evolutionary algorithm designed a circuit that is extremely difficult to analyze but is optimal at achieving its narrow purpose).
> At the most basic level, the entire "thought space" of a computer lies within the instructions fed to it by a human programmer. Until we can figure out how to build a general AI, whatever the computer decides to do or ends up doing is completely bounded by the content of the instructions.
(emphasis mine)
How would this change if you built a "general AI"? Assume I don't believe that a computer, even a general AI, is ensouled.
If we assume that a general AI can "understand" things in general and can "learn" over time, there is nothing stopping it from understanding the instructions it consists of, and subsequently learning how it can dynamically reprogram itself. If we extend that further, the program could also potentially obfuscate its activity by detecting logging or debugging activity.
In a narrow sense, yes. But if the program "learns" self-modification rather than being explicitly told how to do so, and then uses this knowledge to perform operations that were not included in the original instructions, I'd call that a general AI.
Also the cognitive gap between a AI and a human brain has to grow and transform unpredictably in the next decades. We may be puzzled to understand how it "thinks" the same way we still don't get exactly how DNA operates.
But then the decryption key, or at least the code to generate it, would have to be embedded in the source. This is how malware crypters work on a basic level. Besides, a program that suddenly decides to encrypt itself means we essentially have a general AI on our hands.
It definitely will get harder for us to reason about what AI software is doing over time, but it's way too early to start comparing ML algorithms to the complexity of DNA. We may not understand exactly why a NN is using certain values to get the result, but we can reverse that logic through the use of debugging and logging.
But there's no "source code" here to encrypt. The model in this paper isn't even self-modifying.
The only "source code" are the learned network weights. These are the only parameters that describe the encryption function. You can still run the model forwards or backwards.
We already don't understand neural network weights on an intuitive level. No need to encrypt them. :-)
Not entirely true with ANNs built through deep learning. They can and often do exhibit unintended behaviors. When you start getting into a system of connected sub neural networks, the "why" can be obfuscated by the hidden layers.
