No - emergent properties are primarily a function of scaling up NN size and training data. I don't think they are much dependent on the training process.
Of course they are? If you train in a different order, start with different weights, or change the gradient delta amount, different things will emerge out of an otherwise exactly the same NN.
You can see this out of videos where people train a NN to do something multiple times and each time, the NN picks up on something slightly different. Slight variances in what is fed as inputs during training can cause actually high variation in what is picked up on.
I’m getting decently annoyed with HNs constant pretending that this is all just “magic”.
You're talking about something a bit different - dependence on how the NN is initialized, etc. When people talk about "emergent properties" of LLMs, this is not what they are talking about - they are talking about specific capabilities that the net has that were not anticipated. For example, LLMs can translate between different languages, but were not trained to do this - this would be considered as an emergent property.
Nobody is saying this is magic - it's just something that is (with our current level of knowledge) impossible to predict will happen. If you scale a model up, and/or give it more training data, then it'll usually get better at what it could already do, but it may also develop some new (emergent) capabilities that no-one had anticipated.
Finding unexpected connections is something we’ve known LLMs are good at for ages. Connecting things you didn’t even know are connected is like “selling LLM business to business 101”. It’s the first line of a sales pitch dude.
And that’s still beside the point that the properties that emerge can greatly differ just by changing the ordering of your training.
Again, we see this on NNs training to play games. The strategies that emerge are completely unexpected, and when you train a NN multiple times, often differ greatly, or slightly.
