On: Competitive Programming with AlphaCode

[algon33]

How suprising did you guys find this? I'd have said there was a 20% chance of this performing at the median+level if I was asked to predict things beforehand.

[machiaweliczny → algon33]

I am surprised, as recently OpenAI had ~25% of easy problems and ~2% in competitive problems. Seems like DeepMind is ahead in this topic as well.

Actually I think Meta AI had some interesting discovery recently that could possibly improve NNs in genral, so probably this as well.

I am not in field but wonder if some other approaches like Tsetlin machines would be more useful for programming.

[algon33 → machiaweliczny]

Somehow I have never heard of Tsetlin machines before this. Are you talking about this https://ai.facebook.com/blog/the-first-high-performance-self... result by MetaAI?

[machiaweliczny → algon33]

Probably not. Tsetlin machines have logic expressions instead of weights in NN, so it's easy to interpret them. I guess some meta algorithm could maybe work on top of them.

https://arxiv.org/abs/2102.10952

EDIT: Missread, I meant this from meta https://arxiv.org/abs/2105.04906 - not sure how much it's productised

[hackinthebochs → algon33]

I didn't find it very surprising, but then I tend to be more optimistic than average about the capabilities of transformer models and the prospect of general AI in the relatively near term.

[marcusbuffett → algon33]

I would have guessed around the same chance, this was surprising to me after playing around with copilot and not being impressed at all.

[baobabKoodaa → algon33]

I would have said there is a ~0% chance of this happening within our lifetimes.

[Isinlor → algon33]

There is a prediction market called Metaculus.

TL;DR In 2020 community of 169 people and the best forecasters were assigning ~15% that it will happen by July 2021.

More specifically, on Dec 31, 2016 in partnership with Center for the Study of Existential Risk, Machine Intelligence Research Institute, and The Future of Life Institute they asked:

How long until a machine-learning system can take a simple text description and turn it into a program coded in C/Python?

https://www.metaculus.com/questions/405/when-will-programs-w...

First 19 forecasters in March 2017 were predicting mid-2021, the best forecasters were predicting late 2024. When the question closed in 2020 the community was predicting January 2027 and the best forecasters were predicting March 2030.

The question resolved on July 2021 when Codex was published.

Community and the best forecasters were assigning ~15% that it will happen by July 2021.

I'm currently 14th best forecaster there and I was predicting 33% before July 2021. It was my last prediction, and it was made on October 2018.

I'm also predicting 75% that we will have AGI by 2040 as defined in this question:

https://www.metaculus.com/questions/3479/when-will-the-first...

20% that it will happen before 2030.

There is also stronger operationalization:

https://www.metaculus.com/questions/5121/when-will-the-first...

My prediction here is 60% before 2040 and 5% before 2030.

I have also "canary in the coal mine" questions:

When will AI achieve competency on multi-choice questions across diverse fields of expertise? Community predicts 50% before 2030, I agree.

https://www.metaculus.com/questions/5276/ai-competence-in-di...

When will AI be able to learn to play Montezuma's Revenge in less than 30 min? Community predicts 50% before 2025, I think 50% before 2027.

https://www.metaculus.com/questions/5460/ai-rapidly-learning...

[algon33 → Isinlor]

For some reason I forgot to check metaculus for this. Thanks for the reminder.