On: Research recitation: A first look at rote learning in GitHub Copilot suggestions

[onionisafruit]

Generally, yes. It’s not guaranteed to do that correctly, but I’ve not seen it get variable names wrong so far.

[tyingq → onionisafruit]

There's an example called "fetch_tweets.py" at the bottom of page on https://copilot.github.com/ that gets it wrong:

  def fetch_tweets_from_user(user_name):
      # deleted some lines here...
      # fetch tweets
      tweets = api.user_timeline(screen_name=user, count=200, 
  include_rts=False)

screen_name=user there isn't right.

It's a nit, but it is interesting how many of the hand-picked examples on that page aren't right. Since they were hand-picked, presumably to show the product off.

[the8472 → onionisafruit]

> It’s not guaranteed to do that correctly

Which is odd considering they could run this as beam search with the checking part of a compiler in the loop.

[0-_-0 → the8472]

Which would be extremely slow

[the8472 → 0-_-0]

Not necessarily. If you emit a batch of suggestions for 1 line and prune them with syntax checks you can already prune them and present them to the user while working on the next iterations.

The real difficulty is getting good checking on partial source code, some languages have better tooling for that than others.