On: DrEureka: Language Model Guided SIM-to-Real Transf...

[refulgentis]

Every single second of every example has a handler holding a leash - and not just holding it, holding it without any slack.

Blindingly obvious interference from Ouija board effect.

I don't mean to denigrate the work, I believe the researchers are honest and I hope there's demoes outside the published one. Just, at best, an obvious unforced error that leaves open a big question.

EDIT: Replier below shared a gif with failures, tl;dr this looks like two different experiment protocols, one for success, one for failure. https://imgur.com/a/DmepBVU

[Imnimo → refulgentis]

This sample on Twitter shows how other controllers fail:

https://twitter.com/JasonMa2020/status/1786433841613390023

I agree it's hard to tell whether the controller learned with DrEureka would be sufficient without the leash, but I'm at least convinced that the leash is not sufficient to hold a robot on the ball without a decently competent controller.

[refulgentis → Imnimo]

Oh my, that looks quite damning. https://imgur.com/a/DmepBVU

The good case leash is held taught at half the distance of failures, at a parallel angle to the bot and orthogonal to failures.

The failures all held with slack, on a leash held at 2x the distance of successes, at an angle orthogonal to the bot.

(do correct me, we're seeing opposite things, and those are very small and I last took physics...16 years ago :< )

[Imnimo → refulgentis]

Hmm, I do see what you mean.