Are there any difficulties in generating a program in standard languages in Python? Did you choose a DSL because neural network is sensitive to the output programming language?
It turns out the full grammar of Python (and almost all real programming languages) is quite large; this is very early and new work in neural program synthesis, and so we chose a pretty limited DSL to make sure that we could at least solve this one before moving on to more general ones that contain state, conditionals, for-loops, etc. In theory however, we can apply the exact architecture to Python programs and see what happens. We haven't tried yet. :)
Why does the final example in figure 14 fail completely? The outputs are correct as far as they go, but they're all incomplete.
Is it because the scoring metric has a point where enough of a good start outscores an alternative in the beam search that could lead to a more complete solution? In non-trivial real-world examples, would the be a major problem?
Theorem solving is very closely related to program induction (we just change the grammar). Just as with Python, the underlying search space would be incredibly large, and while in theory, we could simply change the DSL and it should work, it'll probably involve a few more iterations of the model or other insights to see this to fruition (but it's definitely not impossible).
It looks like this is a more comprehensive version of FlashFill (can do more tasks), and it is based on deep learning instead of previous rule-based techniques in FlashFill.
As much as I'd love a program to infer my intent to a specific language, I wouldn't think it would help me as much. In my everyday work, most of the time I need to handle edge cases, which need specific instructions for specific scenarios. Once the logic is figured out, the writing part is just minimal.
Machine learning/AI can learn from common cases to infer logic from intent. For such special cases, it may not be as common and need instructions. By then, I may as well write code by hand.
@suryabhupa How similar is this work to the Grammatical Inference field? There has been a lot of work over the years in specification inference which feels similar. Many of the studies in specification inference learn automata representations of object interactions. I know there have been other application grammatical inference in Software Engineering as well.
Program synthesis is grammatical inference grown up, and statistical approaches are being experimented w/ for modern synthesis just as they were for the genetic programming & grammatical inference era. (I believe even now at the SAT solver level today.)
At a quick skim, this seems fun more as (1) an experience report of jumping on the DNN train instead of other ML algs and (2) more intriguing to me, the training formulation (irrespective of neural nets). Dawn Song's recent explorations here also sounded pretty interesting in terms of bridging logical synthesis of general programs with statistical..
Grammatical inference learns a grammar from a set of examples, where here it seems the paper is learning a program (a derivation in the grammar) from examples.
Which Dawn Song paper are you talking about here? I think among all the recent approaches proposed recently on neural program induction, this is the first one that is end-to-end trained and learns only from input-output examples without any hacks!
