Hacker News new | past | comments | ask | show | jobs | submit login
Launch HN: Optic (YC S18) – Automate Routine Programming
183 points by addcn on July 18, 2018 | hide | past | favorite | 48 comments
Hi — I’m Aidan YC (S18), the founder of Optic (https://useoptic.com). We’re building a smarter code generator that helps developers write and maintain the tedious parts of their codebase. Optic is fully open source (https://github.com/opticdev) and all our code is MIT licensed.

A really simple use case that Optic enabled for some of our early users is keeping their backend and frontend in sync. Optic can read the endpoints in a backend and use that to generate the HTTP requests on the frontend. It even maintains that code over time so if the backend is updated, a pull request is generated to update the networking code. You can test that here: https://github.com/acunniffe/optic-networking-code-demo.

We have also had teams configure Optic to:

  - Write/Maintain standard tests for their React components
  - Migrate to GraphQL by generating wrappers for each endpoint
  - Generate/Maintain CRUD routes from models
  - Wrap their Tensorflow models in APIs

There’s a checkered history of optimism and failure around automated programming and many tools have promised to make developers' lives easier. In each generation of programmers there’s an acknowledgement that much of the code we write is routine, but none of the solutions have caught on. I failed once before when I founded Dropsource. While we built that product into one of the most popular tools for non-programmers to build mobile apps, we failed at our goal of building a tool we ourselves or other developers would use.

I left the company and went into research mode for around 18 months. A lot of the automated programming projects do their functional job well, but when it come time to integrating into a developer’s workflow they are littered with tradeoffs. Automation isn’t worth it if it means giving up control, rewriting lots of code, rebuilding your app visually, describing your project in some foreign dsl, or becoming tied to a vendor. Many of these projects haven’t caught on because the tradeoffs of using them have been too high.

When I started working on this problem again I was determined to get the developer experience right. For me this meant making Optic:

  - work with your existing code 
  - plug in to your favorite IDE
  - useful throughout the lifetime of your project

I knew that whatever product I ended up building would need to interface with source code so I started building an API for code that let you:

GET JSON objects describing different types of code. Powered by a regex like pattern matcher we wrote that walks AST Trees.

PUT new values of the same shape back to update the code. This was the hardest part by far. We ended up training decision trees on different programming languages using the raw code and the resulting AST tree as input. We use these models to regenerate code in the smallest sections possible so formatting and manual changes are preserved.

POST new code into the project. This was actually the easiest. A generator can be reformulated as a Parser + Mutator so we bootstrapped all the generation in Optic by combining the GET / PUT functionality.

This API was really powerful and it became the foundation of Optic. Generating code, doing transformations, syncing projects — all Optic’s major features are built on top of this API. It’s such a solid foundation (you get parsing, generating and the round-trip problem in a box) that I believe a lot of new meta-programming tools will be built on our platform. Our work is open source (https://github.com/opticdev) and there will always be a free-forever version available. We plan to make money from some specialty features aimed at larger teams.

It’s still the early days but we hope to build Optic into a valuable open source resource that improves the workflow of developers and their teams. I’m looking forward to hearing your feedback, experiences and ideas for tasks we should automate. Thanks!




Looks very powerful especially if you can popularize it. Maybe mention the plugins and other platforms besides MacOS on the website. Also the chat thing wouldn't allow me to click a reply on my Nexus Android phone.

I wonder if the parsing, schemas and the transformations etc. could be a starting point for an advanced neural network AI programmer. Perhaps if you could combine it with an interactive system that converts text to schemas.

For example, say you had a way (a skill?) to convert a schema to a web form. Then the AI would need to handle speech-to-schema-change such as "add a field for Middle Name". Then perhaps there is a skill to go from the web form to a save data request and another to accept the request and another to update the database schema etc.

But the AI might come more in handy for interpreting more complex schemas such as nesting, grouping or workflow etc.

But besides an advanced domain specific AI it seems like this foundation of parsing, schemas and transformations would be a good start for speech-based or other higher level program generation tools. I can envision a web page that generates a CRUD application from mobile front end to Firebase back end and then spits out the code that you can then add more advanced features to.

So it seems a smart strategy to embrace open source because that can allow other developers to add mappings that you can leverage for your own app or program generation tools which you can sell.

To me the interactive web-based CRUD application generator is low hanging fruit but more interesting would be to combine your tech with a Google Duplex style interactive speech based AI programmer front end. Probably only Google and a relatively small number of companies could pull something like that off though.


Thanks for sharing all these awesome ideas. I can't say I know enough about NLP to evaluate how feasible some of your speech to code ideas are, but I do think Optic is a great base to any project trying to do meta-coding.

I'd love to see a 2-way code generator built on Optic for a simple crud builder like you describe. It could let people choose the libraries they want to use from the list and design their schemas, routes and queries at some higher level of abstraction. Unlike the current generator tool you could actually make changes to the high level models and get Optic to update the underlying code for you even after you make changes.

That whole field could be huge and I think Optic's major contribution will be abstracting away all the dirtiness of code gen. People who build this next generation of tools could interact purely with Swagger-like-JSON.

Is this something you'd be interested in collaborating on?


I think ordinary NLP wouldn't get you very far at all, but it seems the Google Duplex people have some advanced tricks. But its still probably a bit of a stretch.

A two-way code generator for CRUD applications is a lot more realistic for a start. I am interested in trying to collaborate on that, although I can't promise how much time I will have because of other obligations.

But it would be fun to give it a start at least. You started describing something, maybe if you want you could elaborate a little bit more about how you think this type of tool would work in a document online somewhere or just in a reply. It would be fun to play around with Optic with that type of goal. I will need to read the docs and experiment some to make sure I understand the system better before I get too far along in coding something. Mainly I use Node and JavaScript these days so theoretically that could work for a web-based CRUD builder.. I guess by talking to the Scala server over REST? I will need to read your docs.


I agree, two-way crud is the best starting point.

Optic React / JS for all its GUIs. We’re planning to have microeditors that pop up and help you work with specific kinds of code so this might just run from within Optic somehow.

All fun ideas to brainstorm. Let’s take it offline. Email me at aidan@useoptic.com

Excited to chat / collaborate


Optic looks amazing, congrats on shipping! Do you think it can be integrated with visual programming languages IDE development? There are a lot of such things in the IoT, industrial and laboratory control, game Dev, media synthesis industries and they all suffer from the limitation of poor coupling between generated code and the front end graphical elements as the type checker and static analysis on the graphical elements level have to match the syntactic rules of the language constructs.

We use it for algorithmic trading here at KloudTrader and this is an issue we suffers from. Looking forward to adding Optic to our CI pipeline.

https://KloudTrader.com

Business rules engines would be another really great application. For a lot of IFTTT type of applications, if the rules can be compiled into finite state machines instead of simply interpreted, that would be a great boost for productivity and efficiency!


Thanks! Appreciate you sharing your ideas. I started out in this space so I love the idea of solving some of these old problems.

Absolutely agree there are some meta-programming applications for Optic. It's always been difficult to build graphical tools that generate code -- everyone basically reimplements string concatenation / static analysis over and over again and then they can't handle even the slightest manual change.

Ideally visual tools would tap into Optic API to parse the code, display it, then press user changes back into code. This would allow users to add custom code as well and not break your abstraction. Our source of truth is code so there's no dsl or model saved on your end anymore.

KloudTrader looks great. Let's chat offline about your requirements and I'll make sure to document / open some of the functionality you'll need.


Nice podcast episode from earlier this year where Aidan talk about his work: http://futureofcoding.org/episodes/21


Thanks for sharing this. Steve Krouse is one of the best thinkers right now when it comes to novel approaches to programming. I encourage everyone interested in this space to check out his work.


I recently started listening to his podcast and I'm digging it. Any particular episode you liked?


One of my personal favorites was episode 11 about how React was created. The dawn of a new abstraction is always a good story. http://futureofcoding.org/episodes/11-how-reactjs-was-create...


(Thanks Aidan!)

My favorite recent episode was futureofcoding.org/episodes/26.


Hey, something where my two cents might actually be valuable :)

I created the VS Code plugin for Optic: https://github.com/opticdev/optic-vscode-plugin. It was actually my first time writing a text editor plugin, but between the VS Code docs, Optic SDK, and Optic plugin spec, it was surprisingly easy. The information just gets sent through web sockets under the hood, but it was all just basic javascript callbacks and some VS Code specifics for finding ranges, etc.

Optic itself isn't particularly useful for me as a solo developer building small website frontends, but I must say I was a little jealous of those that this is geared towards because even developing the plugin gave me a couple of those "magic" moments. Getting to make a change to just some javascript object's property and then get to keep it in sync was pretty satisfying.


Thank you for contributing our first outside code! Let's talk about some use cases that might help you as a solo frontend engineer. I'm sure we can figure out a few things to make your life easier.


This looks really interesting. I hope it's successful.

What kind of AST are you using? Can you translate code between languages?

Also, what do you think of the Graal/Truffle project?

(BTW I'm getting 404 at https://useoptic.com/docs/what_is_optic/what-is-optic/ )


Thank you! We have a common AST format we use as the representation of every language we support. You add new languages to Optic by writing a mapper from their AST to ours. It should work with pretty much any context-free AST. Here's what that looks like for JS https://github.com/opticdev/es7-parser/blob/master/src/main/...

We can't translate code between languages but we do the next best thing by moving the meaning of that code around. ie backend is Flask and frontend is JS. We can generate JS code that connects to the Flask API automatically.

Graal is pretty cool, and I'm happy to see a big org backing a polygot tool. I'm most interested in Truffle and looking into ways it can make Optic better. Have any ideas?

Thanks for catching the 404! I'll look into it ASAP


How much work is it to add a new language if you have to get it into your own format? We use some esoteric languages (Clojure and Reason/OCaml) so I don't expect Optic to support our use case any time soon, but I'd be curious for e.g. Python/Ruby.

Also, a couple of other question that come to mind:

1. Jared Forsyth gave a great talk on avoiding breaking changes as a library author by using code-mods - would Optic be able to wrap up some of the changes it makes to my library code as a code-mod that I could ship with a new version of my code to auto-upgrade all my dependents? That could significantly help with some of the higher-churn libraries (some people are iffy about this though).

2. Have you considered using something closer to lsp for the editor protocol? Emacs support would be nice, and that'd be a good way to jumpstart the implementation.

3. It actually doesn't look too bad to implement the plugin for e.g. even without lsp (https://useoptic.com/docs/authoring/adding-optic-support-for...). Looking at the 'search' event though, why use `///` instead of picking up `cmd-f` (or `C-s` in emacs)? I was a bit confused about the interaction model.

Optic looks pretty interesting, best of luck!


All you have to do is pick an existing parser that works well and map the output AST to Optic's AST. I'm exploring building an Antlr extension so any grammar in their library could output in Optic's AST but that's still not implemented. Every AST have a notion of type, range, and properties which can either be primitives, arrays of child nodes, or individual child nodes. We have Python and Ruby on our roadmap for the next couple of weeks.

1. We've been exploring migrating one Optic skill to a newer version. So far it seems promising for the cases you'd expect a code-mod to help with, but if you completely change the API and deprecate some stuff it can point out the changes but won't be able to help you -- no magic here.

2. I looked into language servers (lsp) but the implementation was a lot more involved and I found bugs in some of the IDEs that claim to have full lsp support. As the protocol matures and we add more functionality to Optic I expect to see lsp become our main avenue for IDE/Optic interactions.

3. We wanted to make it really natural to ask Optic a question. We thought '///' would be unique and easy to remember, but we're open to changing it. It's nice that '///' also evaluates as a comment too because before we used it sometimes characters in your search would cause the AST parse to fail. We're open to any ideas here -- a lot of users have said it's funky.


Two things: first, it's very impressive that you're doing the hard work to integrate this with IDEs and such. To me that shows that you're really serious about getting real-world uptake. There are a few tools and systems that do what you're doing but most of them never seem to make it out of some tiny niche and remain academic.

Second, your Marvin sub-project, if I understand it correctly, is just awesome. Congratulations.

In re: Truffle, and lang-to-AST conversion in general, I've taken a winding path over the better part of the last twenty-odd years. The closest thing to a serious idea I have to offer is this: Use Lisp as your inter-language AST.

I'm grinning but I'm not quite joking. ;-)

Think it through. What are the pros and cons?

(I can't help but mention, I'm working on a system that may one day "eat" other software and translate it into essentially an AST. The AST is actually just code in a language called Joy that resembles Forth+Lisp. The language is very simple and can be implemented on top of other languages in a few days. So I can (eventually) read in other code to this Joy/AST, clean it up, then run it on whatever substratum makes sense for a given use-case. Joy code is easily refactored, and can be fairly easily converted to logical relations and then you can use e.g. Prolog or miniKanren to search for optimizations. This is sort of "one step beyond" type inference, moving into the more esoteric but useful things where you can do partial evaluation and "super-compilation" (chase down alternate control-flow paths) to improve the abstract code and emit specialized e.g. assembly or LLVM IR or JVM bytecode or whatever.

My motivations are two: I want to teach normal people to program, and I want to contract the total amount of [distinct] software in the world, eliminating incorrect code and collecting, refining, and curating all correct code into a single unified codebase.

The Joy notation is easy enough to teach to normal people. I've created a simple GUI that presents a Joy environment where users can build new commands (Joy code). The system uses type inference to prevent the user from executing commands that won't work with the current system state, and to prevent the construction of new commands that won't work at all. It's impossible to make incorrect code in this (virtual) world.

The Joy language is highly-factorable, enabling a kind of "semantic compression". It's also Categorical (in the sense of Category Theory), meaning that programs in Joy represent abstract computations over categories and can deliver multiple correct programs for the same Joy expression. For example, the type inferencer works by evaluating Joy expressions over a category of stack effects and type variables to generate the stack effects of the expressions. Conal Elliot has a paper "Compiling to Categories" where he's exploring this sort of thing using Haskell, converting it to a "point-free" Joy-like notation and then implementing categories for dataflow diagrams and circuit descriptions and other cool things, to produce those all from the same original Haskell code.

I'm hopeful that I've got something here. :-) I've been sitting in a room in front of a computer for about a year and I'm just at the point where I have to go out and talk to people about it. Egad.

Anyhow, that's part of why I'm so impressed with what you've accomplished (and are no doubt going to accomplish!) I look at what you've done already and I think, "Daaaamn, this dude is running circles around me." Makes me feel sheepish, but I'm stoked for you! You're bringing the fire down from Olympus my Promethean friend, don't doubt it.


Thanks for your kind works. I think I just heard someone whisper "memento mori".

If I'm understanding your idea, it sounds like you want to use a compiler to a) collect all the useful/best control flows into some kind of library that you then b) use to generate code when compiling any other language. Like an ouroboros of compilers. Then you want to have some simpler programming interface on top of that which can be expanded into great code?

In my notebook I've been calling Optic an up-piler because it turns raw code from multiple languages into the same higher level abstractions. So on a spectrum of processor instructions -> ideas it's further to the right than the host AST. Sounds kind of similar to what you've been working on for part A.

What confuses me about your idea is how I as a developer would benefit from it? Are you a drop-in compiler that will compile my code into something 'better'? Can you just take some code I wrote and refactor it into a better format that does the same thing as before? Will the way a project compiles change over time? I'm not sure if I want the nails in my log cabin going form 20mm to 30mm overnight. Shouldn't break anything but you never know...

Happy to chat offline sometime aidan@useoptic.com Think we'd have some fun convos


Optic and the (much simpler) thing I'm working on have some similarities but I think they're mostly orthogonal.

To directly answer your question, I don't think developers would benefit from it. If it works as intended, normal non-developers will be able to use it to build simple programs themselves. I could see this putting a lot of mid-level developers out of work.

The "ouroboros of compilers" is a side-effect. The simpler programming interface came first. I realized that other languages could be translated into the Joy language. Higher level abstractions are captured in Joy by defining new combinators. Ideally, the system would form a global Gödel machine with an "oracle" consisting of the world's best and brightest mathematician/programmers.


This is awesome! I'd love if you could do things like help upgrade library versions that have breaking changes--say something like generate a PR to go from React 16 to React 17, for instance.

It would also be cool if this could help generate code in the best-practice style defined by third-party libraries. For instance, if the Redux team defined the canonical way to define a selector, it would be handy to be able to have Optic use that information to generate selectors for me. This would take something like create-react-app and break down the best practices into more bite sized chunks.

Really excited about this!


Great ideas. We’ve experimented with doing version migrations before and I think it’s definately something we will have good support for in the next several months.

If you haven’t read about our Optic Markdown it’s designed for what you’re describing. A team could specify their conventions and best practices in Markdown and Optic will use the example code in the generators.


Great work! We are definitely interested in maintaining our React ccodebase using this.

Other than that, one vertical usecase that would be super appealing would be the ability to automatically generate (and maintain) api servers ontop of AI modules. As in if we had a tensorflow model, can we automatically create an endpoint to serve live traffic? I guess more generally, if we can define the JSON input to a function (any python function doesnt have to be AI model), Optic can write all the boiler plate microservice for us!


Thanks! That's a really cool use case. We have another AI company using Optic for something similar and I hope they share their on their blog work soon.

If you can put together an example of the kind of code you want Optic to write for each module I'm happy to help you guys create Optic skills around it. There's also information in our docs on how to do this yourself. If you generated the routes with Optic you'd also be able to consume them easily and keep all your code synced as you update the input your models require.

How often do you find your team writing these microservices / tests?


Reminds me of F# type providers. Maybe more powerful? I think defining everything at the DB makes sense and then you can generate much of the code beyond that and then you only need to write the custom code that a programmer actually needs to do.

Glad you are doing this!


What are your expectations on when you might publish variants for some non-Mac platforms? Which code/modules specifically are platform-dependent? If it is only the GUI, can one already use the engine/JSON API (if available) on some other platform?

The whole idea is still not fully clear to me. After watching the video, it initially looks like some kind of code templating engine. From further reading, I seem to understand it also links many "instantiated" templates together, so that changing one of them can change others; is that correct?

Also, one thing that is still not clear to me: do I have to build such templates by hand, or does the engine somehow automatically infer / detect new templates from an existing codebase?


Hey thank you for sharing your first take. We definately are iterating on our messaging trying to make all your main points clearer.

We have plans to release on windows and Linux later this year. Currently investigating cross platform kits for the GUI.

The engine should run anywhere a JVM does but I haven’t written any docs on that yet. Is a scriptable version of interest to you?

Your initiation is basically right. Optic is a code generator with templates. What’s special about it is that it uses your existing code as input so there’s no manual specification done. You can also manually update the generated code and it still works with Optic. All this combined allows you to “sync” different parts of your code that are related as you said.

You have to specify the skills by hand or import them from our registry. https://useoptic.com/docs/authoring/writing-skills


Yes, a scriptable version would be of interest. Ideally, if there was a docker image I could just easily download and run on my dev machine (and please allow passing an ENV option to limit Java heap size), with a tutorial how to use it and how to add "skills", I might be curious enough to run it. Um; and we're mostly a Go shop, but we have some JS too. Still, given infamously poor/nonexistent generics/metaprogramming support in Go, it might be an interesting niche/target for you. OTOH, as a Go dev, I must admit I'm somewhat allergic to JVM... but having it isolated in docker could be acceptable.


Thanks for your feedback. When we ship a scriptable version Docker seems like a great choice for wrapping it. I'll add an env variable for ya :)

The JVM is somewhat allergy inducing, but I think Graal may solve a lot of these issues. Optic is written in Scala because a) it's a great functional language and b) the JVM is super portable. If we can get Graal working nicely in our builds, binary sizes should drop markably and performance should improve.


This looks great. I'd be curious to see the decision trees, and the training corpus/objective for them.


Thank you. You can find the code for that here: https://github.com/opticdev/marvin

The objective of the training is bootstrappig a prettier. String template (pattern in the code) for each node type.

So we can learn things like arguments are separated by “, “ but if the total # of chars of all of them is greater than X it’s more common to use “,\n”. Stuff like that.

We had LSTMs for a while but they performed worse than a decision tree with a few key features as input.


Thanks, sounds very interesting. I'll take a look.


Where are the skills and schemas stored? Are they in the main GitHub repo somewhere or separate JSON files or what? Is it possible for me to add my own transformations?

Is there a package manager or common repository for sharing and searching Optics skills?


We have a registry with a frontend coming soon.

You can also resolve them locally by putting them in your optic.yml file

You can add a skill and containing as many transformations and lenses as your want.

It’s explained at the bottom of these docs https://useoptic.com/docs/authoring/writing-skills


Awesome thanks for the response. Looking forward to the registry. Will there be a cli for the registry?


Yep it's all right there. You can use your Optic account credentials https://useoptic.com/docs/authoring/publishing

  npm install optic-markdown -g
  opticmd adduser


Cool. For some reason that link is a 404 at the moment.


Do you mind emailing me at aidan@useoptic.com with the name of your town and a trace route to useoptic.com

It’s working here so not sure what could be causing this besides an issue with our CDN


For some reason it kept coming up in Chrome with an extra / at the end of the URL.. but its working now.


Got it. We’re using Gatsby and there’s a rewrite rule on our CDN. I’ll take a look tonight and see if I can fix the issue.


I take it this only works for web dev? Might want to specify that.


JavaScript is the first language we have stable support for. I should have made that more clear up front.

Do you see any other areas this kind of tool would be useful? Happy to chat about our roadmap for supporting other languages.

Thanks!


I'd say keeping repos in sync with changes in $RPC for thrift, grpc, jsonrpc etc. could be a good step. Would love to see ability to write backend in go and fe in JS/react w/ auto pull requests for changes to structs.


What IDEs are you currently integrated with?

I'm particularly interested in VS Code and perhaps Intellij out of the gate


Our community built plugins for Atom, VSCode, Sublime, WebStorm & JetBrains. Someone started one for vim but it's still unfinished...hope it's done before emacs

All the plugins do is forward the state (col/row and filepath) to the local Optic server so they're usually <70loc. If you wanted to build support for another one we wrote a spec that all the plugin authors relied on during their development https://github.com/opticdev/optic-editor-plugin-spec/blob/ma...


Could you post some samples of the JSON used to declare new code descriptors?


So we created a markdown flavor called Optic Markdown. We want to meet programmers where they already write about code/styles instead of making them write templates in some dsl. Our markdown relies on HTML comment annotations (so they're hidden) to describe the important parts of a piece of code -- it's kind of like pointing out different parts of a snippet and explaining what they do.

We've been working really hard to build tooling around this because the first few versions were hard to learn/use. Would love to hear any feedback you have on the teaching process. We don't think Optic will take off until we get this just right.

https://useoptic.com/docs/authoring/writing-lenses


Best of luck looks like a great protect!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: