Show HN: Synthesize TikZ Graphics Programs for Scientific Figures and Sketches

[potamides]

https://github.com/potamides/DeTikZify

Creating high-quality scientific figures can be time-consuming and challenging, even though sketching ideas on paper is relatively easy. Furthermore, recreating existing figures that are not stored in formats preserving semantic information is equally complex. To tackle this problem, we introduce DeTikZify, a novel multimodal language model that automatically synthesizes scientific figures as semantics-preserving TikZ graphics programs based on sketches and existing figures. We also introduce a Monte Carlo Tree Search-based inference algorithm that enables DeTikZify to iteratively refine its outputs without the need for additional training.

[nequo]

This is brilliant work, OP! A great use case for machine learning.

Why does the arXiv non-exclusive license preclude the inclusion of arXiv figures in the published data set? I don't see how the conditions set by the license imply that:

    I grant arXiv.org a perpetual, non-exclusive license to distribute this article.
    I certify that I have the right to grant this license.
    I understand that submissions cannot be completely removed once accepted.
    I understand that arXiv.org reserves the right to reclassify or reject any submission.

https://arxiv.org/licenses/nonexclusive-distrib/1.0/license....

[wrs → nequo]

Uploading to arXiv gives distribution rights to arXiv, not the users of arXiv.

[wrs]

Shout out to an open source model that includes the dataset crawling and training code, not just the weights and inference.

[tmaly]

I am a little confused by the use cases.

Would someone feed the model a hand drawn scientific figure and it would output the TeX to create the figure digitally?

[potamides → tmaly]

The use cases we see are (i) assisting researchers in creating scientific figures from scratch (sketching them om paper is usually easy but actually developing them can be daunting), and (ii) enabling semantic edits to existing figures stored in lower-level formats by synthesizing high-level graphics programs that generate them.

[abdullahkhalids → tmaly]

When I am teaching online, I write on a digital board. Lots of math, figures here and there. I have attempted to convert some of my lectures into written notes in the past. Something like this would help quite a lot with the conversion.

Would be insane to have an end to end product. One that transcribes my audio, at the same time annotating it with the equations and figures I have drawn on the whiteboard.

[Xerox9213 → tmaly]

As a teacher who regularly rips off large expressions from textbooks and test generators with MathPix, this would allow me to also capture diagrams into my exercises.

[dimatura → tmaly]

Yeah, I think that's one of the primary use cases. As a researcher I've looked at using Tikz in the past but it's a nontrivial learning curve. Something like this seems useful to at least get a starting point to tweak.

[a_e_k]

As someone who's used TikZ for paper figures recently, this is brilliant! I could see this saving a lot of time spent both on studying the documentation and on trial-and-error to coax TikZ to render the figures the way I want. I would absolutely use this.

[taopai]

Awesome! Really awesome.

I hope that German secondary physics and chemistry teacher who has an amazing free pdf book about tikz sees this.

This was such a great and didactic book to real tkiz. I can't find it right now, but must be somewhere.

[abdullahkhalids]

Excellent work. I see it transcribes basic text well. Do you have any plans of creating a model for handwritten math to latex math?

[potamides → abdullahkhalids]

We have some rough ideas to support multiple "backends" depending on the input, so that could definitely happen. That being said, tools like LaTeX-OCR [1] already claim to (at least partially) support this.

[1] https://github.com/lukas-blecher/LaTeX-OCR