I was really impressed until I realized that the app is basically a wrapper around tesseract.js, which is the actually cool part. Tesseract has a wasm port that can operate inside of a webworker.
Not saying that the article was being misleading about this, just saying that the LLM part is basically doing some standard interfacing and HTML/CSS/JS around that core engine, which wasn’t immediately obvious to me when scanning the screenshots.
The LLM part is almost irrelevant to the final result to be honest: I used LLMs to help me build an initial prototype in five minutes that would otherwise have taken me about an hour, but the code really isn't very complex.
The point here is more about highlighting that browsers can do this stuff, and it doesn't take much to wire it all together into a useful interface.
Simon - hope you don't mind me commenting on you in third person in relation to the above. Simon is a great explainer, but I wish he would credit the underlying technology or library (like tesseract.js) a bit more upfront, like you.
It matters in this case because for tesseract, the exact model is incredibly important. For example, v4 is pretty bad (but what is available on most linux distros when ran serverside) whereas v5 is decent. So I would have had a more accurate interest in this post if it was a bit more upfront that "Tesseract.js lets you run OCR against PDFs fairly quickly now, largely because of better processors we as devs have, not because of any real software change in the last 2-3 years".
I felt this before for his NLP content too - but clearly it works because he's such a great explainer and one for teasing content later that you do read it! I must say I've never been left confused by Simons work.
It's all subjective. Reading your linked blog made it perfectly clear for me you built this using tesseract.js. No idea what the other guys are complaining about.
I drafted a few longer responses about the feeling that I had after reading your post after that headline but they were a bit unavoidably asshole-y! So really, just something like "with Tesseract.js" in the headline is all I would think could be helpful to people on sites like HN. I do like your writing. But I do enjoy knowing what I'm reading specifically if possible, when it's tech.
You act like you were misled, but the article, within the first few sentences, says he realized the tools are available to do this (including naming tesseract.js explicitly!), he just needed to glue them together. Then he details how he does that, and only then mentions he used an LLM to help him in that process. The author's article title is equally not misleading.
Was an earlier headline or subtitle here on HN what was misleading, but then that was changed to not be misleading?
In the same vein, I'm building a tool [1] to extract tables from PDFs (no OCR yet) and spreadsheets. The end goal is to make it easy to combine data from multiple sources by joining tables and produce some useful reports out of it.
The PDF parsing is done by the excellent PDFplumber Python library [2], the web app is built with Elixir's Phoenix framework and it is all hosted on Fly.io.
I recently built a similar tool except it’s configured to use some deep learning libraries for the table extraction. I’m excited to integrate unitable which has state of the art performance later this week.
I built this because most of the basic layout detection libraries have terrible performance on anything non trivial. Deep learning is really the long term solution here.
I've been trying out alternative versions of this that pass images through to e.g. the Claude 3 vision models, but they're harder to share with people because they need an API key!
There are a ton of potential tools out there like Tabula and AWS Textract table mode but none of them have felt like the perfect solution.
I've been trying Gemini Pro 1.5 and Claude 3 Opus and they looked like they worked... but in both cases I spotted them getting confused and copying in numbers form the wrong rows.
I think the best I've tried is the camera import mode in iOS Excel! Just wish there was an API for calling that one programmatically.
The example on the Tesseract.js page shows it highlighting the rectangles of where the selected text originated. Does this level of information get surfaced through the library for consumption?
I just grabbed a two-column academic PDF, which performed as well as you would expect. If I was returned a json list of text + coordinates, I could do some dirty munging (eg footer is anything below this y index, column 1 is between these x ranges, column 2 is between these other x ranges) to self-assemble it a bit better.
I am the primary caregiver for a blind academic and I can tell you that any work in this area by you will be more than welcome, by both he and I. There is a lot of progress to be made in this area. There are tools, but they are primitive to where it is headed (hopefully in part to your efforts).
You can also use MacOS's OCR capability to create a shortcut that allows you to copy and paste the text out of any region on the screen -- for example, a stack trace someone is showing you in a screen share.
Tesseract is way outdated though, to the point of being borderline useless when compared to alternatives. What’s the current deep learning based FOSS SOTA, does anyone know? I want something that does what FineReader does - create a high quality searchable text underlay for scanned PDFs.
The amazing thing here is that this tool is almost all compiled using LLM. This is very exciting. I have been using GPT-4 a lot lately to make tiny utilities. Things I wouldn't have even tried because of how much effort it takes to get started on those simple things.
I always wanted to make a chrome extension for one thing or another, but all the learning involved around the boilerplate always drained the motivation. But with GPT I built the initial POC in an hour and then polished and published it on store even. Recently I compiled some bash and cmd helper scripts, I don't know either of these enough (do know some bash) and don't have it in me to learn them. Specially the windows batch scripts. Using LLM it was matter of an hour to write a script for my need as either a windows batch script or even bash script.
Oh I even used GPT it to write 2-3 AutoHotKey scripts. LLMs are amazing. If you know what you are looking for, you can direct them to your advantage.
Very exciting to see that people are using LLMs similarly to build things they want and how they want.
Oh I think I see. No, I'm not being paid to promote LLMs.
The point of my blog post was two-fold: first, to introduce the OCR tool I built. And second, to provide yet another documented example of how I use LLMs in my daily development work.
The tool doesn't use LLMs itself, they were just a useful speed-up in building it.
My reasoning: There are hundreds of billions of dollars at stake getting the wider world to embrace LLMs on a long-term basis. If I were a VC, or Nvidia/OpenAI/MS marketing person, I'd be paying trusted names such as yourself to post about using LLMs. That coupled with the loose link between OCR and LLMs in your latest post created an itch I thought was worthy of a scratch.
Not saying that the article was being misleading about this, just saying that the LLM part is basically doing some standard interfacing and HTML/CSS/JS around that core engine, which wasn’t immediately obvious to me when scanning the screenshots.