What I would really like is a little bit like this but not quite the same: full text search over everything I have ever seen on the computer. It would read and index the emails, web pages, word docs, etc as I open them, then later when I think "I know I saw a doc about cache oblivious algorithms", I can search for it without being distracted by 100K documents I haven't seen. Or I can find that email I read, without finding the same phrase in a bunch of junk mail I never opened.
This does not help with the email part because email programs do not register opened messages in recently used. Work-around: install a DBus or AT-SPI hook and write your own database of recently opened messages.
I've seen this been called a 'personal search engine' before. One person who is well known for their personal search setup is thesephist[0]. My friend is also working on an extension that uses NLP to semantically index your browsing history so that any text on the internet can be turned into a hyperlink to something else you've read[1].
I've read comments from people (don't remember the forum, perhaps HN) where people have said that they did this. No idea if there's a public project for this that you can use, but people have definitely done it. I agree that it'd be useful to have, though you probably need a good way to filter out irrelevant stuff.
I considered doing this - take screenshots of your screen constantly, OCR them and index them. It's fairly simple. However, there are some problems
- OCR constantly running in the background is power consuming
- What granularity do you take your screenshots? Imagine each screenshot is 500 Kb and you take one each second. This'd result in 40 gigs of data per day. How are we gonna store it? How many days data do you want to keep?
Since parent is taking power consumption and disk storage into consideration, it's fair to assume they are considering a local approach, meaning privacy is as good/bad as any other local data you have on disk today.
There was once a thread here on hacker news about missing features of operating systems, as such a thing could only be achieved on OS level... can't find it anymore, unfortunately. It was mentioned that passwords etc. could be a nightmare.
A feature like that would be a big dream of mine: Some kind of an individualized semantic archiving processing and a vector search engine to search through it.
Open-text question answering. Just make your own; index all paragraphs of all documents using TF-IDF as you access them, then when trying to search for something, use this index to get a set of candidate paragraphs and run them through BERT-QA trained on SQuAD v2. You can extend this to the content of images - first run image captioning using CNN and transformers, then index the resulting paragraphs the same way (in both cases, include a link to the original in the metadata). You might need to write some browser plugin/system driver to do it automatically as you access documents/images.
There used to be a project that kind if did this: https://beagle-project.org/. It's long since defunct and I'm always surprised that nothing emerged to fill the gap?
EDIT: I did a bit of Wikipedia rabbit holing only to discover that tracker [1] is currently running on my computer and indexing my files
This does introduce sorting contention though, what to sort for first? relevance or date accessed? Ideally you would want to introduce date accessed as an aspect of relevance itself.
Submitters need to get back to more descriptive titles on HN. If this post is the first and only exposure to this project for a user, "Project Naptha" alone doesn't give me confidence to roll the dice and click on an unknown link unless there are existing comments I can investigate. Thus, chicken and egg, scroll by.
Browser extension OCR:
From the homepage
"Project Naptha automatically applies state-of-the-art computer vision algorithms on every image you see while browsing the web. The result is a seamless and intuitive experience, where you can highlight as well as copy and paste and even edit and translate the text formerly trapped within an image. "
I never heard of it and I clicked because the name intrigued me. You might benefit from a link preview extension to reduce the friction to explore links before comments.
True, that said, it made it to the front page... and while not everything on the front page is super interesting, it does give it a bit weight on the dice roll fun.
Interesting but I thought my mac already does this? Maybe just M1 and Monterey.
I was trying to debug some image the other day, upon inspection I got confused by the shadow dom doing weird stuff, but only in safari; to then realise that it was macOS converting the text in the image to text in shadow dom! ... Good/bad experience report I suppose.
Macs and iPhones have been doing selectable text for a while now. Especially if you use Safari instead of Chrome. All images have selectable text since Monterey and iOS 15. It existed to a lesser extent in previous years.
Some of the features Naptha aims to do are not part of it though, like translating, removing the text from the image, and some other right click behavior. On the other hand, Apple is using this without any connection, which Napha says it is availble but with degraded quality and slower speed. Apple probably uses their optimized ML silicon, so I would imagine is is more battery efficient for using on the go.
I think the extra step of going into the app switcher is not really an issue. It works for every app, and I actually like that I know whether I'm using the OCR or traditional copy/paste because OCR isn't perfect. When I copy from the app switcher I know I need to double check the results after pasting. If it just did OCR automatically all the time I'd probably be surprised after copying some text and realizing later that there was an OCR mistake in the result because I didn't realize that OCR had been used.
Yeah, I wonder if there's a more modern version of this somewhere. On-device OCR is probably good enough now that a fully-offline-by-default version of this might make sense, and that's something I might actually use.
This makes me feel old. I saw it for the first time in 2014 on HN, just when it was announced. I felt kind of envious of Kevin Kwok, author of this project. I had just graduated and he was still an undergrad. He had already shipped so many complex projects, including a full-fledged Flash Animator[1] for the web.
It's pretty surprising to see that his site hasn't been updated since 2015 and not many projects have been shipped since then.
Ahh, I did an Algolia search just now and found the 2019 post[0] but thought that felt a bit recent for what I remember. I can't find that 2014 post, however - do you have a link?
This extension is a privacy nightmare. "By default, when you begin selecting text, it sends a secure HTTPS request containing the URL of the specific image... The server responds with a list of existing translations and OCR languages that have been done."
That is some pretty sensitive data to keep around. There seems to be some rudimentary thinking around privacy: "no user tokens, no website information, no cookies or analytics" Yet keeping an index of all the image requests from any IP would not pass muster by any GDPR lawyer I have met.
A related trick for text that you can't usually select: Hold Alt. For instance try using the mouse to select text in http://www.google.com with and without Alt.
I had this extension on my Chrome browser a few years ago, and was dumbfounded when I thought that Chrome had added the ability to not only OCR PNG files - but also replace the text in them!
Completely forgot that I installed this extension years earlier.
I love this idea, Apple recently added it for saved photos on iOS and I think in preview too? It doesn't seem to be working for me though. I highlighted some text, hit ctrl + c and got this in my clipboard:
<[ TEXT RECOGNITION IN PROGRESS / MORE INFO: http://projectnaptha.com/process/ (IDX:a:0-a:1-a:2-b:7&a:0&a:0&168&817:XDI) / ELAPSED 26.11SEC / DATE Mon, 11 Jul 2022 08:08:50 GMT / TEXT RECOGNITION IN PROGRESS ]>
The right-click translate doesn't seem to work either, it just selects a whole paragraph.
It works on all images in Safari on macOS as well. It's actually been helpful but was very odd when all of a sudden the functionality appeared out of the blue.
Even though I watch WWDC every year and the iPhone release event, I was surprised with how it works, and how fast it is. Basically hovering my mouse over any image (ouside of Google Chrome/Brave/Firefox) changes the cursor to selectable after half a second or so.
Took me a while to get used to. But it has ben actually useful a number of times. My most common use-case (as a front-end developer) is when people send me screenshots of bugs, and I can select, copy, paste texts from the screenshots to find the source code files =)
Nice! This works slightly better than apple's version, which makes it very hard to select the main image, once it detects text in the image. Naphtha seems to handle this correctly!
This is a cool project, but this is a bit embarrassing:
> I started building a text recognizer algorithm specifically designed for Impact font, and it was actually working pretty well, but I kind of misplaced the code somewhere. So, until I find it or replace it, you'll have to use Tesseract configured with the "Internet Meme" language.
With a chrome extension, are there any trust, security, privacy issues enabling an extension with an unfamiliar organization as well as leak what sensitive images are being transferred.
I wonder also why such "convert" options are not made available or are they in the works?
I find things like TextSniper to be more useful than this functionality, which has been in iOS and MacOS for a little bit. I use it more on hypothetically selectable elements that for whatever DOM-related reason aren't selectable than I use it on text in images.
This project is almost a decade old! I wonder why it resurfaced? Coincidentally or perhaps not the book in the post "How to do nothing with nobody all by yourself" also is a top HN post right now. Maybe related?
what I would like to find is a free a cheap way to OCR my handwritten notes. I know that handwritten is much more difficult but it would be possible to use supervised learning mon my specific handwriting.
Really cool! I went to try the translate feature but each time I left-click to open the nav menu (macOs) it deselects the text? I know it says right-click but for me it's left-click to open the menu.
last time i saw this the ocr component of it was tesseract (originally hp's c++ ocr engine, later acquired and open sourced by google, even later rewritten to use neural networks) compiled into webassembly.
Safari has its own, built-in implementation of that called Live Text since last year. Just try highlighting any text in the image and voila - it works.
Very interesting that Firefox would have this level of integration with the OS. Firefox of old was criticized specifically for being completelly not native to macOS. Times change. I might give Firefox a shot in the recent future.
Does anything remotely similar exist?