Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: What are your favorite methodologies for organizing digital life
75 points by sn0n on June 30, 2023 | hide | past | favorite | 49 comments
Ok, so this is a bit of an odd one, but I'm about to finally attempt to organize 20 years of digital clutter...

Before taking off on this endeavor, I want to make sure I have a good system in place for where things go.. I have thousands and thousands of various folders and files, things like projects, music, photos, zip files and other downloads, personal notes, backups, etc..

How do you organize your digital library?




I love culling things and losing things which have aggregated over the years. I have tried NAS'es, Disks and realize never ever come back to them.

Media - An auto-backup of photos, videos & music is what I like. Use google drive for cloud backup. Have a premium account to accommodate for all the space needed

Notes & Documents & Writings - Public ones - Already part of the internet space. The rest are really things which I can afford to lose. They are like a closet which I don't really mind cleaning out once in a bit and losing it.

Work - Tax filings & income documents are in a digital private locker. The rest of it like work related repositories, code and other stuff. Either open sourced or in private git repos.

Movies & Books - I used to collect things like movies, books and then realized that could just rent and buy them over so no more into hoarding the stuff here.

Best advice, I can give after trying multiple organizational systems is constant pruning and sharing out what you can in public domain.


- Have a password manager that someone else also knows the password to

- Have one obvious spot to put things. For example for photos: have one directory for all of them, and make subdirectories there by either the device that took it (I remember better whether I took a photo with device A or B than whether it was in 2017 or 2016) or whatever else you prefer. Use broad, obvious categories ("photos", "audiobooks/English", "audiobooks/Dutch", "games/downloaded", "games/self-made", etc.) for the main folder names.

- Don't let data get stuck on old devices. Keep track of in which folders you've stored data and, once those are copied over, give it e.g. a few months. If you didn't need to go back to the device, wipe the device. Don't be me. I have a phone from 2012 that I still haven't finished migrating.

- Have backups. Make making a backup easy. Just take the whole disk. (Games will store your save files in places you didn't know existed.) Mark the backups as backups, so you don't have to think "eh, am I going to find any original data on this 2008 hard drive or was it just a backup copy I can wipe?". Don't be teenage me.

- When in doubt, use year-month-NAME for files and directories, then the rest of the name matters a lot less as you're likely to remember the year and season that a given thing happened in. (System timestamps are prone to being lost upon migration, or get updated when you fix a typo while reading.)

- Folders can have an 'archive' subfolder for things you may need one day, but that you aren't likely to ever touch again. I used to keep one folder per school/internship/employer, and it got a bit much. Moving old things into an archive subfolder helps and I don't have to feel like I will lose the data (it's not like I marked it for expunging).

- Have a temporary folder whose entries are prefixed with an expiry date. This contains "just in case" files, like a copy of my Signal messages database before I start messing with the database (I'll know latest in a few days that I didn't need the backup). I also put call notes here, like when someone calls me and I want to make sure I've saved the file but don't have a project folder for it, but it might turn into a project, I could put it here with some months' expiry. If nothing came of it, it can be safely deleted and I don't have to review the contents to know whether it's irrelevant.


Interesting, I came up with almost the same rules over the last 20 years.


Took me about as long. What guidelines will we find, in another 20 years, that we are doing wrong today?!


I'm able to backup my files with a similar organization structure.

The problem is that this doesn't scale for immediate or extended family members. I guess the realistic option is for them to pay for cloud storage indefinitely, but that's not really ideal either.


What do you mean by it does not scale? Do you keep their pictures collection around or something?


I like Karl Voit's methodology. He goes more deep into Personal Information Management than anyone I've seen.

> For a couple of years, I taught PIM in a lecture at the St. Pölten University of Applied Sciences. My students are surprised, how inefficient they were doing even «common» things like handling email, managing files like photographs on your computer, or searching for information. It is so rewarding to see students realizing how much improvement is possible with some background-knowledge and a few hints here or there. Starting with 2021, that lecture started at Graz University of Technology.

> I hereby claim that I am able to save you at least twenty minutes of your computer time a day just by looking over your shoulder and suggesting more advanced tools, better methods, and efficient workflows.

https://karl-voit.at/tags/pim/


A notepad window with a TODO list and set of links out to stuff (like docs or PRs to review). Or quick ideas, etc.

I just brain dump what’s most important in the morning and triage my work. Then get to it. If something comes up instead of reacting immediately, I paste it to the TODO list and maybe a short note.

Otherwise I’ll just work on first thing that comes to mind, or first message I react to, not the most important.


Ebooks:

- Calibre to manage my offline collection: In case the Internet goes down or some site disappears

- Zotero (Papers and books to learn from)

Images:

- Apple Photos: I only take pictures with my iPhone

- Screenshot: Automatically goes to my desktop, which is synced with iCloud.

Movies:

- I have a old Mini that I use as a file server. Everything is there

Documents:

- PARA method (I barely produced documents on my personal devices, so I haven't upgraded anything)

Music:

- I have an offline FLAC library, but streaming is more convenient right now. I'm using the `Collection/Artists/Albums` folder pattern.

There's not much else, I only hoard books (and my favorite albums). Anything else can be deleted.


Calibre is amazing. It's the software that feels closest to physical tools for me. I've been using it for over a decade at this point and I've never had serious issues. There was definitely some user error and a learning curve in the early years, but it's earned a figurative place in my garage alongside the hammer and chisels.


What are your goals for this system? Quick inserts or quick retrieval? Perfection or good enough? How important is it to you, and how much work are you willing to put into it?

It's tempting to design the perfect system for an archivist, but you might realise that it's too much maintenance for a regular guy who has a life.

Personally, I put my photos on a timeline, and the rest in a reasonable folder structure. That's about it.


I have all of my photos (with the exception of smartphone photos... ugh) in a nicely constructed set of folders \photos\yyyy\yyyymmmdd\ then the folder made by the camera, etc. I've got a small python script to generate the folders.

I use Digikam[1] to do facial recognition and tagging on them. It's finally gotten to the point where it doesn't crash all the time writing metadata, and the facial recognition is acceptable. It's nowhere near as good as Picassa was, but the last release had a horrible bug that sometimes swapped face tags if there were multiple people in a photo. Those would them lead to bad training data, and it kept getting worse.

At one point in time, I JPEG2000 compressed all my photos to save disk space.... it was a huge mistake, and now I've got to try to recover the original jpegs from CDs and old backups.

[1] https://digikam.org/


Do any of these face recognition apps work through time?

I have photos of my kids (who look similar, obviously, I guess), going from age 0. The face apps all get confused, and often match younger kids, and older photos of older kids, and fail to match younger and older photos of the same kid.

Do any of the apps use date-of-photo to help guide this?


Photoview (self-hosted web app) does a pretty good job to group pictures by faces. It first self-detects similar faces and then lets you merge these groups together. New pictures then get directly assigned to these groups. For kids, I usually had pretty large self-detected groups, e.g. years <1; 1-2, 1-7; 7>. The most mismatched group was baby pictures (<1 years old kids).


My daughter and niece look almost alike, but separated by a decade. I keep a watch out for mistakes, but for the most part, Digikam seems to do a good job. I've not noticed significant trouble recognizing faces as people age.

I am fairly certain that the dates on photos are NOT used.


I use org-roam and hyperlink to files / folders on disk to make them discoverable like any other thoughts I store in org-roam. Using TRAMP you can even link to remote files over SSH or other protocols.

If you're not an Emacs fan and you din't want to be, Logseq is another tool that looks like it can be used in a similar way.


For home, I have a local nas that gets backed up to glacier with files mostly all in one folder with filenames prefixed with yyyy-mm-dd-topic-filename

I used to have lots of subfolders but found that it was easier to just prefix with the date name and have everything flat so I can search in a single directory for things like the washing machine receipt from 8 years ago or whatnot.

Then I have diary files with text descriptions that link to the docs.

For work, I use obsidian with lots of notes in markdown notes linking to files with deep nesting structures for projects. I have more time to organize and work pays for OneDrive that seems to find things better in deep folders.

I have a todo that I manually roll over from day to day to give me a chance to prune or revise.

I have a standing wish to explore PARA or some more systematic organization scheme but for now just my own ad how structure works. No one else sees it.


On every drive, I have a two top level subfolders with a one-character name. One for stuff that I back up, and one for stuff I don't (e.g. Steam Libraries usually go in the second one).

I'll just give examples of folders you might find on those drives.

- /Applications/Audio/Renoise/

- /AppsPortable/Images/Krita/

- /AppsData/SublimeText/ <-- project files go here

- /Drop/Downloads/Browser

- /Drop/HTTrack/SiteName

All "my" stuff (mostly things I made or personal documents, or things I curated for personal use) goes into a /Docs/ directory structure, which is similarly straightforward:

- /Docs/Projects/NameOfProject

- /Docs/Resources/Icons/Objects/Tech/joystick.png

- /Docs/Images/Created/ <- everything I made that's an image but not a photo

- /Docs/Images/In/Canon50D/012 <- I just increase that by one each time I dump my CF card, and then take my sweet time to sort that

- /Docs/Images/Out/Pub/2023/ <- uploaded RAW photos named [year]-[month]-[day]_[hour]_[minute]-[second] go here, in case of events or vacations I might make a subfolder for them

- /Docs/Images/Scanned

I also use synthing extensively, so there's a subset of that structure for things I want to have on my phone, too.

- SyncThings/Thang/Audio/AudioBooks/

- SyncThings/Thang/Audio/Music/

- SyncThings/Thang/Audio/Talks/

- SyncThings/Thang/keybass.dbx

etc.

Contacts I keep on a little Baïkal installation, and thanks to DAVx⁵ that means I can edit my phone contacts in Thunderbird, which is so awesome.

I don't want to make this post go on forever, but FWIW I made good exsperiences with going overboard in this way: because at this point, I know where any given thing "belongs". Sometimes I even download something, put it "where it belongs", and see it's already there. Of course I also have plenty of unsorted files here and there, but knowing I could sort them rather easily is nice.


It really depends on what data you're processing with. For someone whose data is hundreds of tetrabytes, the workflow to saving those data and manipulating or retrieving those data as request becomes very tricky, very hard task, especially when you think about the backup time. Maybe you can scale down the concept of L1, L2, L3 cache but at a higher, bigger scale. And always figure out what's the most important things to you at different stages and make those choices. You definitely compromise something in order to have large scale of data. So, again, it's about figuring out what's most important to you.


Well, disk space has consistently gotten cheaper, so for me it's c:\old_drive_2020\old_drive_2016\old_drive_2010 ... :P

IMO first you want to:

1. Gauge how likely you are to need things, and to split them apart on that basis. Don't bother organizing stuff that doesn't (yet, perhaps ever) need organizing.

2. Determine which things you can filename/text-search for if you need them, versus which things _need_ a good folder-organization because they are not easy to index or find metadata for.

I feel no shame having my MIDI collection from the 90s buried somewhere inside "Old Music" if I never really end up looking at it.


SMS: I decant my phone’s database into my IMAP mailbox. I love being able to search across all my conversations and access then on any device that can access my email.


Using what tools?


I'm using smsgate for this, you can also send sms with it if it's the dafault messaging app.

https://f-droid.org/ja/packages/com.github.axet.smsgate/

Also there's a play store version


I’m using an iPhone. I pull the SMS database from an iTunes backup, query it with command-line SQLite, and write out files into my Maildir (which is served to clients over IMAP). It’s a bit tediously manual. (Back when I had a jailbroken iPhone I could just rsync the database and do everything automatically.)

I haven’t reverse engineered the schema for MMS so I lose photos. I don’t much care but that would be something to do for completeness— adding the photos to the messages as MIME attachments.


Also interested in hearing more about this


Let my storage/backup harddrives finally fail so that I lose everything. OK, not the strategy, but has happened. It seems like an unwinnable war. Seeing how digital artifacts are reportedly showing up in photos on Google Photo, knowing that bitrot happens, etc, I'm not sure what a great answer is. So I'm resigned to the idea that I eventually lose it all anyway. Maybe I try to print out all the photos that matter to me this time.


I'd advice you to ask people who have done this for decades; not the tinkerers who play with the next new shiny tech out there (pseudo-literal the definition of the HN front page).

Who may have this knowledge you may ask?

https://en.wikipedia.org/wiki/Library_and_information_scienc...


On my NAS, I have 8 directories in root. Audio, Data, Documents, Images, Literature, Software, and Video, and one I haven't quite decided what its proper name should be. Everything worth keeping fits into one of those.

Literature, I use UDC which is a Dewey Decimal variant that fixes alot of what's wrong with DD. Audio, Video, and Images are Plex.

The "not-named-well" directory still gives me trouble years later, and the concept makes sense to me. I keep (food) recipes in it, as a subfolder, but also stl physible files. And though I've never managed to find any anywhere, if I do, I'll keep house building plans there. I even spent an hour or two the other day learning what those would look like (I was surprised that plumbing plans are never included, and that most places charge extra for the materials list). It was just after the Sears home kit story hit here on HN, felt briefly inspired.


I just delete pretty much everything every couple years, with the exception of photos, which don't need to be organized as they're already sorted by date and geotagged and can be searched with computer vision nowadays...

No point in hanging on to anything you don't use at least once a year.


I’ve needed tax and property documents going back 20 years.

I have many documents that I use extremely infrequently but will need eventually.

Marginal cost of storing 1,000,000 files over 1 is near zero. If I have to keep track of one file, I may as well keep track of many.


Why even bother, besides the fact that it’s fun to tinker with storage and backup solutions?

When’s the last time you looked at any of that stuff? If it was all deleted today, what would you honestly miss most? Save that and forget about everything else.


its a good question. i accumulate so many temporary images saved from day to day internet usage that at the end of the year there's just thousands and thousands of files. i started a ritual a few years ago where i would comb through all of my 'temp' images throughout the year and pick my favorite and turn it into a slideshow; for me that's been the best way of taking my favorite images, doing something with them, then not having to think about it any more. heres one from 2021 [0]

[0] https://telnet.asia/abc/2021.html


Start with goals. “With all this stuff I hope…” or think what you hope to be true in three years. Work back from that. The default (which is great if you like) is I’d like to spend a lot of time shuffling digital resources.


Here's one possibility: https://johnnydecimal.com/. Good luck - please share what works for you so the rest of us can learn.


> We call these buckets your areas. An area might be Project management or Documentation.

Which area will documentation on project management go to?

Attempts to organize complex data using exclusive taxonomies, in my experience, never works. One needs at least something like tags.


Favorite methodology is tagging. If I can put info in a system where it can be tagged, and retrieved by tag, I can attach any and all words I would ever need to find any item.


It is an endeavor! For books and movies, others made good suggestions for tools. For music, reminders, notes and photos I find it's worth paying for icloud. It works quite well on windows too these days so subsequently I use it to store most of my files too[1], except files which are too big or copyrighted. There's a compelling argument to use a file server instead of the cloud because it's really the only way to have everything in one place. So first of all you need to decide to use cloud or your own drive/file server.

For file organization, I tried to organize so many times but it was too big of a job to start. So I finally decided to do it by year. I have a documents folder with folders such as "2023". Inside that I simply make a folder for anything such as "taxes" or "CV". For some years everything is in one "unsorted" folder. It's low friction to throw stuff in there as you find it, and not too overwhelming to sort through one year at a time. It tends to be fairly easy to remember what year things are from later.

I have a limited number of exceptions that make sense to use different organization, for example a top level folder called "gitlab"

My biggest piece of advice, as someone who struggled to get organization done for years is to be very deliberate when moving stuff and delete things from the old location as soon as you've confirmed the move and backed it up. Use a tool like rsync that can give you piece of mind that nothing was missed. If you don't delete it, you'll end up like me and finding duplicate files later and wasting time making sure it's really a duplicate before deleting it.

Same goes for backups. Put your old ones on a drive and put that in a closet or delete them if you're sure you didn't miss anything. Going forward, you'll keep your new system backed up and you don't need the confusion of nested old backups-of-backups later.

[1]I also have some of these documents in google drive (from before icloud was good), some on a file server (from a period of a few years where I thought I wanted to self-host everything). It's on my to-do list to get everything into icloud only except for files which are too big or copyrighted, then figure out a good system to mirror icloud to my file server so that I am back to having one single drive to backup and trusting that everything is there.


when saving files i type something like fjfjwoaefawoefuawojefjiaowef.jpg and then am always amazed when it says ive already created a file with that exact file name


How do you guys deal with deduplication?

Say, I back up the home directory in my computer to a NAS. The computer dies at some point. I get a new computer and recover home from a backup (or part of data in the home directory). New data is added to the home directory. New computer is backed up to NAS. Over time, most of the data in backups are the same.

Any idea how to deduplicate, or avoid this problem?


Besides backups, I like to keep only one copy of any given file/project in existence. So in your case I would delete the backup after I restore it to the new computer.


> How do you organize your digital library?

Files in nested folders. Symbolic links, when a file or folder does not fit into a single category.


I just use a load of nested folders in my `Documents` directory. I keep a `_dirinfo.txt` in each one explaining what it contains. If I see something I haven't used in ages and don't expect to need again, I move it to an `old/` directory and forget about it. And I keep my miscellaneous notes in an obsidian vault.


This will probably not work for everyone, but works pretty well for me. I simply use the "default" Mac OS/iOS apps:

- Photos (shared via iCloud)

- Notes

- Music

- iCloud for file storage

- I also use google docs for spreadsheets etc.

This pretty much covers my digital life and mostly meet my requirement to be on auto-pilot.


(Ask HN) - may also get more responses if correctly titled.


markdown files

tools:

vim, rg, fd, fzf (ff as wrapper on Windows), sometimes ctags

Concepts learned with books:

- Make it Stick: The Science of Successful Learning

- How to Take Smart Notes

- Building a Second Brain

Can be found here and some more:

https://github.com/vbd/Fieldnotes/blob/main/booklist.md#self...


fzf to open anything in HOME.

immutable backups of HOME in s3[1].

usb drives with encryption keys and s3 creds stashed all over.

always setup new pc/os via backup restore with usb drive keys.

every new pc/os gets a new backup location.

1. https://github.com/nathants/backup


> What are your favorite methodologies for organizing digital life

My guess is that my answer will not be common or popular with the Hacker News audience!

(1) Emphasize simple text. For handling the text, use my favorite text editor KEdit with 100+ macros in the editor's macro language Kexx.

(2) Keep data in files in directories, and use the hierarchical file system to construct a taxonomic hierarchy of the contents of the files. I have no files in any cloud. I'm shopping for a smart phone but so far don't have one.

(3) Use text windows with command line scripts written in the interpretive language Rexx. In each text window, run a simple Rexx program that acts like a shell. The editor's macro language Kexx is a version of Rexx.

So, to do some piece of work, I start with a text window, use some tree walking scripts to get to the relevant directory, start KEdit with that directory the current one, and then get to work.

Each directory has a special file that has some basic documentation of the contents of the directory.

(4) Have a file FACTS open in KEdit and with some useful macros. When I see a fact I want to remember, I put an entry in that file with some appropriate keywords. Since my use of the hierarchical file system does not always have an obvious, unique directory for each file, for an important file I put its tree name in an entry in FACTS.

Over the last 8 years, have put on average 3.66 entries a day in FACTS, and on average an entry has 566 bytes. So currently FACTS has 4,414,649 bytes, and KEdit finds things in the file very quickly, even without using the key words.

(5) As a last resort I run a little Rexx program that puts all the tree names in my most important drive letter into a file. That file has 230,159,949 bytes with 6232 lines. KEdit can read and search this file quickly.

(6) I do file system backups using carefully selected options of ROBOCOPY.

(7) My most important directory is DATA05. That name is reserved so that my scripts and macros can work with the tree rooted there independent of the drive letter. So, drive letters can change, and I can essentially ignore the change and still just keep on working.

(8) Due to a disaster, I had to rush out and buy a laptop. It runs Windows 10 Home Edition. I'm working to get back to the desktop I built with an 8 core AMD processor and running Windows 7 Professional. I like that version of Windows and see no reason to change, but I do intend to plug together a server and there run Windows Server 2019.

(9) For writing important letters, math, other technical material I use Knuth's word processing software TeX. I have about 100 macros I've written in TeX, e.g., for verbatim content, cross-references, putting annotation on figures, .... I do not use LaTeX. The TeX distribution I use came with a spell checking program Aspell -- I like it a lot; it's my main means of spell checking.

(10) For Windows and Linux, I wanted to emphasize exactly 1 and selected Windows. For software development for my startup, I program almost entirely in the Microsoft Visual Basic .NET (VB.NET) with ASP.NET, ADO.NET, etc.

From my imperfect memory, the book by Kernighan and Ritchie claimed that the programming language C had an "idiosyncratic" syntax. And, then, similarly for C++ and C#.

Supposedly at one time there was a source code translator to go, either direction, between C# and VB.NET. So, the semantics were the same and the only difference was syntactic sugar.

E.g., I wanted something like Redis so wrote it quickly in VB.NET using two instances of a collection class. The Microsoft documentation for the class seemed to be essentially the same for C#, VB.NET, etc. So, I guessed that, really, C# and VB.NET were, at least in practice, essentially equally powerful.

To me VB.NET has more traditional syntax, more like the original Basic, Fortran, Algol, PL/I, Pascal and is easier to teach, learn, read, and write. So, I like VB.NET.

If I have to write some C#, then I will. Similarly for C, C++, Python, etc.

(11) To write software, e.g., in VB.NET, I just type it into my favorite text editor KEdit. I've never used an IDE (Integrated Development Environment). Once I tried but gave up in a few minutes.

The software I write has a lot of documentation, and some of that is in the same file as the code. So, to me, in the file there is the documentation and the code; given one of these two, with some work, can try to reconstruct the other; and the easier direction is to read the documentation and reconstruct the code -- that is, having just the code and then trying to reconstruct the documentation is harder. Then, writing the documentation is more like other writing in a natural language that I do with a text editor. So, I just use a text editor, not an IDE.

One final point: Sure, early in my career, I eagerly learned all the algorithms, program languages, etc. I could so that I could do the work of employers. Now that I'm doing my own startup, the technical tools I use are the ones I need for my startup. Then, so far, for my startup, I see VB.NET as just fine and see no reason to spend time learning other languages.


I am building just that. A few weeks off.


reading:

RSS feeds

Facebook

Email

saving:

documents/subject folders

pictures/subject folders or

pictures/yyyy mm dd subject folders

videos as for pictures also youtube channel organized chronologically




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: