I once got hired to fix an application where the original dev heard "message queue" and implemented in using email boxes because... wait for it... because email has queues full of messages. So, instead of reaching for an actual message queue, everything was queued via emails, and worker tasks started by fetching an email message. About all I can say, is, it worked... sometimes slowly.. but it did work, and had the benefit of allowing you to inject stuff into the queue by sending in a properly formatted email.
Thinking strategically about the most reliable systems in our organization today, email is probably at the top of the list. I feel like there is a semi-auto magic thing email is still missing for a lot of businesses. Many things can be managed within the latency constraints of email. All of the hardest & most complex things almost certainly can be. For instance, on-boarding a new employee could be totally driven by some back-end service that simply sends & interprets emails all day to ensure that all of the required activities are executed.
Now I am wondering if I can rewrite our B2B product interface to just be email. The more I think about this the more it starts to make a ridiculous amount of sense...
User=>Server (Email 1):
Subject: Get Workflow List
Server=>User (Email 2):
Subject: Workflow List [hash code]
Body: <all workflows by name and id>
User=>Server (Email 3):
Subject: Start Workflow #21
Server=>User (Email 4):
Subject: Workflow #12345532
Body: <First page of onboarding questions the user needs to fill out>
User=>Server (Email 4 - reply):
Subject: Re: Workflow #12345532
Body: <First page of questions with responses>
Server=>User (Email 4 - reply):
Subject: Workflow #12345532
Body: <Second page of questions, or error regarding first page responses>
...
Server=>User (Email 4 - reply):
Subject: Workflow #12345532 [COMPLETED]
Body: <Completion summary with whatever final artifacts attached>
I know our users would totally reject this proposal if it were made to be the only way to interact with our product, but if it were an additional way to interact, they might grow accustomed over time.
Think about the technical advantages of the above. If you keep it plaintext, anyone with a computer made within the last 3 decades could access your business system unconditionally. Developer productivity is likely excellent. I can't imagine you would be able to get very distracted with plain text emails.
The email server is very reliable, in the same way that an Apache web server or MySQL database is reliable.
It's the script pulling data over the network, parsing it, developers updating it, etc, where the instability comes in. And don't forget the edge cases where somebody uses weird encoding, a large email with attachments, a client that formats it in weird ways, etc.
As it happens we have an onboarding platform which certainly emits quite a few emails.
We use online forms for data capture, but funnily enough things sometimes pan out like you say.
It seems that if you send someone an email saying "You forgot to upload your drivers licence, please click _HERE_ to upload it", a good 50% of the time they will simply reply to your email, attaching the document. And why not really. We used to try again to direct them to the online system, but eventually started handling these things outside the usual flow.
Why not chat interface? This kind of communication you just described feels unnatural for email, but something people would do over chat/messenger in a heartbeat.
You laugh, but Active Directory supports SMTP as one of its transport options, because it allows occasionally-connected sites to participate in replication.
Well, email is by orders of magnitude the most hardened and most widespread message queue system out there. It’s like, it seems stupid... until it doesn’t.
This is something I can definitely see myself doing, even if I might regret it later. Now that I've got code for sending emails, it's only logical that I write code to pull them back down again and...
I have seen that too in a previous job! It worked until there was a restart in the e-mail relay server, or something in the message (I think it was Java, sending multipart messages with some custom message format).
ServiceNow had (still probably has) something called inbound actions.
This is people clicking on mailto: hyperlinks in servicenow notification emails to do stuff like approve/reject a service request or reopen an incident ticket.
You can go beyond this functionality and build any workflow to be triggered after consuming an inbound email sent to client@servicenow.com
I am sure there are a number of email based integrations built using this especially when the upstream application didn't speak API but could do email.
Potentially in original dev's defense, there was a time when OTC queue libraries, like ActiveMQ, weren't usable.
During the rise of J2EE, long before Jespin, we were directed to use JMS. Because architecture. All of the products we tried for our trivial use case would ABEND or break FIFO.
We submitted bug reports, repo steps, and patches to ActiveMQ. Closed, WONTFIX.
So I gave up. Nobody has time for all that.
Motivated by postfix, I just wrote inbound messages to disk. File name included timestamp and counter. fsync() for the win. (What more could we do?)
Processing a queue meant moving a file to a different directory. Logging, archiving, and grooming were just cron jobs and shell scripts. Tada!
Like your predecessor's solution, this allowed trivial testing, observation, and admin.
I wouldn't do this again today for anything high volume. But for less than 100s of messages per minute? Probably.
Very useful project! The great thing about email is that it's proactively pushed to all your devices, and the client keeps track of what you've already looked at. You're given powerful tools like labels and folders to organize the alerts and filter them as you see fit.
Years ago I built a little REST API for sending emails to myself. The main advantage is that you don't have to bother setting up a mail server. I still use it constantly for personal projects. It's just the quickest way for me to make my webapp tell me about new user registrations, or for some random Google Apps Script I wrote to let me know about an unexpected condition, or for me to create a bash script that tells me when my VPS is running out of disk space. I've even written scripts to monitor election results or check websites for vaccine availability.
I never did much to promote it, but anyone's welcome to use it as well (website: varmail.me). While I have had almost no downtime, I wouldn't rely on it for something that needs a particular SLA :).
Edit: I forgot to mention another tiny feature that I use all the time. The API lets you specify an idempotence key that will prevent the email from being sent if it's already been used. It's very useful to be able to only send an email the first time a user takes a certain action, or to set it to time()/86400 and only get one disk space reminder email per day. I've found a lot of uses for that.
> I never did much to promote it, but anyone's welcome to use it as well (website: varmail.me). While I have had almost no downtime, I wouldn't rely on it for something that needs a particular SLA :).
This is one dudes side projects, not an Enterprise K8n deployment - I think email is a great option. Of course when you go full "web scale" you should start using products like Sentry, but we're talking alternatives to "dump an error to stdout" here
Sentry is very easy to self-host[1] on a single node with Docker Compose. They make it a little tricky to self-host with multiple nodes, but I assume their SaaS product can scale indefinitely.
I can't think of anything easier for error-tracking than Sentry, given its ability to automatically intercept exceptions in languages like Python. Sentry also has some automatic handling for stack traces, recording the state of Redis clusters and similar bits of infra, and redacting information that appears to be sensitive (e.g. such as database passwords).
Developers can carry the practice into a company if they aren't careful.
At my last job, some of the senior/architect developers had baked in code that would email them with various application statuses. They pointed the logs at their main email addresses for maximum visibility. But what ended up happening is that they let their inboxes get so flooded with email that they simply never bothered to check their email again, including any work-related messages. So to solve that problem, a couple of them just set up keyword filters to auto-forward inbox messages to yet another service that they would check on.
Even now, there's some legacy system that emails all of the developers with some error messages. I think only two out of our 20+ development team even knows what they're for.
Further, if you flood your email server, you can miss logs. And if you hand your project off to someone else, you'd have to figure out if you also want to hand over your email account, or if you want to point the logs to their email account.
Thanks for your comment and your experience. I agree that at a large scale it would be silly to receive individual emails for error messages. It would make more sense to have a dashboard with an aggregated view and statistics and everything. Piecing together a story or determining long-term performance by email would be no good.
I would also dread the idea of multiple people logging into a single email account and triaging things without knowing who read what, or everyone getting their own copy of everything and not knowing what needs doing.
But to know that my monthly backups are working or having trouble, this is working well for me so far!
> Thanks for your comment and your experience. I agree that at a large scale it would be silly to receive individual emails for error messages.
At small to medium scale, having a mailing list for the dev team which gets emailed when issues come up can be quite handy. It can't be your whole process - someone still needs to take responsibility for actually fixing problems. And you might need to aggressively rate limit it when errors happen. But for the occasional email it can work quite well. Its much easier than building a dashboard.
Eg "[ops] Monthly backup process FAILED", "[ops] Warning: prod4 at 95% RAM usage"
You could just log to a file and then use logrotate to help manage the old logs, which should already be available on your system. Hopefully your application doesn't get so big that you need to rethink your logging solution.
I had to turn Outlook rules off due to an issue temporarily. I was getting over 1500 notification emails a day that I usually just auto-delete. They just email entire departments and have us setup rules for things we don't care about, rather than manage proper lists so we can subscribe to things we do care about.
I was thinking the same thing! Any logging tool will help take the errors and group them or some kind of housekeeping to help vs 200 emails slamming an inbox.
In my "my_operatornotify" file, which isn't published, I did test a batching mechanism that used a background thread to wait 60 seconds before sending the email, in case more with the same subject line came along. It worked but I decided to stop using it. Most of my programs send their complete logs after they finish, only the long-running daemon programs send individual warning emails right away. I got a laugh out of the YCDL explosion and fixed the issue, for now anyway.
This is _such_ a great idea, especially for personal projects that don't need something as full-featured as Sentry.
One thing to note is that it's easy to exceed your email provider's quotas with a system like this. For example, Gmail allows a maximum of 3600 emails per hour [1] before incoming mail bounces.
I've done this multiple times, it is very useful but more than once an unexpected condition triggered an infinite loop of emails. Hundreds per second. If you rely on third party email service that can get you banned.
Nice work. In a similar vein I've considered using twtxt for this kind of thing, having each server/service generate a stream of messages that you can selectively subscribe to. Obviously there a other (and better) ways to do this but twtxt seems like a really easy option that could be just be used by default so I've always got something and if I decide I need better logging/monitoring for a particular thing then I can put that in alongside twtxt
[https://pypi.org/project/twtxt/] not affiliated just thought it was a cool idea with potential for a lot more than just 'microblogging'
I deal with a number of devices and applications that don't support SNMP traps for notification but do support email. The couple times I've been "mail-bombed" by a device sending hundreds, thousands, or more notifications in a short period of time have been frustrating.
Has anybody heard of a rate-limiting SMTP proxy or MTA? I've thought about writing such a thing, but I'd gladly use something already-developed. I'm thinking of a simple token-bucket that uses to/from (or perhaps to/from/subject) to limit duplicate notifications and/or collapse duplicates in a manner similar to a syslog daemon logging "Last message repeated xxx times".
You don't need a proxy; Postfix can rate-limit natively. Plugins, "milters" and stuff inherited from Sendmail; and socket-level filters like Spamassassin, provide pretty-much comprehensive control.
I agree that Postfix config is confusing. It beats the bejabers out of Sendmail config, but that's whataboutism.
Dovecot config isn't that hard, unless you're trying to do hard stuff. Even then, your site config is an overlay on the default config, which should be what you get with your distro. The site-specific stuff is usually ony a couple of dozen lines.
I expect you could also rig Postfix as a rate-limiting front-end for some inferior MTA :-)
[Edit: oh - and Postfix/Dovecot doesn't need a machine to itself, if you're dealing with less than 20-or-so accounts, and you haven't set up a bot to spam the MTA.
Actually, I think there are exactly zero circumstances in which a mailserver shouldn't be kicked-off the machine, in favour of a higher-priority job - email is a best-efforts, store-and-forward system]
It looks like PostFWD would probably do what I want. I haven't had occasion to configure Postfix for much more than a small office plain vanilla MTA w/ a smart host for outbound in thr last few years. I really need to spend some quality time doing oddball things with it.
Most MTAs have rate-limiting out of the box. Postfix, Courier, and OpenSMTPD have rate-limiting in-built, and though I'm less familiar with Qmail, there's probably patches to make Qmail work with rate limiting.
Since I do Java code, my guess is to use Spring AOP and create an aspect for this that has a point-cut on an annotation called something like @EmailOnError.
Using loggers in python can be quite handy for a number of reasons. But similar to this you can log ship the data to DD, NR, ES, Loki, etc. And all you need to do is write a class to handle the deets.
Synchronous vs asynchronous is a choice for the author of the my_operatornotify file. This module just hands the subject + body to your code.
I did some experimenting with async sending and even batching in mine, but decided to remove that and stick with synchronous. The SMTP connection is really fast and most programs send the email at the end of runtime, so it doesn't matter anyway. I prefer the simplicity. I certainly did not want to get into managing an outbound spool, waiting for the server to come back, running a separate daemon for this...
In my_operatornotify I use tenacity to do a couple of retries, then write the message and the fatal exception to a txt file on my desktop or ~/ if there is an error sending the email. I'll notice it eventually. The stuff I'm doing is not important enough to demand more retrying.
