Our product is a XMPP client, which can't work if it is not actively running in the background, so we were working with making apps run in background on Android for over 10 years, since version 2.0.
The trend is clear: with every update, we get more restrictions. Not only bg services are killed off from memory by aggressive memory managers of crappy phone vendors, Google themselves push developers to rely on proprietary FCM services. The writing is on the wall, and one day background services in Android will simply cease to work, all in the name of better battery life and customer care. So this OpenPush service will be short lived, it'll likely be disallowed to work in this form in Android 12 or 13.
The only proper long-term solution here is an anti-monopoly legislation, that would make Google allow Android users to subscribe to a third-party push notification service. Such service is, actually, a relatively simple: current FCM is just a modified XMPP connection that a phone keeps, and it will be easy to replicate and deploy.
Apple is guilty of this, too. A proper anti-monopoly legislations should target them as well, and not only make them allow third-party app stores, but third party push notifications as well.
This control of push notification services and restriction creep that occurs on both platforms makes Google's and Apple's grip on their respective OSs much tighter than Microsoft's grip on Windows ever was. This tyranny must be stopped, and I don't believe it can be done without legal action, by technical means only.
The problem is that every app thinks that they must run in the background. Every app must have millisecond push updates for their social media network that another Like was just received. Worse yet, most of them don't know how to do so in a symbiotic way. It makes complete sense that OS manufacturers have taken a stand to protect users from bad developers. They don't want the blame and they want to keep selling devices.
It is a tough spot. How do you have open app stores and not have them filled with malware? Android hasn't figured this out yet. Users are dumb and will just click on anything. They need to be protected from themselves sometimes.
Well, it's understandable that many apps want to run as a regular process. Good old UNUX was and it's GNU/Linux descendants are quite OK with it. The users have control and can choose which apps to run and which not. If some app misbehaved, it was up to a user to identify it and remove a problem.
But Google decided that it will be our nanny and will guard all users against misbehaving apps, whether the users asked about it, or not, and you can't refuse this care. That is tyranny and should be fought against.
Now, I don't think that push notifications are bad. They are rather good idea and let them be. It is the monopoly on push notifications which is bad.
> How do you have open app stores and not have them filled with malware?
Long before the 'app store' term was coined, Linux had software repositories, which worked like magic, and without issues:
sudo aptitude install libreoffice
Boom, you now have libreoffice on your PC! If repositories had massive problems with malware, I'm not aware of them. So it was done decades ago and can be done again.
> Comparing Unix/Linux running on a PC and a iOS/Android running on a smartphone is insanity.
And comparing software repos with app stores is just as bad.
Strange idea. How is comparing two types of consumer-market computers and ecosystems is "insane"?
You should update your knowledge of modern operating systems.
My mom and my wife parents are extremely happy with Ubuntu computers. They do browsing, email, Skype, print stuff and whatever else they need just fine. That are way more happy with Ubuntu than they were with Windows, and I am, too, cause in the last three years I have spent exactly zero minutes maintaining their computers. If that's not a consumer-market OS, I don't know, what is? Windows? When mom used it, she always had one problem after another with it. I'm truly sorry for anyone who has to deal with it every day.
Alternative app stores are not a problem on Android. There are lots of them. Alternative way to deliver push notifications is a problem. Currently, to solve it in a satisfactory fashion you need an alternative ROM to connect to a different push service.
Not really -- on Linux, I can (and I do) disable the daemons. Moreover, most apps are sane, and work well without background daemons (KDE/Gnome being the exception, but they offer no unique functionality not available anywhere else)
This is not the case on Android. Looks like every app wants to run in the background and drain my battery. Even seemingly foreground-only things like maps apps install background services.
Ideally, I'd prefer to have an explicit list of apps which are allowed in background.. but missing that, I guess I will have to settle on automatic guarding.
The reason you don't have a way to disable a background app on Android is because you don't have means to do so. Google could have made controls easily, but they give you automatic guarding instead, which you can't refuse.
What are Google's motives? Do they really care about their users, or do they simply want more control over user devices?
The battery usage reports on Samsung phones are pretty well done, with graphs and all.
Users will also be prompted to kill an app if it's detected running in the background a lot. That's perfect because there are many cases where only the user knows whether the app updating in the background is important to them or not.
I really like the web push notification architecture: you register with the service you want to be notified from (like a chat server or an IRC not) and give them an ID from the push notification server your device listens to.
There’s no extra control outside of the user and the services they want to talk to (I think you can even host your own on Firefox), no vendor lockin, and the power and bandwidth usage are all great. There’s too much money to be made breaking useful push notifications and charging for ads though so I’ll be shocked if it ever gets done right on popular consumer devices.
The filesystem APIs are moving in a similar direction if I'm not mistaken? Less and less control for developers. It makes me sad, but also gives me hope that there might be a legitimate opening for a 3rd mobile OS to enter the market, built around OSS and privacy. Sometimes it's hard to keep the dream alive though. So many attempts have been made.
100%, I was one of the many vocal devs on the android bug report who complained about the change of scoped storage.
They caved in for the current release, but they are going to force it in android 11.
On the bright side I think this will cause android engineers to become more of a specialty as the platform migrates more and more from just another java implementation to a whole new world of apis!
> one day background services in Android will simply cease to work, all in the name of better battery life and customer care.
In that instance, having an app which centralizes the push service needs of other apps would be highly useful, as it could then be installed as a system app with the same kind of access as the FCM service. For many phones, custom ROMs are available. Average people probably won't have the skills to install custom ROMs. But the goal of OpenPush is not to replace FCM from all phones everywhere. It's to meet the needs of the subcommunity of people who want to use degooglified Android, maybe also later on integrate with non-Android mobile OSs. In that subcommunity, people likely already use custom ROMs, and among those who don't, the readiness is higher to install one.
You see, unless this app is built in a custom ROM, it will cease to work, too! Because it needs to be running in the background and maintain connection to push server.
I think it's alright to have the _option_ to keep a background service running, as long as the user is kept informed of the power impact (preferably with hard numbers). There are some use cases where it's the most reasonable option.
Running background services in Android is tricky. Most device manufacturers put their own battery management software on devices, which makes it almost impossible to run long-running services in the background. And since this framework would either perpetually run in the background or run frequently to check for new messages, your app is guaranteed to be throttled(force-killed) by most devices. And being force killed means that your app cannot receive push notifications even from FCM.
I'm curious to know if anyone is using this framework in the wild.
The solution to the background limitations for now is running as a foreground service. This shows a persistent notification (which can be hidden by the user) but can then run indefinitely. The other option is to integrate this at the system level (via a custom rom, root, magisk, etc.).
> I'm curious to know if anyone is using this framework in the wild.
The client side implementation of this is still WiP, so it's not used yet in the wild. I hope to be at a point where I can work with some apps on integrating it there in the next few months.
Maintainer of Red Moon, a screen filter app, here. I've struggled with similar problems as others describe for a long time.
As far as I know, the only truly 100% reliable way to never be killed is to run as an accessibility service. I haven't implemented this in Red Moon but will likely add it as an option. I believe it has a large impact on bettery life.
You can also set alarms using AlarmManager to restart the service if it has been killed, and optionally wake up the device. This is what DNS66 does.
This might have changed since, but in 2017 LastPass had to work with Google to update how they run their app. Google introduced "a new policy restricting the use of Android Accessibility Services" [1]
I'd look into that before starting that implementation.
If a OS is actively sabotaging you, then yes, the only real solution is to replace it with something else. That's the simple and obvious solution.
The simplest way to do this is to not buy a phone with shitty OS in the first place. As in "Don't buy a phone from Xiaomi".
And when users complain about it you tell them that the problem is their phone manufacturer. If you can, of course, give them a work around so the app will work. But eventually they'll plug all the holes that allow for a functional phone. So it may not always be a option.
> tell them that the problem is their phone manufacturer
Despite this being true many will see is as an excuse, or worse just a plain lie, and leave you a 1* (or 0* if possible) review ranting about your attitude and bad coding. How dare you criticize their choice of device?!
I wish this was more obvious to people. Had to use my retire parent's phone to long into Netflix for them and it was overflowing with of persistent notifications.
Under Nokia it says that its Android One phones are aggressively killing apps as well. Isn't Android One a phone with vanilla Android install without vendor crap by definition?
I'm happy that I'm using LineageOS. My phone is turning 7 years old, and it's running Android 9, and I've no reason to switch so far.
They are (I own a Nokia), but you can go into Battery Optimization in the settings and tell it to not optimize an application. That, I find, tends to leave it alone much better.
It is the only way I can get my K-9 mail app to stay alive to sync my email.
My previous phone was Oneplus with such an extreme battery manager (rated by some resource in top 3 of most aggressive managers). There were messaging apps which worked just fine in background and send push notifications even after prolonged periods in background (e.g. Telegram and others) and some apps which were killed in minutes after being minimized (e.g. official Skype app). Point is - I suspect that there is a correct way to make inoffensive apps which will work in background, but certain minority of developers simply don't give a damn about battery usage and proper coding guidelines.
Oneplus incidentally maintains a whitelist of apps which are allowed to run in the background and only apps in that list can run for extended periods. While it is possible for users to add apps to that list, it comes with a set of apps already whitelisted ( like whatsapp). I found it out the hard way while working on Android.
> I believe this aggressive behaviour tends to come from user-feedback, given users care a lot about battery endurance and heat dissipation.
Absolutely. I don't want my phone to run hot or low on battery when I'm not using it. If there was a way to shut down all background tasks when the screen is off, I'd gladly do that (second device, no sim, WiFi only use case).
And by poorly written apps which actually DO drain the battery while doing nothing useful. So I don't feel like putting all the blame on the OEM manufacturers.
The problem is they're using bad heuristics. Just because a service runs 24 hours a day doesn't mean it's using a significant amount of battery. The harder problem is, just because a service uses a significant amount of battery doesn't mean it's doing nothing useful.
If I want my email app to download my emails as they come in because I'm on a slow network connection, I want it to do that even if I get a lot of emails. Especially then. Even if that means I have to charge my phone every day instead of every week.
What they should be doing is using a blacklist instead of a whitelist. Not because they're actually going to be able to blacklist every misbehaving app (though they could certainly catch the most popular offenders), but because that's how you get developers to change their behavior.
If everybody is restricted by default then there is no incentive for bad developers to do better. They get restricted, they weren't going to get whitelisted either way, so they go on making a wasteful app. By contrast, if making a wasteful app could get you blacklisted and restricted when you wouldn't have been otherwise, well now you've got a deterrent. Meanwhile they wouldn't be defaulting to breaking well-behaved apps that are designed to efficiently run all the time.
The reason why some apps work is they've been whitelisted by the manufacturer, not a choice by the developers (except maybe a choice not to pay to be whitelisted?)
If you're not whitelisted, there's very little you can do except for ask users to take various manual steps to change operating systems.
This is a great and important project for F-Droid.
The problem is harder than it seems on the surface as idle connections can be terminated by the network operator. There's a paper about this: https://ieeexplore.ieee.org/document/8214213
I guess ultimately one should build a database about phone networks and store measurements of their maximum heartbeat intervals in there so that new connections can use those heartbeat intervals to save battery.
Also ideally the app should send its data via WiFi if available. So it has to detect when a WiFi connection is established and open a connection wia WiFi. Ideally the mobile connection is kept alive though for the case the WiFi proves unreliable. E.g. when I leave my home on foot I get further and further away from the WiFi and there's a short time span where my phone keeps trying to use the WiFi connection even though nothing gets through any more.
Detecting all these edge cases can be tricky. Fortunately the design makes the protocol between the push server and the on-phone app open to iteration and innovation. It's a good design!
If you can build a database, great, but you can probably get by with a device based adaptive ping frequency, reset when connectivity changes.
Start at a 1 second ping, and after each successful ping, double the interval until you get to over a minute. Then add a minute between each ping until you get to 10 minutes, then add ten seconds between each ping. If a connection drops, use the last successful interval for tenish pings, then probe with an additional one second. If the connection drops again while probing, reconnect and wait longer to reprobe. If the first ping failed, then assume the network changed and reprobe from the top.
And yes, you do need to start at one second; I've seen networks with a ten second NAT timeout, which was amazingly disruptive, and thankfully temporary.
Excellent point. So how does Google solve the problem? Do they just coordinate the heartbeat intervals with the carriers, or do they bypass the Internet altogether and use some cell-network-specific push mechanism? If the latter, is this mechanism available to everyone or just partners of the carriers?
> Do they just coordinate the heartbeat intervals with the carriers, or do they bypass the Internet altogether and use some cell-network-specific push mechanism?
IIRC they use the xmpp protocol, so standard TCP over IP. In threads about this topic you often hear claims that Google has special deals with carriers. Whether this is true or not, no idea. I haven't seen any confirmation of this. It would be interesting to conduct measurements whether FCM sends less frequent heartbeat messages than the timeout is for "normal" long lived tcp connections.
In any case, deals or not, I guess Google has built a database as I've mentioned. But I'm only guessing here.
AFAIK these disconnects still happen (my knowledge might be outdated). It would be cool if the phone figures out by the weakening signal + location that it should turn on the data over cell connection.
I am really excited about this project! Having a source of push notifications without routing them through the google servers will really advance the state of open source and/or google free android. It is great to see it being backed by f-droid and apparently funded by the German government (through prototype fund and Federal Ministry of Education)
Considering the BND's (German equivalent of NSA) love of intercepting traffic and gathering metadata about citizens, I wouldn't put "funded by the German government" in the Pro column.
It may be a case of German government's right hand not knowing what German government's left hand is doing. An open source distributed framework can't be coerced into routing its messages through a central server for surveillance.
While Signal client app source is open, the server side isn't and the global system (and Moxie in particular) is notoriously hostile to federation or any sort of free interoperability. For a communication network, this reduces the benefits of openness to almost nothing, and, at least from my PoV, do not bring any kind of freedom or security to the users.
While I can imagine a free ecosystem being somehow centralized, I firmly believe that Signal is on the contrary an illustration of centralization imposed against user freedom which suggest that these properties may not be so orthogonal.
No, you need to use a library that allows the user to configure the push server, which should be configured device-wide and communicated to the app server.
One, end-to-end encrypted push service can make any intercepted data useless, as you cannot even know if the traffic corresponds to a message or is just a ping, let alone which app or contact the message is from.
I actually doubt this is the case. Pings probably happen on some relatively-predictable interval, while a back-and-forth conversation between two parties is going to result in push notifications with much shorter intervals than any reasonable ping implementation would use. The metadata might be limited in value, but it still has some value, and I think it could easily be distinguished from pings.
Sure a determined attacked with control over hardware in the network path can analyze the timing and packet sizes.
However a smart application/protocol writer could make make a ping and a message delivery take the same number of bytes. Which of course means some wasted payload on pings.
Additionally if trying to track peers you could every N seconds send a ping OR a message. After all if you succeed in sending a message you know the peer is reachable.
So it's possible, but no idea how careful OpenPush is though.
Any attacker in a position to inspect traffic in the path between a push service and a mobile device is probably determined. I agree about the countermeasures, though.
I've pondered DHT nodes for a basis privacy protecting network of peers. There's a fair bit of traffic for tracking peers. Each peer could use their peerID as a public key. A bit of padding on the normal traffic (mostly get, put, ping, and findpeer) could hide messaging.
Likely not a good fit for TCP streams (like tor), but could easily hide enough bandwidth for instant messaging (like signal) or email, er email like service. Could hide store/forwarding, include onion routing (bouncing through a few nodes), and be quite resistant to traffic analysis.
If each node send X packets of Y bytes to Z nodes per minute then it would be quite hard indeed to tell who you were talking to.
Maybe give peers a few choices in high bandwidth, medium bandwidth, and low bandwidth configurations. All would just vary in the number of nodes they track.
I see no reason make this negative link. The BMBF is a research agency which gives out grants for research through public calls (somewhat similar to the NSF in the US).
There are no strings attached, besides providing reports on scientific contributions achieved throughout the project.
It's not like the BND will come knocking on your door saying make algorithm X weak or you will not receive funding (or worse).
I have a tangentially related tip or suggestion of which many appear to be unaware: most (all?) cellphone service providers (at least in the US) let you send and receive emails over text message. For example, with Project Fi you'd use $NUMBER@msg.fi.google.com [0]. You can use this as a super simple alert or notification system for scripting and small self-hosted apps without having to setup any additional services or tools. However, it's important to keep in mind that both email and text messaging are unencrypted by default, so you should be careful with what you send.
Let's consider a situation in which this might be useful: You have a 15 to 30 minute task running, such as downloading a large file (e.g. Linux ISOs) from a remote server or compiling a small C++ project. Instead of sitting around waiting you could go outside for a short walk in order to get a bit of exercise and sunlight, which helps you get to the recommended minimum of 10k daily steps. Once the task is finished you receive a text message which prompts you to head back inside.
If you just want to send yourself notifications then you can actually get pretty far only using email and text messaging. However, I'm aware that OpenPush solves a few different problems, and I love that we're finally getting an open-source self-hosted alternative to GCM.
This is an interesting idea. Sadly, applications that use this are probably not eligible to be distributed through Google Play. One of the main reasons, as I understand, is that Google wants to discourage lots of apps from each having their own network connections open in the background.
There's really no good reason that a network connection open in the background needs to use extra battery, if configured carefully. If configured incorrectly, it absolutely would, but a background connection doesn't need to regularly wake the phone up, just as Google's own push messaging service doesn't.
More connections probably does mean more wakeups, in general, though. FCM will coalesce notifications that happen close together in time, for example. Even if OpenPush does that (I haven't read all the details yet), it still means two coalesced batches of notifications rather than one if you have some apps using FCM and some using OpenPush. The situation obviously gets even worse if every app implements push itself.
I still think it's worth it, to be clear, and for people who want a fully de-Googled device, hopefully eventually all their apps will use OpenPush and the battery life situation is the same as a device where all the apps use FCM. (Assuming OpenPush implements similar coalescing.)
You're right, there isn't. But what years of unrestricted background execution have shown us is that developers these days will choose the lazy solution over the right solution about 9 times out of 10.
I'd be the richest guy if I'd get a penny every time I've seen an Android app do background polling every 5 minutes (!) instead of using push notifications or even long polling.
The usual response to recommendation to improve the design was "Why? It works and it's a lot of work to run a push server!".
So as much as I dislike restrictions on my operating systems, Google forcing developers to push notifications has massively improved battery life across the ecosystem.
This will probably be used for apps that distribute through, e.g., F-Droid. Apps that also distribute on Google Play could use FCM for the version distributed through Google Play, and OpenPush for the version distributed elsewhere.
There is still a centralized push notification app on the phone. So it's not every app having its own network connection to the push service. The decentralization refers to the fact that the notification app likely will get the ability to configure different push services. That configuration is then sent to the apps that use it to upload it to their upstream backends. Those upstream backends now have to send push notifications according to that configuration (token + hostname).
OpenPush could be very valuable for open source Android apps once it is fully developed. Currently, open source Android apps that do not depend on Google Play Services are using WebSocket or Server-Sent Events (SSE) for push notifications. For example, Signal falls back to WebSocket when Firebase Cloud Messaging (FCM) is not available.
Tutanota published a blog post explaining how they deliver push notifications to their Android app with SSE instead of FCM:
OpenPush would reduce the amount of work needed to implement push notifications compared to the made-from-scratch implementations developed by Signal and Tutanota.
I wish something like this existed that could additionally proxy FCM notifications. Allowing push notifications without any Google involvement is the ideal long term goal, but one transitional path I dream about would be forcing Google to proxy all their notifications through a host of your choice (ideally that you control).
I think I have about 30 false start projects with names like PushMux and rough outlines on potential ways to accomplish it.
Tangentially, I also dream of having true one-way push notifications. Send a minimal notification to the target device via FLEX/POCSAG or even one-way Iridium, and allow the device to connect to the internet as needed to fetch the rest of the notification.
Wow, I didn't know that the Apps on Android do implement their own push services in the background.
For all the edge cases out there, OpenPush is great news.
However, this is a huge no-no for me as it means that I would need to actively manage the device to make sure that things do work as intended.
Not happy with Google "owning" the push notifications? Then maybe the solution should be a non-profit independent corporation running it.
Having a service-per-app app cannot be as efficient as having a single service serving them all. Both in terms of device resources and user time & experience.
Surely having an alternative is a good thing but the issue seems to be institutional(You don't want to have Google in charge), not technical(Google's service works great), therefore the solution out to be institutional too. Or something that acts like an institution, something like bl_______n.
Our product is a XMPP client, which can't work if it is not actively running in the background, so we were working with making apps run in background on Android for over 10 years, since version 2.0.
The trend is clear: with every update, we get more restrictions. Not only bg services are killed off from memory by aggressive memory managers of crappy phone vendors, Google themselves push developers to rely on proprietary FCM services. The writing is on the wall, and one day background services in Android will simply cease to work, all in the name of better battery life and customer care. So this OpenPush service will be short lived, it'll likely be disallowed to work in this form in Android 12 or 13.
The only proper long-term solution here is an anti-monopoly legislation, that would make Google allow Android users to subscribe to a third-party push notification service. Such service is, actually, a relatively simple: current FCM is just a modified XMPP connection that a phone keeps, and it will be easy to replicate and deploy.
Apple is guilty of this, too. A proper anti-monopoly legislations should target them as well, and not only make them allow third-party app stores, but third party push notifications as well.
This control of push notification services and restriction creep that occurs on both platforms makes Google's and Apple's grip on their respective OSs much tighter than Microsoft's grip on Windows ever was. This tyranny must be stopped, and I don't believe it can be done without legal action, by technical means only.