This article is correct, although slightly outdated - Python has been shipping its own timezone library called zoneinfo to solve this specific issue[0]. On the various *nix systems it just uses your systems timezone data, but it will fall back on the tzdata pypi package it that's not available (for example on Windows you need it if I'm not mistaken.)
I have no choice, because 3.9 isn't supported on Windows 7. I've always wondered what feature of Windows was so important that they had to drop compatibility with older versions, it's intentionally a very stable interface.
There is a discussion with some insight into the technical reasons why they don't support Windows 7: https://discuss.python.org/t/windows-7-support-for-python-3-... (summary: api-ms-win-core-path-l1-1-0.dll is not available on Windows 7, so they'd have to switch from static import to feature detection + dynamic import in several places; and the Python team don't have access to Windows 7 so they can't do testing).
An unofficial Python installer for Windows 7 exists: https://github.com/adang1345/PythonWin7 (it uses an unofficial implementation of api-ms-win-core-path-l1-1-0.dll).
Sure, I wouldn't expect to get any help from the Python folks if I had some trouble. I was hoping for more of a don't care philosophy. But the installer goes out of its way to disallow installing it.
No, I haven't tried to build from source. That's a rabbit hole I'm not particularly interested in jumping down.
Tangential, but man time zones are just really hard. Some theoretically “simple” rules seem to routinely violate human intuition, and the almanac actually changes based on laws and stuff.
Keeping everything in Zulu ms or ns and rendering into timezones as late as possible is a useful guideline as far as it carries one, but sometimes it’s not enough, this is one of a zillion reasons why e.g. calendaring is just really hard.
I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
It's even harder. Is the time the user entered actually an always-local time?
As in, if I set my alarm for 08:00, I'd expect it to wake me up 08:00 local regardless if I'm in Oslo or I've traveled to say New York.
However if I make a calendar event things are less clear cut. For meetings and such it makes sense that the time is global time (hence stored in UTC), but for tasks it can be be either or, depending.
This is the part that isn’t even technically hard per se. It’s just hard because there’s literally no way to know what the computer should do because there’s no way to know what a human would even expect.
In many cases reasonable assumptions can be made, e.g. a calendar event for two people in different time zones to meet on a video call almost certainly needs to be scheduled for a proper time (or likely UTC, which is close enough since if you can ignore leap seconds). Although even then I wonder what the two attendees would expect if, between the scheduling of the event and the event itself, both of their local time zones changed their DST rules such that their UTC offset changed by +1 hours. If only one time zone changed its UTC offset we might assume that the previously agreed upon UTC time would hold, but it’s hard to say what the humans would expect if both of their time zones shifted the same way.
If there’s a clear “main office” that’s probably a reasonable assumption, and it’s probably a reasonable feature for calendars to explicitly allow scheduling an event for a specific time zone (instead of converting every datetime to UTC or similar before storing it, which is the usual recommendation among programmers).
I think it's fixed now, but DB (german transit company) used to always use your local time in its app, making it incredibly confusing to book tickets or even check on tickets you already have. I planned a trip from Japan and the train times showing up were something like 2:30 in the morning.
Another bad example is using Windows bootcamp on a mac. Windows writes local time to the RTC and MacOS writes universal time, so when switching between MacOS and Windows, it's always wrong at first.
Even worse with transit in Vienna/Austria where if you buy tickets in advance for a specific day or date range they seem to take your local time into account if those tickets are bought while in another timezone. That's for tickets where time shouldn't even matter because they are for full days (from midnight to midnight).
If you're in Pacific time and you buy the ticket for a date in the future (let's say June 1st) before 3pm your local time everything works fine. But if you buy the same ticket for June 1st after 3pm Pacific local time (which is already the next day in Vienna) the ticket that you're buying isn't for June 1st but actually for June 2nd (only visible once you receive the confirmation).
So apparently they're somehow translating between timezones for the displayed dates, although that doesn't make sense: If I buy a ticket for June 1st on Vienna public transport I don't care what my local time is. Even if I'm in California while buying that ticket I still want it to be valid on June 1st instead of June 2nd.
I do this, If you have a unix install on the same host, It really helps keep a sane time config. there are ways to do it the other way around. sysctl kern.utc_offset on openbsd. But the unix way feels better to me, so I change windows.
> But the unix way feels better to me, so I change windows.
I'm with you as well. Sometimes you don't have a choice when dual-booting, MS-DOS only wants to work in local time while on the opposite end of the spectrum MacOS only wants to work in UTC, but if I have the choice I'm going to change Windows to use UTC.
I was baffled when I first entered my schedule for a conference into Google Maps, only to find when I flew across the country that all the times were wrong. I understand that it's a reasonable choice to use the local time zone at event creation, but it sure wasn't what I expected (I vaguely thought it would be like your alarm clock example, where it would move with the local time).
I once made a flight by the slimmest of margins possible (the gate crew stopped closing the check in because they saw me sprinting through the terminal hall), because of this and only one hour difference between where I started from and where I went.
The whole DST time change is predicated on the fact that almost nobody sets alarms (or does anything time-sensitive) at 130AM local time. So yeah, if you do that, that’ll happen.
Is that actually true though? Does, say, an iPhone actually fire the alarm both times the clock flips to 1:30am? I tried Googling it but I could only find info about missing alarms entirely when springing forward. I also found some info about how cron works but even then the answer was “it depends which version of cron you’re using.”
You don't even need timezones or DST to run into issues along these lines.
Let's say you have have a repeating 7am alarm on your phone. What should happen if you want to set it to 8am for one day? I mean, without walking you through a dialog that looks like: do you mean today 8, or tomorrow 8?
> For meetings and such it makes sense that the time is global time (hence stored in UTC)
I think this is tricky too - if you have a 9AM meeting, if your time zone switches, should the time of your meeting change too? Most of the time, no, but if you have a meeting with other people in far away time zones, maybe you don't want it to change.
Doesn’t “meeting” imply multiple people, and the only time you’d need to worry about time zone issues is when those multiple people are not all in the same time zone?
That's hard enough for a calendar program to figure out, but also it can also be affected by whether other meetings involved multiple time zones, especially when something is recurring.
> - in samoa, there was never a december 30th 2011.
That was interesting, so I had to read about it. Apparently they skipped a day to be closer to Asian time zones. They did the same thing in the opposite direction in 1892 by having two 4th of Julys to be closer to American timezones. Now they kinda just walked back on that.
Interestingly I think the Python datetime approach can handle both of these with no problem. The rest of your list sounds... a lot tougher.
The transition from the Julian to Gregorian calendar also happened on vastly different dates in different countries. Most of Western Europe changed in 1582, the British Empire (and its American colonies) in 1752 as you said, and other countries switched even later like Russia in 1918.
Is that timestamps as in unix timestamps? That won't work with birthdates for people more than 53 years old for example; a colleague mentioned a weird issue for someone born in 1938, but that was because the timezone in that location at that time was something like GMT+1:23, then the Germans came around and standardized it to +1:20, then later it was normalized to +1:00.
tl;dr, time and date can only be accurate if time, date and location are known. Actually, some places have moved to different timezones over time as well, so you can't just say "this time has a timezone of GMT", you have to ask "what is the timezone of this location at this time/date"
To get to/from that local time representation, you can still store it as a timestamp (to know when it actually happened) and use a timezone database that tells you what the people living there in that moment labeled that time as, right?
You still have the fundamental issue of storing it correctly in the first place. Do most languages time libraries account for localized temporal shifts of this nature by default? It does seem true that you could query an authoritative source, but on entry:
If I say the event happened 60 years ago exactly, and it’s 1pm, what do I mean by “exactly?” Do I mean the day 60 years ago, at 1pm? Or 60 years ago, taking into account timezone shifts? Do I, the person entering this information, even know or care about this distinction? If you enter an event as 60 years ago, would your average joe expect that the time would shift by 23 minutes, or would they think there’s something wrong with the software?
What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
> Do I, the person entering this information, even know or care about [the time of the event I'm referring to]?
I'm sorry but if you don't even know yourself what you mean, the computer cannot know it either. The topic here is how to store a datetime, not how to guess meanings.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Again, display conversion. Have these natives put their date system into the timezone database and make the UI have a selector for this timezone. Under the hood it still happened at a certain actual moment in time which is what should be stored in the database.
No significant difference from any other timezone that you have to apply to a unix timestamp to get a display time, or conversely, apply to input data to get the timestamp to store.
> What about timezone differences that vary depending not only on geography, but also on the ethnicity of the people associated with the event?
Typically, recently, this problem would occur in disputed territories where the “legal time” at a certain point differs because it is claimed by different polities with different timezone rules, so its more of political allegiance than ethnicity (though those correlate).
You are correct but also don't have to look that far: in Indiana USA, different counties have different daylight-saving rules, so "August 13 1pm" is a different time depending in which county you are (or live, or work).
You mean ethnicities that are still deprived of nationhood, so the empire they are subjects of assigns different timezones to them compared to other ethnicities they rule ?? (Any examples ??)
Yeah, ISTR that there are (or have recently been) some at least theoretical (that is, where there were political claims with different timezones, but not necessarily populations in the affected area using both) ones besides that, but that may well be the only current one.
Time zones, character encoding, localization. I admire people who dig in and really understand them deeply, but as far as I’m concerned they are all exposed high voltage wires that I avoid going near.
> I’ve long since mentally tagged “timezone handling”, in any language, as a slow zone: let’s stop and really think this through.
I will second you ok that: date & Time handling across timezones was one of the 2 hardest things I've ever worked on that are deceptively simple on the surface. The other was adding localization/making an app translatable to any language after shipping the first version with hard-coded text.
Same here. Worrying about current timezones is a hard enough problem. Getting accurate historical timezones handling working is an even harder problem. Getting consistent results across Windows, Mac and Linux is an even harder problem. Needed it for proper handling of historical time series for financial data.
Non-programmers don't care about this nearly as much as we do. The prevailing attitude is "Ok sure there are edge cases, but so what? My use case is simple".
And after thinking about it a lot more, I think this attitude is actually more sensible than mine used to be. Why do we as programmers keep trying to fit a square peg into a round hole? Timezones are a political construct, not a mathematical one.
We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
> We should address edge cases by deferring to people as much as possible. Oh, you find it annoying to enter an offset? Don't worry about it, it's optional! Your dentist will expect you for your cleaning at 8:00am +/- 5 hours.
> If we keep smoothing over the rough edges for our users, the problem will only get worse. Better to make timezones as painful as possible for as many users as possible. They'll eventually vote for something less obnoxious.
This is gonna go over great with the people making purchasing decisions.
As much as I would love this too from a programmer's perspective, reading [1] convinced me that time zones are a necessary evil we just kinda have to deal with. They could be improved by a lot, though. First thing I'd do is get rid of DST everywhere.
Thanks for the reference. I have a number of issues with that essay, but fundamentally, once we have people on Mars we'll have exactly this issue. There's no easy way around it and we'll always have to negotiate hours that work for both parties (which is what I basically have to do with my family who are about 90º phase shifted from me).
Implement everything in Zulu, then watch as the frontend ticket for proper rendering stays on the bottom of the backlog, because you always have more concerning problems than users having to do mental math on times.
Time zones are themselves a footgun. There is no actual reason for morning to start at approximately the same time everywhere. 9am can be 9am everywhere. In some places 9am can be sunrise, in some it can be sunset, and in some it can be the middle of the night. No conversions, no confusion, no resetting of watches...
We don't change the months in the southern hemisphere. Australia is cold in July, Europe is cold in January, this doesn't seem to bother us.
There may be no technical reason for it, but having a common reference point for times of the day in terms of human experience and behavior is an actual reason. Saying you were walking down the street at 2am or 3am is immediately distinct from saying you were walking down the street at 10pm or 11pm in ways many people understand. Having to qualify it with the date and geographic location and/or sunrise or sunset patterns would be a much bigger PITA than your average person will ever experience because of time zone inconsistencies. It's just not that important for most people to be able to pinpoint a timestamp to one exact time throughout the world.
We've got GMT. Nothing is stopping anybody from using that to schedule their lives. There's a reason people don't.
It's a lot easier to remember "north and south are opposite" than it is to remember, "Oh right, 9 am is morning in England, early morning in New York, and the middle of the night in Los Angeles." And so on for every possible time of day.
Whereas it's (relatively) easy to guess, "Los Angeles is in California, that's three hours behind me in New York, so it would be 6 am there at 9 am here, so it's too early to call them yet." Not perfect, but not too bad for simple cases, and easy to do in your head most of the time.
Timezones are about standardizing _human_ time, not actual time.
I know some of the HN crowd are gonna wag their finger at me but the moment I have to deal with timestamps in Python, even for the most simplest of cases, I parachute in pandas. `pd.Timestamp` (and its friends like `pd.Timedelta`) in my opinion are the most powerful timestamp toolkit available anywhere.
To me it's very often easily worth the extra 50-150MB of install size that just works. I've used pandas for around 7 years now and it was inconsistent and buggy at the beginning. But these days, it's practically flawless. Also I'm aware that pd.Timestamp is built on top of many awesome libraries.
Handling timestamps well was one of the motivations for building pandas in the first place (Wes McKinney started pandas while he was working at an investment management company).
I'm not a Pandas person - is it possible to install the Pandas timestamp functionality as a smaller library, or do you need to install the whole shebang?
I had investigated if this was possible to do but I couldn't find a way to do it myself. It's possible that pd.Timestamp might be intricately tied interally to vectorization optimizations using numpy or others. Therefore separating out just the timestamp stuff from pandas would be a lot of work.
Python is in many ways the anti-JavaScript when it comes to packaging. Where JavaScript will have a installable package that contains a single trivial function, Python packages will bundle hundreds of functions, all vaguely related to an area, with no easy way of just installing what you actually need.
My experience has been exactly the opposite, with two exceptions: pandas and numpy.
Generally every other library works in terms of standard python types where possible. Very few other places will you see things like a re-implementation of a time library.
...and scipy and all the scikit libraries and whole bunch of others. 'Accidentally' pulling in hundreds of MBs of libraries you never use for a 100 line data processing script is par for the course in python land.
Well, pytz actually is technically flawed at the moment because it cannot read TZif Version 2+ files, so it breaks for dates >2038 (and it would break if you tried to point it at system data generated with zic's "slim" mode), see this: https://github.com/stub42/pytz/issues/48
But you are right that pandas' use of pytz is benign and as far as I know doesn't suffer the same usability issues.
It's a bit unfair to complain that pytz doesn't use the "standard" interface, when that standard interface didn't exist when it was created. The necessary calls for syncing the time zone with the date on datetime creation were a relatively recent addition.
If you are looking to solve a timezone problem right now, then you care about what the standard interface is right now... not "when the library was created".
The documentation is very clear on how to use it to solve your timezone problem. The trouble comes when people expect it to "just work" without understanding how it works.
Until now I never took a close look at how Python deals with timezones. Now I did, and it turns out the way Python handles arithmetic on timezone-aware datetimes is not what I expected. An example to illustrate:
from zoneinfo import ZoneInfo
from datetime import datetime, timedelta, timezone
localtz = ZoneInfo('Europe/Brussels')
utctz = timezone.utc
a_local = datetime(2023, 3, 26, 1, 59, 59, tzinfo=localtz) # right before change to DST
b_local = a_local + timedelta(seconds=2) # 2 seconds later; should be right after change to DST
c_local = a_local + timedelta(hours=2) # 2 hours later
a_utc = a_local.astimezone(utctz)
b_utc = b_local.astimezone(utctz)
c_utc = c_local.astimezone(utctz)
print('datetimes:')
print(f'(A) {a_local} (utc: {a_local.astimezone(utctz)})')
print(f'(B) {b_local} (utc: {b_local.astimezone(utctz)})')
print(f'(C) {c_local} (utc: {c_local.astimezone(utctz)})')
print('deltas:')
print(f'(B-A) local: {b_local-a_local} utc: {b_utc-a_utc}')
print(f'(C-A) local: {c_local-a_local} utc: {c_utc-a_utc}')
For (B), I would expect 2023-03-26 03:00:01+02:00: that is the point in time 2 seconds after (A). The datetime Python produced doesn't even exist (because of the start of DST, time jumped from 2:00 to 3:00).
For (C), I would expect 2023-03-26 04:59:59+02:00: that is the point in time 2 hours after (A). As you can see by the times in UTC, (C) is only 1 hour later then (A) instead of 2.
Both feel really wrong.
The article compares pointer arithmetic using Python's time zone model vs pytz's time zone model, and concludes that pytz is worse because you have to normalize the result for the time zone to be adjusted. But the pytz result is correct (according to my expectations, at least), while Python's standard approach does adjust the time zone but without properly adjusting the time (and hence is incorrect, according to my expectations). It seems the author only looked at the time zone, without checking the time itself.
I can work around it by converting to UTC and back, fortunately. I had a quick look at Pendulum and it seems it has functions to do arithmetic properly.
I've run into this at work and I handle by always add timedeltas in UTC or just design everything to work in UTC.
Unfortunately, some people dislike the former because it's 3 lines of code (convert to UTC, add delta, convert to local). I've had the same data scientist reject 3 MRs fixing this sort of issue in his code. At least it only causes a once per year error or a blip that no one notices.
Always using UTC doesn't work always depending on requirements. The worst was interacting with an API that required precisely every hour of the following day to be sent without good documentation.
Interesting. I got to thinking why a negative fold (e.g. -1) is not used to indicate local times that do not exist due to standard time to DST transition (moving clocks forward in spring). It is in fact discussed by the PEP: https://peps.python.org/pep-0495/#strict-invalid-time-checki...
The PEP provides instructions for detecting invalid times (that work as long as you use the standard library zoneinfo database.) In short - for invalid times, the UTC offset will be different for different fold versions (1/0) but with converse order than for times that exist doubly:
I was going to complain that Python for all of its "batteries included" lacks a simple facility in the standard library to "please give me a timezone for EDT", but apparently one has been added in 3.9: https://docs.python.org/3/library/zoneinfo.html#module-zonei...
The real problem that still is not fixed is that `datetime.date.now()` and others do not do the right thing and encode the system timezone. Rather they record a time without timezone. I have seen this cause so many problems because people are just not aware.
Actually, naïve times are considered aware in the system time zone. There's no good way to create a `tzinfo` object that behaves like "the system time zone" and doesn't have unfortunate consequences. See this article: https://blog.ganssle.io/articles/2022/04/naive-local-datetim...
It's not an ideal situation, but given path dependence I don't think it can be improved much from here.
Even worse is that utcnow() returns timezone unaware dates, when it could have easily returned dates with timezone set to utc. Leading to accidentally mix of local and utc times.
Can stop and step time predictably. Any code using datetime functions will get the time controlled by the test, with no additional abstractions or other boilerplate needed.
I've read the title and thought "oh this seems to be a weird name for a footgun. It would totally make sense if it was about timezones". Then I clicked the link.
> The title of this post claims that pytz is the fastest footgun in the west, by which I mean that pytz is a quite well-optimized library, and historically it has been faster than dateutil.tz
Wat. The author has no idea what a foot gun is.
Pytz is a must because using the standard library and timezone manipulations often leads to bugs. There are nuances around “naive” date times where one can easily just add a fake timezone value, not realizing that the time didn’t actually get localized to the timezone.
There are other libraries such as Arrow and Pendulum worth checking out but pytz is probably enough.
Eh? They're pointing out it's an extremely well optimised and fast library, and then pointing out a number of ways in which it's a footgun because it doesn't behave as you'd expect, e.g.
Did you read the article? The author explains exactly why they think Pytz is not enough, and why they think dateutil.tz (not the standard library) is a better alternative.
To be clear, when I wrote this article, I had not yet created the `zoneinfo` module in the standard library. All the recommendations for `dateutil.tz` apply to `zoneinfo` as well, except that `zoneinfo` is also faster than `pytz`.
Thank you so much for your work on zoneinfo. I spend my days working on a legacy Django app and can't wait to switch from pytz to zoneinfo. In my experience, timezones are always a struggle so your efforts to make them easier to deal with is hugely appreciated.
[0]: https://docs.python.org/3/library/zoneinfo.html