I'll write it up when I finally have this whipped, but we have a unit test stumper ourselves. In our Django project, the tests have always run slow. Once we got above 50 tests I finally decided to hunt for the cause. It creates a test DB from fixtures for each test case but i couldn't understand how that could possibly take several seconds per test.
Then I was coding on the train home one night and in the tube my hotspot lost connection. Then I happened to notice... with wifi off the tests ran 10x quicker. Pull out netstat.... I see http requests to akamai essentially as each test setUp() is run. It's SSL so I only see a hostname, not the full path, and they just look like this, but the hostname is dynamic:
a23-34-132-181.deploy.static.akamaitechnologies.com
I spent 30 mins reviewing our code, grepping my code and packages directories, and stepping thru in a debugger. No luck.
Is the URL class in Java special, versus just a Uri structure?. Because relying on DNS for equivalency sounds totally wrong for many cases. At least such a compare should have a special name to indicate resolution will be attempted.
The URL class in Java is special, as in especially broken. It's very old code that probably wasn't a good idea even back then, and it's never been changed for backwards compat reasons.
Use java.net.Uri, or e.g. the equivalent from Jersey or whatever your local framework/HTTP client/... brings with it.
There is a couple of other places where this fails pretty hard. If you compare to URL's on the same shared host, they will be considered equal and equality of URI's can change as the network goes up and down. I would imagine at some point (if it hasn't already) become deprecated.
Joshua Bloch used this as an example in one of the brilliant Java Puzzler video series with the net advise of don't depend on non-local state for equality.
Ned Batchelder had an interesting post recently about finding temp file creators through monkey-patching[0]. I could see a similar approach being helpful here by patching the socket module to find all call stacks.
Take a tcpdump and open it in Wireshark - you can't see the content of the requests, but the first packet will probably have an SNI header that tells you what hostname you're trying to connect to.
That's a weird one. I've got some advice about your testing methods though. Fixtures are sloooooooow. They also run before every single test method, not just once per TestCase class, unless you're using Django 1.8 (which introduced setUpTestData which is run once per class rather than once per method). You'll get some additional speed up by converting your fixtures into python.
If that was with Java, I would do something to make network terrible slow or even unresponsive and try to dump stacktrace to catch the bugger. Not sure if you can print stacktrace in Python in the middle of something.
Two test engineers were in a crunch. The floppy drive they were currently testing would work all day while they ran a variety of stress tests, but the exact same tests would run for only eight hours at night. After a few days of double-checking the hardware, the testing procedure, and the recording devices, they decided to stay the night and watch what happened. For eight hours they stared at the floppy drive and drank espresso. The long dark night slowly turned into day and the sun shone in the window. The angled sunlight triggered the write-protection mechanism, which caused a write failure. A new casing was designed and the problem was solved. Who knew?
I have a relative who loves to tell the story of a school building's clocks that would show the wrong time every Monday morning. Turned out their janitor's new vacuum cleaner was putting interference on the power line and disrupting the master clock system's signal enough to require a manual reset. Problem went away after replacing the vacuum.
Before clicking the link and knowing absolutely nothing about what project this even is, I knew from the headline that it would be a timezone or DST issue. I've had the displeasure of dealing with a lot of TZ offset issues in a time-critical app. They are a always a joy to discover, fix and test.
I was also thinking this would be related to DST. The "tricky" bit about Australia is that DST is "backwards", at least compared to those of us in the northern hemisphere. Because the seasons are reversed from those north of the equator, DST is also likewise reversed. They spring forward in October, and fall back in April. It all should make sense when you think about it. (Longer days are still in the summer, and thus DST is still in the summer.) This is of course not unique to Australia. (I think they just happen to crop up probabilistically by having a high population using technology, and are south of the equator.)
The cool bit, at least to me, is that the mnemonic "spring forward" still works. Likely, this is only cool to me because the whole thing is unusual.
The other odd bit is that some TZs sit on the "wrong" side of the international date line, and thus get interesting offsets such as +13 or +14h from UTC. I think there's some that are -13h too, so the total "length" of day (that is, how long you could remain on a calendar date) is >48h. Thus, at any given time, people in their local time are in as many as three different calendar dates.
There's also a few TZs with half-hour offsets. (e.g., they're something like 4.5 hours off UTC.) There's a 3-hour TZ jump at one point on China's border, I think, because all of China is on a single timezone, and the nation is quite wide, east-to-west.
We've not even gotten to leap seconds, or that 2100 won't be a leap year.
And governments regularly muck with this stuff. The set of timezones and their offsets and DST preferences are constantly changing.
To get anything done, I feel like the general programmer[1] is allowed to assume, at a bare minimum, a Proleptic Gregorian calendar. Anything else just makes handling early (pre-Gregorian) dates madness. (Various geographical locations adopted the calendar at different times, so you end up needing to know where you are to compute the "local" date. Even further back in history the rules get even more odd, and unknown. [2], if I read it correctly, includes a table of when scholars _think_ people inserted leap years.)
Once you go with "Proleptic Gregorian calendar", I think points such as
> Months have either 28, 29, 30, or 31 days.
…are actually always true. Of course, two bullets down is,
> There is only one calendar system in use at one time.
Which is essentially what we're using as axiom here. It's true because we've defined it as such. (Cheating, I admit, but if you're building the next great mobile app, do you care?)
I do wish that site included fold-out sections to de-mystify some of the points.
> Non leap years will never contain a leap day.
Your definition of leap year isn't "year with a leap day"?
> Unix time is the number of seconds since Jan 1st 1970.
> Unix time […] is […] defined as the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970, not counting leap seconds.[3]
Again I wish for pop-out explanations, because I'm left to wonder whether POSIX's lack of leap seconds is what the author is hinting at.
I'm bored though, so perhaps I should write a gist showing which of these I believe to hold (and why) and which do not (and why).
[1]: I think anyone breaking this rule would know that their area of interest requires them to break it, and thus know that they need attention.
What about the Islamic calendar? If your software handles anything having to do with dates and is being used internationally then it seems to me you have to take that one into account: eventually somebody will want to see their calender or input a date in their local date handling system.
Edit: I am confused about the downvotes. I point out that he can't assume there is only one calendar system in use, even for general software.
I guess you're being downvoted because your point about Islamic calendars is (mostly) wrong. Islamic calendars (but I don't even live in a country where it's used, so it might be more nuanced than what I say) are used mostly for religious purposes; you 'can't' (practically speaking) use only the Islamic calendar because many things you care about (like, your barber's appointment) would be done in 'Gregorian calendar'. So an Islamic calendar is only needed in a very limited subset of all software packages, even in Islamic regions. It's not as simple as 'in some areas people use Gregorian, on others Islamic', which your post seems to suggest.
Iran is using the Islamic calender. Just look at their railways website http://www.rai.ir/index.aspx Or the website for the airport in Tehran ( http://ikia.airport.ir/ ), which uses persian dates for the weather forcase and gregorian for flight details.
And yet, when I am contacted by Iranian researchers, they all express dates and times in the Gregorian calendar. Obviously those with international contacts are not 'average', but still there is/must be wide spread use of the gregorian calendar - as your airport example confirms.
Either way, let's still for the sake of the argument say that all of Iran only uses the Islamic calendar. That still reduces the GP's point from 'to sell software in countries that are mostly Islamic, you must support multiple calendars' to 'to sell software to Iranian customers you must support multiple calendars'. Iran, just to state the obvious, is widely embargoed across the world, and it's quite challenging to sell anything there, legally (I have first hand experience to the extent that it wasn't worth the time for 5 figure contracts - and I suspect that even 6 figures wouldn't make it worth it).
To conclude, I don't think your contra-anecdotes are enough to counter the position that assuming Gregorian is 'good enough' in the vast majority of cases (again, those few writing software for Mormon record keeping, or history journal analysis tools, or -yes- software for Iranian state systems, probably know so much about dates and times that they don't need '100 things' lists to know what to look out for).
> There is only one calendar system in use at one time.
This is useful, but technically false. Go tour graveyards around Boston. You'll find stones with two death dates on them from the mid-1700's (most of the US changed in 1752).
As a dev living in one of those half-hour offsets (ACST +9.5 Adelaide, South Australia) i think the only thing it's changed is our awareness of timezone issues, although there's discussion about moving us into AEST or back to AWST which … well … will be painful
Brazil is another fun one that has caused my team's systems to blow up: since their DST transition occurs at midnight, there is no midnight on the DST transition day.
Brazil is the most populous, which is why you've blown up there. They are actually one of a group of ten locales that transition at midnight. They have the dubious distinction of making the JS Date object revert back to the previous day if you create a Date with the transition day due to the constructor defaulting to midnight if not specified. (e.g. new Date(2013, 9, 19) in Brazil) The rest of the locales are much smaller and less likely to have customers you will break: Brazil, Chile, Paraguay, Cuba, Greenland, Iran, Lebanon, Syria, West Bank, Gaza Strip.
There's even a silly bit of Australia that has a TZ on a 45-minute boundary (UTC+8:45). Eucla is a tiny place in the middle of nowhere, population 86 in the town, though I don't know how many others might be in that TZ.
If I recall correctly from the last time I had to deal with this, it's the first 3 towns on the Eyre highway in WA coming from the SA border. This[1] suggests there are actually 4.
Well that makes sense in the real-world; Broken Hill gets South Australian television and radio and is much closer to Adelaide than Sydney (500km to 1200km), it makes sense to be on ACST not AEST
Border Village, a roadhouse on the other side of the WA/SA border, also observes Central Western Time Zone. However, the next stop on the Eyre Highway does not.
Not as much of a joy as when an upstream dev is snottily reverting fixes for a commercially-supported open source product because they refuse to believe in the TZ the customer is in exists.
I am currently working on an application which just uses offset numbers to tell how far away from Greenwich time the zone is. No daylight savings time offset, no south of the equator, etc. Just a hard coded number for each client.
I haven't verified it by any means, but this to me smells like ES5 15.9.1.8[1]. Basically, prior to ES6, Javascript engines did not prescribe the use of IANA (nee Olson) tzdata to resolve historic dates.
The standard says: "The implementation of ECMAScript should not try to determine whether the exact time was subject to daylight saving time, but just whether daylight saving time would have been in effect if the current daylight saving time algorithm had been used at the time. This avoids complications such as taking into account the years that the locale observed daylight saving time year round."
In Spidermonkey, at least, it mapped current years onto prior years where the current day falls on the same day of the week.
Daylight Savings is nothing but a frustration nowadays, especially in Australia where a) our DST does not quite align with the switchover in the US, but only by a couple of days, and b) out of the four states on the east coast time zone, one of them does not use DST which makes things more confusing.
I never liked the argument that DST gives you a consistent sunrise and sunset time, because it can't. DST can't extend the amount of daylight for obvious reasons.
It's rather like redefining the metre so that everyone is 2 metres tall - you're adding a lot of complexity for not a lot of consistency.
Who makes that argument? Daylight savings doesn't give you consistent sunrise/sunsets.
For example, in Melbourne, Australia, in the winter it will be dark by around 5:30pm. In the summer it gets dark by around 8:30pm. I like having inconsistent sunrise and sunset times (e.g. extra daylight time in the evenings in the summer).
Neither do I think daylight savings extends the amount of daylight. What it does is shift the typical working day so that by the end of it, there are still a few more hours of daylight left. That's what I like about.
I sometimes think that people on HN live in this artificial bubble where everyone has same arrangements as programmers or work in startups. There is plenty of posts like yours, or "why don't you just work from home a couple days a week?". Most people work fixed schedules, they can't "go home an hour earlier".
I do live in a bubble. That said, it seems to me that there's nothing stopping, say, a factory, from saying "well now, why don't y'all come in at 730am so you can get home earlier". Seems more elegant than changing your UTC offset, especially if you actually use local time for anything, as now you've gotta account for 25 and 23 hour days.
I guess if a business is in the past where screwing around with the clocks at nighttime is unnoticeable, maybe it's an acceptable solution. But in 2015, DST seems as bad as leap seconds. Perhaps even worse.
>here's nothing stopping, say, a factory, from saying "well now, why don't y'all come in at 730am so you can get home earlier".
John is a factory worker who can't afford a car, so he takes the bus to work. Does the bus schedule allow him to come in and leave an hour early?
Frank and his wife both share a car; Frank has to drop his wife off on his way to work. Can Frank's wife come into work an hour early as well, or does she have to spend 9 hours at work and Frank has to wait an hour before he can pick her up? How about Jerry, who gets dropped off by his friend Lloyd because Lloyd has to drive that way anyways and doesn't mind doing Jerry the favour? Does Lloyd have to start running his daily errands earlier, now?
And don't forget that whatever solution you engineer for all of these people, it can't be too permanent, because we're just going to reverse it in a few months. And then re-reverse it later on.
What you say would apply if I were talking about people who aren't on HN.
We are talking about a person who is using his personal preferences to dictate the life patterns of everyone in the world and characterizes it as a "trade-off".
It's not always as simple as that. Besides it's far simpler from a personal perspective for me to set my clocks forward/back twice a year that move my entire daily schedule forward and back twice a year.
It's maybe a selfish reason, just to get a little more daylight in the evening, but I'm not the only person who thinks this way.
But... isn't that just semantics? I mean, you ARE moving your schedule twice a year. At least according to your brain. That a clock says is secondary I'd think.
No, what the clock says is primary, because it means that's what the people around me are doing also.
If it's only me moving my schedule, then I'm getting up an hour earlier, eating breakfast an hour earlier, starting work an hour earlier, getting hungry for lunch an hour earlier, finishing work an hour earlier, going to sleep an hour earlier etc.
It puts me out of sync with everyone else around me. Yes I'd get used to it eventually as would people around me and then I'd switch back half a year later.
If however the clock moves, then everyone around me is moving their schedule too, and we are in sync.
Put another way, it's taking advantage of light hours in the morning that are otherwise wasted sleeping. In Tokyo it's light outside from 4am in Summer, if you can instead have an hour of that in the evening then that's an extra hour people don't need to turn on their lights.
I was under the impression that people can decide for themselves the times when they want to go to bed and wake up. If I lived in Tokyo, I would likely choose to wake at an earlier clock time, but still at the same offset from whenever local noon may be.
If someone built an alarm clock with a GPS receiver that used the equation of time to calculate apparent solar time, mean solar time, and civil time from GPS time, and if it allowed the user to set the alarm as an offset from local sunrise, noon, or sunset, would anyone besides myself buy it?
I think if more timepieces were more informative about how much difference there is between civil time and mean solar time, people would be less tolerant of daylight savings time. But those timepieces would also have to handle all the time zones plus the multitude of stupid rules different civil jurisdictions use to calculate DST. Hopefully, the wrist-mount computers emerging onto the market will make it possible to create "DST howling outrage generator" applications.
>I was under the impression that people can decide for themselves the times when they want to go to bed and wake up.
That's correct, and most people choose times that are roughly in sync with the people around them. Waking up an hour earlier is less useful if the gym isn't open an hour earlier, or your favourite breakfast stall isn't open an hour earlier etc.
If your life is so strictly scheduled that you can't adjust parts of it an hour or two in either direction, or swap morning activities with evening activities, your problems are bigger than daylight savings time.
Some people work 2nd or 3rd shift, and a few of them even do it by choice. If you think moving things around by one hour is difficult, try shifting them eight hours. Those people still manage, though they often acquire sleep-related health problems and have to deal with more inconveniences in their daily life.
>our DST does not quite align with the switchover in the US, but only by a couple of days, and b) out of the four states on the east coast time zone, one of them does not use DST which makes things more confusing.
Quite a few years ago I was working in WA on a system that frequently aggregated data from servers located in different states when they decided to try daylight savings. From memory it was decided by the WA state government that the state would adopt daylight savings on a trial basis, try it for a year or two, then consider if we wanted to keep it.
For programmers it was pretty much the worst possible solution. Have it or don't have it, don't force us to make sure everything supported having daylight savings in WA and also require the ability to possibly turn it off again in the relatively near future.
I remember when they changed the daylight savings start/finish times for the 2000 Sydney Olympics. Had to go though a bunch of machines and manually patch in the new tzfile (this was before config management).
Also remember at one point NZ was having a potential power shortage caused by a drought[1]. Some senior government officials were suggesting bringing forward the start of daylight saving by a month with around 2 weeks notice.[2]
[1] - NZ gets most of it's power from Hydro
[2] - The jury is still out on if daylight savings saves power, especially since different places have low/high aircon usage.
I once tried to work out, based on power usage levels, if DST saves power (it's original purpose in wartime)
At best it's a wash. People all think that domestic lighting is a big power user, but in reality it's things like commercial shopping centres and office buildings that really suck down the juice. And typically those are run on a timer, so the pattern doesn't change (even if the timer is adjusted, the lights,ac and services are on for a fixed period of time).
Any savings for people having their lights off for 60 minutes less is swamped by people switching their AC on when they get home an hour earlier.
I still support DST as a quality of life thing, though.
> I still support DST as a quality of life thing, though.
I was amused at this comment, since i always dread the week or so after transitioning. Invariably my biological clock is screwed up and i'm something of a zombie in the morning at work and the later evening at parties.
It probably did save energy once, back when it was first introduced because I doubt most people used electricity for much more than a radio and lighting. Sure the rich likely did, but there aren't that many of them.
Well the idea is from before electric lighting was even a thing, so who really knows? I do wonder if the Russians have it right in their recent abolishment of the system.
As I recall, there was a Windows hotfix at the start and end of the trial, specifically to allow for this change.
WA Govt was very keen on us voting for it, and assumed that with a trial we would get used to it and it would pass. As it turns out, that time it was rejected by the biggest margin so far.
SQL Server 2005 actually shipped with code that used the local time instead of UTC time in CREATE CERTIFICATE. Users in Western hemisphere (local time behind UTC) had no major problems because of it, but users in Eastern hemisphere (local time ahead of UTC) would create certificates not yet valid. Confusing, to say the least. See http://rusanu.com/2008/08/25/certificate-not-yet-valid/
When I worked at Mozilla, we had an issue where the unittests will always past, except on very few occasions. The message was not entirely obvious what the problem was (from as far as I remember) so we ignored it for a bit as it didn't seem to impede any work.
We eventually realized that the unittest will only fail after work hours, at which usually no one is around to start tests. It turns out the test will fail during 6PM - midnight because of an UTC issue.
I actually had to deal with this exact problem two days ago. I was baffled because the timezone was being listed as UTC+14, and that places like Kiribati crossed the date line. UTC+14 didn't exist until the date line was crossed.
I have an app where users are running it while crossing time zones. :-(
I love Tom Scott's computerphile videos. His frustration grows over a few videos; his videos about the difficulties of internationalization[1] and the problems of electronic voting[2] are even crazier.
I used to work at a phone company in Australia. DST was the pits. Think a five minute call where the caller drives across state lines, from a DST-observant state to a non-observant state. Timestamps come from the base station through which the client's phone is connected. several weeks later, client is furious that they have been billed for an hour and five minutes (back when it was expensive).
Every time we switched to or from DST there were a couple of outliers that got caught. We'd update the tables to handle the case correctly next time, but would miss something else, if the change didn't break something else completely.
If the problem is indeed network related, I'd fire up Fiddler. You can install a Fiddler SSL cert locally to decrypt your SSL connections, it'll play man in the middle quite effectively and should hopefully give you some more intuition as to whether that's the cause of your problem.
Why do you have unit tests hitting outside services? If you have developed service clients that need to be unit tested, you should have mock services for them to run against locally so that your unit tests aren't going down every time there's a remote service failure. They don't need to be full blown services, but they should receive the request and respond with the expected result the service would give when hit by the client.
If your clients are generated by the remote service [i.e. by WSDL], then you shouldn't theoretically need to test those as they're generated by the third party - so you should be unit testing against mock clients.
I'd argue that integration tests with real world 3rd party services should be handled by some reporting system on your build pipeline somewhere and your ops team should be investigating that failure, you shouldn't need to be running them locally on your dev machine.
We used to have the opposite problem (being based in Australia): certain unit tests would run fine locally, but then end up failing when run at certain times of the day on our Travis server which was in the US.
"Chrome 31.0.1650 (Mac OS X 10.9.0) ngMock TzDate should fake getHours method FAILED Expected 4 to be 3."
Hard coded dates not taking into account timezone and daylight saving? Been there, done that. This week the time was AEST-DS (Daylight saving) and is now AEST (finished daylight saving) [0]
[0] For example: Melbourne
AEST: Australian Eastern Standard Time == GMT +10Hrs
DS AEST-1 hour when daylight saving is declared. When Daylight saving finishes, DS is AEST+1 hour or AEST again.
Exactly. And the writing appears upside-down, naturally. After all, gravity points in the opposite direction as on the other side of the earth; that's why the Australians are called antipodes :-P
Then I was coding on the train home one night and in the tube my hotspot lost connection. Then I happened to notice... with wifi off the tests ran 10x quicker. Pull out netstat.... I see http requests to akamai essentially as each test setUp() is run. It's SSL so I only see a hostname, not the full path, and they just look like this, but the hostname is dynamic: a23-34-132-181.deploy.static.akamaitechnologies.com
I spent 30 mins reviewing our code, grepping my code and packages directories, and stepping thru in a debugger. No luck.