This reminds me of the story one of my university teachers used to introduce himself. He was the most brilliant educator I've had, really passionate about making sure people could actually follow and so humble. The course was called "Digital and Computer Technology" and it was a first year course teaching you to build a CPU starting from logic gates and then program it with assembly.
He said that when he had recently graduated and just moved from Norway to Sweden his first job here was as a technician for the local railway. One of his first assignments was a call where the computer controlling all the track switches had stopped working. Luckily there was a backup, but they needed him to fix it immediately.
He arrived, took a look at the computer and its backup running next to it. He started out by measuring voltages on both, comparing to see what the difference was. After a while some men in suits came in and asked how things were going. He said it was all good, they said it wasn't because now all trains had stopped. Apparently, he had short circuited the backup.
"And that's when I decided to go for a theoretical career!" My teacher happily concluded. The classroom was left in a stunned silence.
My favorite story about him was when he caught a student sleeping in his class. He paused, mid-sentence, and asked the student if they were too tired. The student looked up sheepishly and confirmed.
"Did it get a bit late last night? Can't keep your eyes open?"
Again, the student agreed. Rolf flashed a huge smile and in a singsong voice said "then I have something for you!". He walked over to his bag, pulled out a stuffed animal and a tiny embroidered pillow. (I have a grainy photo of them somewhere).
Rolf tiptoed up to the student, put the pillow in front of him, and as the student rested his head on the pillow Rolf gave him a pat on the head.
After this Rolf went back, said that if you're that tired it's better to sleep in your own bed. He resumed the lecture, picking up in the middle of the sentence where he'd stopped. No one slept in his class for the rest of the course.
I have a similar anecdote which completely stunned us, but which may possibly be part of his standard routine, and thus something he does for every set of students:
He always wore the same striped sweater. At one point, he took a break to drink some water and remove his sweater. But beneath the sweater was another identical sweater!
I assume everything he did was very well-planned and part of his pedagogic system and routine, but it was often hard to tell if he was acting, or if he was actually being serious but eccentric.
I don't remember that but a lot of his antics seemed to be planned and part of his way of keeping people awake and paying attention. If it is part of his routine I wonder when he added it. I took his courses in 2006-2007.
I took that course in 2015 or 2016, so yeah, I guess that's about a decade in which he could have added it. I just browsed around Chalmers' course pages and am very happy to see that he's still teaching that course.
Concrete example from that article: with improved detection of cancer, it’s true that “people with cancer live longer on average” and also that “people without cancer live longer on average” even if there’s no change to the treatment or anyone’s lifespan.
I absolutely loved his classes. I bumped into him a few years ago when I was holding a recruitment thing for Ghost Games and it happened to be just after his lecture. I told him how much I enjoyed his courses.
I had a professor in college teaching computer systems and design. Somehow he got to talking about the NSA. He said that back in the 60s, the NSA had computers that were running at 1000. “1000?” I asked, wondering if he’d forgotten the units. He replied “Yes, 1000.”
The professor is incompetent because he doesn't actually know how to talk about the performance of computers. It's not really a "joke", more of an anecdote
I heard from a group of teachers (my mother was a teacher) that were at our house once. "Those who can, do; those who can't, teach; those who can't teach, administrate."
> Maybe if we paid those who teach better, we'd attract the best of the best to teach.
I agree struggling for basic survival as a public servant is not ideal, but be careful not to turn education into a lucrative opportunity where people come for money not passion.
Money is certainly not the only way to attract talent (though teachers still need to pay their bills) and in most schools i've attended the talented teachers were just ridiculously hindered by the bureaucrats defining their program and agenda. David Graeber's talk "Manageurial feudalism and the revolt of the caring classes" comes to mind.
I believe that we should be making whatever changes we need to ensure that some of the most capable minds in our field are doing something more useful than ad tech. Speaking from experience, seeing where my classmates went to after we graduated a few years ago.
We should pay better those who teach better, not everyone. Otherwise we just make the market bigger with even more room for incompetence. Look at the software industry.
And what kind of metrics can you use to determine whether someone teaches well? Whatever metrics you use, there's probably ways to game the system without providing better services.
Exactly, and this would be just as true, only on bigger scale, if we just raised wages and tried to filter out 'the best of the best' from a now larger pool of candidates.
I should write "if anything" at the start of my previous comment as I don't really believe a good solution based solely on financial incentives exists.
Your other comment efficiently expresses my intuition on the topic.
While having algorithms and computers dictating every aspect of train movements, crew scheduling, resource allocation is efficient and in a sense optimal, I think there's something compelling to be said about having a system that can survive with human-rememberable patterns when such an optimizer fails. That it can fall back on (or have at the core) simple patterns that people can still operate without algorithmic intervention. I.e. the algorithmic tuning enhances at the edges, and isn't at its core some tangled non-interpretable system that someone can't operate without computer assistance. The idea of a protected set of core routes and staffing patterns that operates without having to be computer-determined?
I used to be so impressed at European train stations that they had a single sheet that would give the entire day's arrivals and departures, down to which track. I assume that behind that were schedules that weren't so far from being hand-calculated with equivalent sheets that told people and equipment where to be at what times.
I think we lose something when we jump so far forward that you cannot fall back gracefully without a system completely breaking down.
I work within the train system of a European country (with ticketing, so I’m on the top and not the bottom of things). From my experience with station and onboard crew, I am very certain that the schedules and patterns are still very much human rememberable, at least collectively: People have to show up for work, drive the train and many other things.
My guess is that the failure is related to signalling or some other knowledge that exists “between” the trains (and brains). Having systems that can negotiate right of way is necessary to increase the number of trains and the speed of each individual train beyond their pre-computer-era limits.
Having all trains stop for one day is probably a judgment call to lower the overall duration: Having all trains stop for a day is a pain in the neck, but you know when you’re done. If you could not wait, you could probably have (quite) a few trains run without problems. However, you would have to work for a very long time to get all the trains back to where they were to resume the regular time table. This would be the preferable option during a time of crisis.
The Dutch railways said (in Dutch) themselves that it was a problem with the scheduling software which also has a role in safety. Apparently this is also used to route around for example a stopped train, so when it's not working it's not safe to operate anything but a handful of trains.
There was a lot of extra impact because this system also feeds the passenger information so their app, website and information screens at the stations were not working either leaving passengers in the dark.
There is no correlation between the schedule you see at Dutch train stations monitor screens, and the actual schedule of Dutch trains:-)
You might see this is an ironic comment, but it is as much that, and as much a technical statement of fact. Sometimes the trains arrive at the correct time
( +/- 10 min) so they correlate to what you are seeing on the screen.
But unless it's a delay of 1 hour or more, neither trains or tracks will be updated on the screens. It's just happen they sometimes coincide, so they might correlate, but only by coincidence. Trains schedule and train reporting on the train station are, I was told..."Run by different companies!"
-> Source: Conversation with platform manager at Amsterdam Central Station a few years ago. ( not that many years...)
They got GPS tracking on the trains. Any delay over 5 minutes will show up automatically, less than 5 minutes you can see on the app. The train operator took over the information screens, so it's all part of the same company now.
You either misunderstood, misremembered or were misinformed. As long as I've been taking the trains (20 years or so) here, the monitors have always updated with schedule changes. Even when the information screens were still analogue, they updated with delays and track changes.
It might be that what you heard only applied to international trains, those have generally lagged behind technology-wise (though nowadays there's certainly also real-time schedule information for them).
Not true. It's not if the screens get updated. They do not get updated based on the actual train on the track but from information info you can infer from an pre-established schedule.
Let me tell more about what prompted the conversation with the platform manager.
I was staying at a hotel in Amsterdam while doing an IT project in the north of the country. One of the days took the wrong train. At that time, I just attributed it to not really carefully reading the platform indicator. The next day had to take the same train, at the same time, carefully checked the platform monitor, but as I had been burned the previous day....Decided to also ask the train ticket controller at the train door. He laughed and pointed me to another different indicator some trains have, and that happened to show a different destination for that specific train.
The cognitive dissonance that I experienced was so strong, I decided right there to also be late that day, look for the train station manager, for a polite but frustrated conversation on my part, and it was then he clarified this.
Also any Internet search will show others with similar experiences.
Edit: Looking at this example from 6 months ago...Continues to happen:-)
> Not true. It's not if the screens get updated. They do not get updated based on the actual train on the track but from information info you can infer from an pre-established schedule.
Not true in general. The screens do actually get updated based on the actual schedule when there are track changes and/or delays. They don't stubbornly stick to the pre-planned schedule. I've seen that happen countless times. That said, occasionally I've seen the screens lag behind reality a bit, usually during and around big disturbances.
> Edit: Looking at this example from 6 months ago...Continues to happen:-)
Different system, that'd be the switches/signalling. Freight and international trains were running just fine, safely. This was literally just planning which train (rolling stock) is going to operate which train (ride) with which personnel.
Safety could be related to duty times, or making sure the rolling stock is fit for a certain trajectory (maximum platform length, weight restrictions etc.)
No the section a train is on is automatically and mechanically secured by the lights on the section ahead and afterwards. (2 sections if you count the yellow warning lights)
Unless they are operating a dynamic block system like the channel tunnel lines and other high speed lines across Europe. Anything that runs over 125mph (or has to mix with traffic which does) isn't using block signalling.
True dynamic block ("moving block") still isn't that widespread, at least as far main line railways are concerned. High-speed lines are still by far and large using fixed block signalling with conventional train detection (track circuits or axle counters), it's just that the movement authority is given via some sort of cab signalling.
Signaling is on a separate system. There is ATB-EG/ATB-NG for max speed and there is ATB-Vv for stopping for red lights.
Part of train history in the Netherlands is the train disaster at Harmelen in 1962 where 93 people died. It spurred the installation of these systems. Every time there is even a minor accident with a train, the national governement is asking questions, up to the highest level. For contrast, every time a child dies in traffic, only the local government (gemeente) asks questions. It is considered completely unacceptable for a train accident to involve more than 2 deaths, it seems.
https://en.wikipedia.org/wiki/Harmelen_train_disaster
The problem was in the scheduling of personnel and trains. The train schedule in the Netherlands is not human memorable. This is something they tried to correct in the past but was met with protest from employees because then they would only have one route on a day which would be boring.
Try to run the pricing system in Denmark if it hasn't been changed over the years.... It was "deem" "impossible" for a human to calculate the pricing within a reasonable amount of time (with all the associated bonus).
Do not forget that there might several companies running on the same track, cargoa and passenger trains etc.
What is remarkable here is that usually there are a lot of answers in dedicated golfing languages that can solve seemingly very complex tasks in just a few tens of bytes, here there is only one such answer, and it takes 115 bytes. So perhaps one could say it's even hard for computers?
The train schedule as seen by passengers stays the same for years, often to the rail number in stations. What changes are the crews, wagons, locomotives, possibly routes on the railway (switches and reactions to delays and failures.) A friend of mine worked at an app used by a railway's personnel to know where to go next. The app was not particularly complicated but the backend, which has been running for ages, took a long time to run. I don't know the details.
I was going to say it kinda reminds me of the AI(s) in Neuromancer. In that novel it felt like the AIs were 1000 steps ahead of humans and pretty much just using then as pawns, selecting them carefully from the population, knowing they could either abuse their conditions to control them, or just flat out blackmail them.
Are you posting it reflexively because 9 out of 10 times Asimov is mentioned that's the story people talk about? Because that's definitely not the plot of it.
> I assume that behind that were schedules that weren't so far from being hand-calculated
It’s more than that: The trains are negotiated at political levels! The “8:18 to Marseilles” (fictitious example) could be a headline in the news if the region refuses to fund it, and the worker’s union may have striked to keep it, while inhabitants’ HOA has negociated with the city to keep it under 12€: They are the object of a convergence of fixed interests. They run for generations: the train I took as a kid is still arriving on the same track today (and in one stop, it stops at track E; tracks A-D were dismantled but never renumbered due to this legacy).
The good thing with rails is that they aren’t going away, and trains can be negotiated for decades, they’re far from being scheduled on-the-fly.
You also have to ask at what cost to maintain the backup system and how easy it is to test the Disaster Recovery process. If once in a while, you have a few hours delay (or in this extreme case, a whole day) you might ask if it would be worth it, especially if you could only test it by adding random failures to the main system like Netflix do!
As another poster mentioned, they are many, many independent variables in a railway system. In the old days, people recovered from problems by gut feel and experience was OK although there is usually no "right" answer. The newer systems apply some statistical anlysis and try and make the best decision usually with humans doing the last part.
I'm not sure about "because safety" though since signalling systems protect trains from each other and they only have automation in the sense of first train that arrives gets it route sets, the system cannot make trains crash unless there is a critical bug.
As a general response not specific to this system failure: Train systems are designed to fail safe when not in state and significant safety engineering is done to ensure that if a rail system remains in a known good state over multiple decades almost every potential threat to human life caused by various equipment failure possibilities is quantified and reduced to a level of risk that’s acceptable. (Zero typically, for any known non-adversarial risks.) Many of the pathways in the fault tree involve shutting something down if it’s out of the correct state. So you will see train networks fail safe rather than run in a regressed state since these vehicles with up to a thousand human lives on board drive themselves to some degree in every modern system and we tend to hold them with a level for safety incompatible with easily winging it when something is wrong. This absolutely makes the system more brittle and less resilient, but for the most part it’s the tradeoff we accept in most modern rail engineering projects which are designed with the goal of reducing human loss of live to zero over the entire span of the system’s operational lifetime.
That said, train systems themselves usually maintain a ton of active operational procedures that remain part of staff training which would theoretically allow them to be more resilient and these procedures are kept current and you will often see them used in emergencies. I.E. on automated systems, trackway signaling often exists in the “not lit” off state and if the automated train control system failures, is often capable of lighting up for human controlled movement (usually required to be done at reduced speeds) of the trains during irregular operations like bringing stuck trains to the end of their lines and letting the people in them out at the next station. And any trains that still have a human conductor in a cab (or a cab for a human conductor at all) have procedures for that human to operate the train directly, even on segments of track where automated train control otherwise control all train movements.
We could try and run these systems in these degraded states under these emergency procedures, but most modern systems have safety analysis and engineering which focus on bringing the system to a safe halt state while fixing the underlying issue and returning to full operational state rather than messing around with rolling along in degraded states with unknown but assumed to be increased potential for catastrophic engineering failures with outcomes including loss of life.
(In the US it is generally illegal for a human being to operate a train unaided by some form of train control above 79mph. Most other systems have similar rules. We no longer trust humans themselves to operate these vehicles just because so many human lives can be at stake when they fail.)
Source: I did a stint working on system security and a little bit of electromagnetic compatibility safety engineering for rail systems. It was enlightening to see how the capital E engineers I was working alongside handle these concepts and design risk out of modern systems. The bulk of my work was in the US but the parts that weren’t were on systems that spanned multiple continents and/or train cars which were distributed worldwide. (To my knowledge none of my work has ever related to system discussed here.)
My curiosity is killing me as to what exactly went wrong. And as someone living in The Netherlands I'm also kind of mad at the apparent fragility of this huge chunk of our infrastructure. If this had happened on a work day, it would have been a real national emergency, rather than just a huge national inconvenience.
I would appreciate one of those post-outage "what happened" reports like with the AWS and Facebook outages last year. But outside of IT I don't think anyone really expects those over here. And there might be national security considerations preventing such disclosure until any chance of a repeat has been engineered away, at which point everyone will have forgotten.
> It affected the system that generates up-to-date schedules for trains and staff.
...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole.
I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system.
> A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days.
Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables.
> In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently.
Cool stuff.
Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum.
Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work.
This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.
Indeed pretty standard, even in non-government it would be 40 hours but all other terms the same and most likely a company car + private use of it instead of the train.
And I'm also sure they have a significant on-call setup of the IT teams for any out of office hours issues, just like every other large 24x7 kind of organization.
Note that a pension is essentially a legal requirement. There might be ways around this, but it would be really hard to offer employment without offering a pension in NL.
Only AOW (state pension) is a requirement by law and for full-time salaried employees there is no way around that. Private pension (what is meant here) is not required by law though financially very lucrative compared to doing it yourself. It can be made required by making a CAO (collective bargaining agreement) "algemeen bindend" for all employees in a sector/company but this is not always the case in all sectors (e.g. IT). Note that this is way more complex and this is just a simplification.
Really, I thought pensions were required. I guess it's just so fiscally attractive that essentially every employer offers it. Combined with the pervasiveness of CAOs, especially among the more stingy sectors.
I worked about a year at NS as a consultant. I wrote python software that broadcasts the audio messages in the passenger compartment, steers the outside lcd screens and such.
It is a great place to work. A nice atmosphere, good coffee and they embraced change (in this case Scaled Agile) better than most places I have seen. They are often listed as top-5 best places to work and I can see why.
>36 hours is considered fulltime by most employers
How standard is that in the private sector or non former state owned enterprises? I haven't looked at the NL market in a while but most tech jobs I saw back then were 40h/week.
It's hard to find actual numbers. Tech jobs, smaller companies ("MKB") and work in the agricultural sector tend to have a higher percentage of 40 hour work weeks than other sectors.
I have been working between 30 and 24 hours as a Senior Software Developer in the past 20 years. I have had several colleagues who work a similar amount of hours. I have been telling employers that I am only productive for about six hours per workng day on average and that never has been a deal breaker. The number of productive hours does vary from day to day, but I know that I push myself beyond that limit, I will run in trouble after some months. I have heard about studies that the average employee is only really productive for about two or three hours per day. With me working six hours per day, I might just as productive as someone being eight hours in the office.
Most private sector tech jobs in NL are 40 hours per week. But many private sector jobs are also flexible in ways that I've never seen in the USA. For example, allowing an employee to work 40 hours in 4 days instead of 5.
> everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife.
My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.
Have done consulting for 15 years, understand your feelings. But I'd like to offer some alternative viewpoints.
Buying an off the shelve package also buys you an industry-standard business process. The idea is that part of value comes from the process that you have to shoe-horn into. Some integration is expected (and provided for) but if you have to modify heavily, You're Not Using It Correctly.
Also, using an OTS package means less development risk (not every IT dept has 10x developers coming out the wazoo) and it also means that you have a huge reservoir of skilled labour if you need it. Not just for development but also on the business side.
Finally, if you're a big, publicly owned company, you likely have many financial / reporting constraints. The scope of your non-negotiable requirements may be bigger than you imagine.
I've seen people implement SAP in a couple of months. It's possible. It's still expensive and clunky though :-)
> I've seen people implement SAP in a couple of months. It's possible. It's still expensive and clunky though :-)
Never worked with SAP, but still coming off my burn-out hangover from $OLD_JOB - worked to migrate the nexus of the firm's accounting systems from COBOL to MS Dynamics. Unto itself, I don't really have a problem with Dynamics - I do have a problem with MS and their piss-poor available materials on the topic. While I understand their business sells training and certifications, this does ass all for folks pulled in as a pinch hitter during a death march. Can scarcely fly out to Las Vegas for a week long vacation while I'm working 16 hour days to beat the clock on our last true COBOL coder's retirement date.
The greater issue I have in enterprise systems - often the consultant / contractor firms[0] involved in the process. The classic fly-out of classy sales-engineers replaced with low-rent off-shore coders for the actual work. Slick-as-shit documentation that does not match reality in even the simplest of cases. Counter-Party contacts that neither care nor hide their lack of caring for the outcome of the project. Pressing for aggressive timelines that do not meet reality
I would assume there is some correlation, however, between buying $ENTERPRISE_TOOL and an unwillingness to hold consultants feet to the fire on their own failings, structuring a contract that keeps them honest.
[0] - Not to state `ArnoVW is one, only stating a pattern I've run into all too often.
Thanks for the footnote =) And I concur. Outsourcing your IT replaces one type of work with another. If you can't manage your externalized projects, you're gonna get eaten alive by IT services suppliers.
In fact my line of work was managing 'complicated IT projects', generally involving formalizing things and chasing said suppliers (ie they outsourced the outsourcing)
You see, when your Foxpro guy retires in 10 years, either you've been training your new hires how to program Foxpro (a waste of time) or you'll pay a specialist to move all your business data and operation macros to a new platform (a waste of money).
SAP lets you stop kicking that can down the road by wasting your time and money today.
I’ve never seen a SAP install that wasn’t modified heavily, usually to the tune of millions of dollars of development - which can’t help but make me think that if nobody can use it “correctly”, then perhaps it can’t be used in its unmodified form.
In addition, a common occurrence seems to be businesses paying for extensive custom developments (such as an integration with a third party system), and footing the entire development cost - despite their “custom” development being 100% identical to that deployed at other businesses. They’re then trapped in a world where changing the URI for an API endpoint costs €150k.
Honestly, I see it as just being a big corrupt hole fuelled by interpersonal relationships rather than any actual rationale.
My previous boss recently tried to pick up an enterprise solution. He runs a non-software company that has a lot of custom internal apps - 90% built/maintained by interns. Of course the intern-built custom apps are super buggy but the bugs get fixed immediately, and the system is tailored to the company's way of doing things. The company spent an obscene amount of money just demoing a full-fledged SAP-type thing to potentially replace the custom stuff, but they ultimately found the rag-tag web apps better and cancelled the SAP contract.
That whole situation convinced me pretty thoroughly that "enterprise" software is indeed a form of large-scale snake oil. I think most businesses can go a long way with just spreadsheets and email.
Nothing gets me as excited as a lowly footsoldier in a company as when the c-suite decide to spend $100mil doing a custom install of a big name software stack that will "totally change the way we work and at the same time be fully customised to the way we work here".
This is one of the reasons I'm bullish on no-code doc-apps like Coda [0]. I recently started consulting with someone who calls himself "The Coda Guy" and pulls together truly wizard-like apps out of Coda [1]. (Coda is effectively a hybrid of Excel, Google Docs, and Notion.)
I think the "Excel+" category should receive a lot more attention than it does now. I can easily see "no-code" tools like Coda taking over Salesforce and effectively filling the niche you describe.
Agreed, except for the spreadsheets. Please get people to use server based databases instead. They are no longer difficult or expensive to install or use and make life so much better even for 'interns' and it will reduce their workload dramatically as well as increasing the quality of the solution..
Basically SAP is a large database/software package that runs all aspects of your business. From accounting to billing. Your business probably needs to change a lot if you’re going to use it. When I did a time card application for a big company we interfaced our app with a backend some sap engineers made available to us.
I have no idea how it works, but apparently it does.. sometimes…
i think this is under-emphasized for people unfamiliar with them.
I think SAP and related modules are "solidified business processes", and their consultants and sales persuade your CxO level people that switching business processes will improve overall efficiency (and somewhat hint at standardization, so that the CxO will have experience at that domain when switching jobs!).
But of course, it's not really true - best practice business processes are an illusion imho, and it will turn out that the subtle differences have to be implemented individually.
To be fair, that is the case for most ERPs. It's not really about the size of the product but rather the important role it plays in a company.
Most ERPs can be quite flexible too, there's a ton of customisation possible. And a whole cottage industry of addons and integrations for every imaginable integration or business model. But the problem is you really screw yourself with added complexity when it's time to update. Then you're really condemning yourself to spending millions on consultants at a time when you might not have the resources. And postponement leads to technical debt.
Hence the idea of adapting the business to the system. Making sense too if you think about it this way: you're not really just buying a system but a methodology.
And another thing is that it makes integration with other companies easier. Makes the company more easy to sell or too acquire other companies and integrate them.
They decided to modify a number of core functions to work the same way as their old terminal system instead of the default ERP process. Now our job for the next couple years is decoupling the custom add-ons so we can safely update to a modern ERP system.
This was true even for Royal Dutch Shell, one of the largest companies in the world. A few years before he retired, my dad led a team responsible for coordinating implementation of SAP and for training thousands or even tens of thousands of employees to work with the new system.
I read recently that not using SAP - or heavily customizing it - in markets where it’s prevalent can break big(ish) companies. Apparently the interface to other companies (from which to buy or sell) is so much easier to setup when all have the same system, that using any other system is prohibitive. If you need to interface with hundreds of enterprises that all implement standard SAP, you become the expensive outsider if you don’t. Integration projects can take years and then fail, with nothing to show for - which can threaten a company’s existence. Better to use standard SAP „off the shelf“ and do as little customization as possible.
Without necessarily wishing to call into question your source on this particular occasion, it is of course in SAP's interest that as many as possible believe this to be true.
One of my customers was taken over during their SAP implementation. The general feeling among management was the work had weakened them so much it made them vulnerable to takeover.
SAP is mainly an ERP system, but it has various modules dealing with stuff from Purchase Orders and Materials Management to HR. It can be very large and extremely expensive, like over $1 billion for one company in ~ 20 years. It can be customized in many ways, good consultants are very expensive and the cheap ones are usually a waste of money. Without customization very few companies can map their processes on out of the box SAP.
It was developed a very long time ago, it has a huge adoption in large companies and a few years ago when they migrated from R3 to Hana one of their own people told us in an architecture review meeting that they have over 300,000 built in reports to migrate.
In many cases companies are adopting SAP because "nobody was ever fired for buying IBM". Anecdotical, the company where my brother works is implementing SAP and changed all existing networking gear with Cisco because "this is what big companies do". They threw away brand new equipment and spent several millions on the SAP implementation that is still not working, forecasting they will pay even more millions to make it work then millions per year to use it.
Think Oracle, with business logic on top. (JDEdwards, Oracle, SAP) were the OGs of BI.
(and bloat, and McKinsey infiltration into many a large F100+)
--
They are old school, entrenched, but yu would jump off a cliff supporting any of this legacy stuff. It will persist as the inertia on huge systems is heavy.
But yeah - I wonder if they are signing any NEW business... aside from military bunker grade stuff (which will cost as much as the bunker to implement) I dont see a reason to implement (not because I am bashing, but because I dont know the modern business case reqs that would call for SAP)
I don't know about “McKinsey infiltration into many a large F100+” first-hand, but from context I'd guess it's simply getting consultants from McKinsey & Company[0] jobs in the ‘infiltrated’ firm, on the basis that it has no in-house employees capable of properly setting up / running the bizarrely complex and idiosyncratic SCM-ERP-HCM-BI-etc. software in question.
Basically: You have a problem; Oracle/SAP says they'll solve it for you. Now you have a second problem; they say McKinsey'll solve it for you. Now you have …
Its the cancer that permeates your C-Suite. Meta-stasize-ing,
God I love etymology:
"-Stasize"
**To stand on a hilltop once. To be, this time, on advance. To reach the highest clouds. To touch the limits of space. To out-grow skyscrapers. And make them look at my face.**
---
So FB is becoming a meta-stacious being..
That which wishes to control from a vantage, or t corrupt from within the minds... The light house.
I have zero knowledge of how the Dutch railway scheduling system works, nor of the nature of this failure, but SAP has been blogging recently about using Prolog for combinatorical optimization tasks such as the one that may be in use here or in SAP's Advanced Planning [1]. There are also well-known Prolog proponents from NL.
because accidents and delays happen anytime, and if the schedule can't adapt, you can't run the trains.
people seem to believe that running trains is some simple thing. it is the exact opposite of a simple thing, even for small systems with only a few dozen trains
1 hour more than the standard French contract, and there are also far more holidays here..
> free unlimited 2nd class + low cost 1st class train travel.
.. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.
I work for the Belgian railways and I have a free unlimited first class pass for Benelux (Belgium, Netherlands and Luxembourg) + lots of international free tickets, also in 1st class. But no company car, obviously.
I love deep diving on public info to determine the internals (even, not only SOPs, policies, but also *governing culture* can be easily gleaned from reading what is placed in public view.
For example, The Salvation Army is one of the biggest jokes in all of charitable history.
I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID)
TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force.
They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion.
They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant...
They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market.
Its a fn racket.
=---
The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..
Please go into details, I always think it's important to call out charities that are fraudulent and going into details helps with detecting if other charities are fraudulent or not.
What lead you to look into this? How did you originally suspect them of being fraudulent? What clues did you see in their financial documents?
For example, I tend to look at Charity Navigator and for Salvation Army, they mostly get dinged by not having an independent board (which to be fair is a clear red flag) but not much more, so as someone who does want any contribution I make to charities to go toward their cause, I'm very interested in any methodology around researching a charity.
I worked in defence on systems that did similiar and i remmember the one my company inherited and wow and wow it was the most complex, nested peice of SQL stored procs i have ever witnessed. It did the job, but it took HOURS to run. We had to re-write it completely to make any changes. But thats what happens when you let companies like Accenture at a system of yours! You pay them the big bux to be sueable, and they milk you like a cow.
A lifetime ago I did a stint as an Oracle consultant and 100% this. A gigantic sql query was always the answer no matter the question. I remember using sql queries to generate pdf and ps file output that would stream directly to printer feeds. So many “concat”s in that massive blob and nearly impossible to maintain for the next poor soul who had to own the code.
Learned a lot about db best practices to make these systems resilient and reportable but the whole “the database is the system” angle was very much a “every problem is a nail to this hammer” situation.
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
And we can assume that a position in the IT department just opened up...
Physical infrastructure can be fragile/fail catastrophically, as well. I recall starting a job near Seattle, 40 miles from my home in Olympia, WA and a passenger train derailed onto the highway killing a few people and backing up traffic for days. That was not a pleasant first day!
Damn this made me read a lot about this specific derailment, but also about Puget Sound, Salish Sea, Positive train control and all this 220-MHz stuff.
No but the failure caused a halt to all motor traffic on I5 in both directions which is possibly even more catastrophic than stopping the entire train system.
My very good guess is that you had a database failure caused by a crypto virus. They will not wish to announce that this was the root cause until
A. A plan of action to avoid a future infection can be enumerated.
B. They can be certain they will restore the functionality of the trains.
The other part of my guess is that they have a SQL db which is stored on /likely/ a windows server instance. I’d even surmise that this instance may be hosted on Azure but that’s speculation, not a good guess.
This is one of my leading theories as well, although I thought they usually hit on Friday night in order to catch admins off-guard and encrypt as much as possible before being stopped? "The IT failure occurred at the end of [Sunday] morning".
On Sunday there is also not as much pressure to get things running as there would be on Monday. Or perhaps this is the incentive to pay now and get things fixed on time for the work week? In that case Friday night would again have made more sense unless the attackers have some very specific insight into how much slower restoring without paying is.
My alternative theory is an expired certificate that makes some core systems just not talk to each other anymore. The announcements on the stations, for example, were also out, and they lost control of the mobile apps (couldn't make the apps show that trains didn't run, the in-app scheduler showed all was A-OK), and that sounds quite dissimilar from the core train operation service, making me think it's more of an infrastructure than a specific system's problem. On the other hand, once the app could be controlled again (assuming this singular underlying cause), you'd think they could then also start putting train service back in place and that didn't happen for hours still.
I can't really make the pieces fit together for any theory, so then presumably something multi-faceted (one thing tripping one or two other things so it escalated from restarting one component to not being able to restart the trains anymore the whole day).
Certificates make the most sense to me. The delay can possibly be due to cached certificates, having to find the person to sign the CSR (assuming in-house PKI or even just someone with access to the right email address for external PKI), and/or a CA that isn’t valid on the machines.
> On the other hand, once the app could be controlled again (assuming this singular underlying cause), you'd think they could then also start putting train service back in place and that didn't happen for hours still.
Before you can get the actual service up and running, you need to get trains and personnel to the right locations. So I don't think this discredits your theory as much as you think.
My guess for the attacker : conti ransomware group. They've been pretty active since Ukraine war. Last weekend they targeted the central bank of Tunisia
I bet their schedule once ran on a commodore 64, and then they upgraded to a 386, and then ignored the IT department for 30 years. Hence today, chickens are coming home to roost.
If it happed on a work day I wonder if peoples pandemic logic would kick in and you would start hearing, "It's not worth closing down the whole economy just because of the risk of a few people dying in a train crash"
Reminds me of when the entire Swiss train network shut down in 2005[0]
This was, however, not due to a computer glitch, but problems in SBB's electrical grid, which lead to overload of some of the lines and then to a complete shutdown of the entire grid.
Warnings were displayed on control consoles, but were drowned in 1000s of other meaningless warnings (there's a lesson for an UI designer here).
I was in the ICE from Basel to Zurich at the time and after fuming for a few minutes I figured: "What can you do?" and moved to the bar wagon. There we had something of an impromptu party until there was no more power for the cash register and they stopped sales.
In the end it was a (relatively) funny adventure for most and a huge embarrassment for SBB. I don't think they like to be reminded of that faithful day.
> There we had something of an impromptu party until there was no more power for the cash register and they stopped sales.
Funny how the entire system has ground to a halt and the bartender is still worried that their cash register receipts and inventory won’t reconcile and shuts that part down too.
I’m puzzled by the lack of resilience in retail sales. I thought “always be closing” was supposed to apply to all types of
sales. A receipt pad and a credit card imprinter under the counter would be a very cheap backup. It would be slow, but at least they would be closing.
It can even be considered a national security issue in the case of food, medicines and other vital services.
I have some experience related to cash/registers/POS.
Card imprinters did exist in Europe (Italy) just fine, but they are not used since (at least) thirty years.
Most cards that are presented to the cash now have no embossed numbers (most people have the national payment circuit called Bancomat/Pagobancomat, sometimes coupled - on the same not-embossed card - with a Visa or Mastercard debit).
So in 99% of cases there is no way to "extract" money from cards without using the card reader and if electricity or landline/internet connection are down, it is a problem, though there are a number of battery powered and GSM/cellular connected POS/terminals that may give a few hours of autonomy.
Since several years sales to the public MUST be recorded on an electronic cash register[1], there is something like a "special emergency exception" where you can annotate manually on a paper registry sales, but as often happens the exact procedure is not largely known and I believe most people would not risk a (hefty) fine should they do any mistake in these cases and prefer to close the shop.
[1] which is BTW connected via internet to the Agenzia delle Entrate, the governement agency that deals with taxes, and data from the cash register is transmitted daily to them automatically, though there is a few days time tolerance in case of connection issues
Hmm, the last time I heard of one being used in the UK I can track to a date in 2002. My uncle bought a commuter from Dan Computers the day they went bust. The credit readers stopped working, because the credit company pulled access; but the retail staff did not realise this, so they broke out the imprint machines and kept going. Presumably the receiver didn't notice the flimsies, because they never either finally got the payment processed through the banks, nor tried to collect the payment any other way.
Originally they were the main means of taking card payments in the UK, though.
> So in 99% of cases there is no way to "extract" money from cards without [...]
The card has the customer's IBAN, which you can use along with their ID and have them sign a contract to allow you to deduct the purchase via direct debit, the system is routinely used for point of sale, but is mostly known for regular recurring billing.
It's too much of a hassle and deviation from standard business practices for shops to use it in practice, so they just wait until the network is up.
But it's not because there's "no way", embossed cards were a hassle too. We've just gotten used to less manual paperwork.
No, not really, most cards have an "own" number, not connected to the IBAN (though a few do have it), it depends on the bank or organization issuing the card and on the specific type of card, I would say (without any proof of it) that the ratio is maybe 80% without IBAN and 20% with IBAN.
And pre-paid cards are also very common, without a connection you have no way to know if the money is there.
Are you sure about that? I'm pretty sure IBAN is universal (or almost so) for all EU bank accounts at this point. Thus consumer debit cards have in IBAN associated with them.
Now, it may not be the fully formatted ISO IBAN number, but some shorter variant you need to prefix with e.g. the country's IBAN prefix. E.g. in The Netherlands all Debit cards have the full number, but Iceland uses a suffixed system, those are just two systems I'm personally familiar with.
I quickly searched for what Italy's largest bank is, and found pictures of their debit cards, which seem to have 4x4 numeric digit on the front. This page then seems to explain how to convert it to IBAN: https://help.finecobank.com/it/il-conto-fineco/conto-corrent...
In any case, I was just using IBAN as a shorthand for "a unique account number suitable for a direct debit contract".
If Italy really uses proprietary bank account numbering system it's probably still true that people there can pay e.g. their electricity via direct debit, which will eventually lead back to a contract where they said such-and-such can withdraw money from such-and-such account.
The store would then print out that contract template to have on hand.
Except that they wouldn't, because you're not going to train your point-of-sale people for the 10 minutes per year the phone system is fully down, or have accounting deal with the mountain of resulting paperwork and manual one-off payment processing.
But that doesn't mean that offline payments are literally impossible if you have neither card nor credit card, it's just become so exceptional that people have forgotten how, even to the point of claiming it can't be done :)
Some Bancomat/Pagobancomat ones (but again not all of them) are connected to the IBAN (amd they sport both the IBAN and the name of the holder).
Many Bancomat/Pagobancomat are connected to Mastercard or Visa (or Maestro) debit cards and they have "their own" number, unrelated to the IBAN (and they don't even have the name of the holder).
Some have the name of the holder, but not the IBAN.
As said it depends on the issuer, I have an account at a bank that issues the Bancomat card with the IBAN number (and my name on it and you have to wait for the card by mail for replacement), have another two accounts in two different banks where you can go to the teller, and he/she fishes a new card from a drawer and "couples" it to your account, the card is "blank", no name, nor IBAN, just the card number.
Fineco is I believe third or fourth bank in size, the two largest ones should be Unicredit (which casually is the one from which I have the IBAN connected card) and Intesa san Paolo (which is one of the two I have the "unconnected" card from).
My current experience is with some 30 payments daily, or 200 per week.
I would say that (from memory and very roughly) what happens here:
American Express or other "real" credit card: 2-10
Other embossed numbers credit or debit cards: 25-30
Bancomat/PagoBancomat (some with IBAN, many without): 90-100
Not embossed number Visa/Mastercard (as said sometimes they are the same as the Bancomat ones): 40-50
This is Europe. Credit card imprinters are rare if they were ever used at all. I've never seen one.
The most resilient back up (especially in the face of national crises) is cash of course, but a growing group of people have stopped carrying any cash at all, and a lot of people in power are doing their best to make sure a cashless future is nearer each day. I don't mind paying with a debit card; I just want to have a backup in place in an increasingly interconnected world where a glitch can take out any nationwide system.
Even for a cash payment the register still needs to work for the payment to be... well, registered as a sale. You need to get a receipt with tax paid, and all that.
That's easily done with a writing pad with carbon paper between each sheet. You write the items, prices and tax, rip off one sheet to give to the customer and another sheet for the books. I checked, these seem to still be sold at office supply places, so there is no excuse.
Cash invoices will probably still work in most jurisdictions, to be recorded electronically after the fact. It's even more work than a handwritten receipt since the customer would have to be identified, not just what is being sold and for how much, so I wouldn't fault a merchant for taking out a service surcharge for it.
I'm in Europe too, and I have never seen an imprinter either, but I keep being issued cards with embossed numbers, so I figure they must be good for something, if not imprinting, what else?
I'm concerned about the phasing out of cash too as a separate issue, I just don't think the cashless options should have to be subject to artificial limitations like requiring electricity or a network connection to be able to be used. I guess online transactions with chip and pin are much more secure and should be the default, but retailers could keep a log of outages that they are required to submit along with the imprinted slips. They could be corroborated with the logs of the utility companies as well as other businesses in the area, and if something doesn't make sense it could trigger an audit.
Some EU countries don't issue credit/debit cards by default but they have their own system. The Netherlands is one of them. Such cards don't have embossing. You can get a MasterCard/Visa compatible card but it's not standard, though I believe this will change soon.
But of course the imprinting system is ancient and extremely prone to fraud. As is the magstripe. I wish we had a choice not to have the latter anymore. I'd gladly give up the ability to pay in places where magstripes are still used in return for more security against skimming. Here in Europe the magstripe is just not used anymore except for supporting foreign cards.
The Dutch debit cards are Maestro (Mastercard) or V Pay (Visa) cards which are a pretty standard systems. There is no point in embossing because transactions need to be authorized immediately by the issuing bank for obvious reasons.
They are pretty non-standard as they don't display a number that can be used where credit card numbers are normally accepted (they do have such a number internally to work with the Maestro/V-Pay system).
My lastest debit card (issued last year) was the first one I got without embossed numbers. I guess retrocompatibility is no longer a priority. Not that it changes anything for me, virtually every shop has already phased out even the magstripes and use either chip cards or NFCs.
> Most credit or debit cards support offline transactions, so a shop can keep a battery powered terminal that reconciles when it comes online.
Almost....
Its been a while since I have been involved in POS payments, but IIRC there is a clear protocol which goes something like this:
1. If the card machine is online and working, use that
2. If the card machine is not working *AND* the business is willing to take on the risk, record an offline transaction for later reconciliation
3. If the card machine is not working *AND* the business *IS NOT* willing to take on the risk, call bank $phone_number for manual transaction validation and record the validation code provided at the end of the call.
There are a lot of different cultural practices in Europe, and unification is a slow process. I for one have seen many imprinters in use in Europe, although it has been years since I last saw one. An online terminal used to be significantly more expensive than an imprinter, so small shops would prefer the imprinter, especially in rural locations with bad connectivity.
My employer has (or had) one. They used it a couple of years ago when they sold some stuff to staff (old furniture etc).
The usual fraud risks don't really apply in this case. Other than that, it must be over 20 years since I saw my parents pay using one, and that was unusual enough that I remember it.
UK too, which is arguably still part of Europe. I remember them being used at less professionally organised events (street parties, church outings etc.).
We had one at work as well 15 years ago as part of a contingency strategy in case of a power outage, but don't remember it ever being used. I left retail so no idea if they still have it.
Aside: in case that's an eggcorn and not a typo, the day would be "fateful", as in, something momentous happened, rather than "faithful", showing loyalty.
What jumps out at me after reading this notice is how it completely lacks any apology. I am not prescribing what it should say but merely observing that having lived in North America for sometime now, it seems hard for me to imagine for such a notice to not sound profusely apologetic here.
Note: I do not consider words like 'unfortunately' and 'extremely unpleasant' apologetic.
Also, I understand there are cultural differences between the Netherlands and the USA- no joke.
This is very consistent with customer service in The Netherlands - nothing is ever the fault of service providers, and you manage to convince them otherwise - you'll get nothing more than "unfortunately (...) not possible"
I don't mind the tone, I got tired of US style politically correctness just for the sake of saying it.
How would "We are trully sorry for this incident" make it any better ?
I do mind the fact that they don't pay the people their money back if they try to find alternatives to get to their destination. That's 100% unacceptable.
When is being apologetic became politically correctness? Are we soon going to call every decent human like behavior in public space as being politically correct?
The cynicism runs deep. Did you know that cynicism is linked to depression?
"The study was conducted on a group of 3,399 individuals, all from London and aged between 35 and 55. The personality traits that were observed included cynicism, suspicion, resentment and distrust. After the study period was completed, it was found that the majority of individuals who expressed these particular personality traits that were monitored at the start of the study had developed depression during the 19 year-period."
Its london - if you are not suspicious and distrusting your phone will be removed from your possession as a you walk down the street. Resentment - from the council estates to mayfair.
Cynical, I would say I am 100%, however, as per your article Being cynical essentially means you are unable to trust the intensions of those around you. I trust their intentions, I dont trust what they say their intentions are.
The intention of the politician (or child) who gets caught lying is not to appologise and make amends, its to get out of trouble. The intention of the diner who says this is delicious before tasting a dish is to ingratiate themselves to the host or appear the "polite one" at the dinner party - otherwise they would find out how the food actually tasted before claiming it is the finest they have ever tasted.
As for the 5* uber rating - The bottom line is if you give your driver less than 5 stars, you have screwed him over. So, unless they fuck up, give 5 stars.
Take care not to slide so far to the other side where there is another trait - toxic positivity ... With toxic positivity, negative emotions are seen as inherently bad. Instead, positivity and happiness are compulsively pushed, and authentic human emotional experiences are denied, minimized, or invalidated.
> The bottom line is if you give your driver less than 5 stars, you have screwed him over
The bottom line is that the system is messed up, isn’t it? Same with performance reviews at big co. Why have reviews at all if it’s only useful as a weapon.
Why would they be sorry? They’re the ones who are missing millions? of Euros in revenue due to this issue while the passengers just had to take a bus or bike somewhere. They probably would tell you that if you had to bike, it’s healthier that way. A little bit of rain won’t make you melt.
"Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today." I think the writer is more pointing the finger at IT than feeling apologetic. Imagine getting called an enormous failure by your adjacent department, that writer wants you to know who screwed up instead of apologize for their screw up. Seems more useful than an apology in my opinion.
I think it’s worth mentioning that while the IT system couldn’t coordinate the trains safely, it was a management decision to stop them from running. Coordination can happen myriad ways. For example, trains could operate according to their normal schedule, and if one needed to deviate they could radio in, and an operator could let downstream trains know to proceed accordingly. Maybe simple cell-phone GPS pings could suffice. Perhaps the state of the system was in such chaos that such orchestration wasn’t possible with the staff on hand. That said, a truly resilient system should have multiple coordination channels to fall back upon.
The drivers and conductors would like to end up at home at the end of the day. Even if you just run the normal schedule, there will always be disruptions, and there needs to be a system to adjust the planning so people and things end up at the right place at the end of the day. That was the system that failed. Have you any idea how frequent the trains in the Netherlands are and how complex the schedule? "Simply" running the existing schedule is an absolute impossibility.
About the only thing they could maybe have done was to fall back on back-and-forth shuttle trains between major node stations. But that would only provide a very basic service, with incorrect train materiel being used for the normally expected number of passengers. It would be just enough to get passengers and personnel home. I would like to see something like a pen-and-paper schedule that they can fall back on, but they don't have it. (They did something like that at the first corona lockdowns, although with more planning involved.)
I was in the station for a different reason when I heard the announcement. The people entering and hearing the announcement were unsurprisingly very crestfallen, but it appears that twitter helped organize a lot of travel, with people posting where they were/going and picking up other travellers.
Also probably worth noting is that trains here are used fairly frequently, however unfortunately this isn't the first time trains are stopping - for example snow is a common reason for delayed/reduced service. (Snow isn't very common here in the NLs)
You could even install rails that go along regular routes, and put on scheduled carridges that people could all jump into at designated places along the route.
The computer could generate a schedule and post it publicly so people knew when they could hop on.
In 2019 there was a power blackout in London. When power was restored, a number of new trains refused to start without a technician physically coming over with a diagnostics laptop to them because power fluctuated outside allowed bounds or something before the blackout.
It's likely the same power fault could occur in a single train in other circumstances. When an electric train cannot access the grid reliably it would become far more dangerous - no more regenerating brakes, no reliable acceleration ...
Better to force the operators to fix the boom arm / whatever other cause of power glitches with fault codes like this. Even if they false-trip when there are grid level outages.
No, it was considered a design fault with these particular trains, and fixed soon afterwards.
Being unable to accelerate or unable to use regenerative braking is not in any way a safety problem. Either situation can happen at any time. (Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply, and this was the normal method for decades before regenerative braking became common.)
> Electric trains have banks of resistors used to dump the unwanted kinetic energy when it's not possible to return it to the power supply
On AC-electrified railways (especially those with their own independent power supply network like Germany/Austria/Switzerland), resistors are actually rather uncommon on stock capable of regenerative braking, because the OHLE not being able to accept regenerated power is a relatively rare occurrence.
Though of course in that case the conventional friction brakes are of course fully rated to bring the train to halt, only with a bit more wear and tear.
DC-electrified railways are a different matter – transmission losses are higher, power supply sections are smaller and substations often not capable of returning any surplus power back into higher network layers of the power supply systems, so the OHLE/third rail not being able to accept the power generated by regenerative braking happens much more frequently and so trains continue carrying braking resistors as a fallback system.
More and more we are seeing "the computer says no" in our lives. We've replaced human activity and decision making with computerized versions, which are nothing more than the automation of stored human knowledge and routines.
For instance, running high speed rail without computer signalling would be a huge risk there's no sense in taking ( due to the speeds involved, by the time you see anything of issue like a train or damage in the tracks in front, it's too late to stop).
I think that's a bit of a leap from what I was saying. I didn't imply that all software and computation is bad. Obviously, a human cannot live handle any control system that needs even remotely fast operation. The point is that due to scaling issues, we're removing humans from systems where humans excel or could serve as backups in the event of failure (and actually vice versa).
Scale creates problems that are partially solved via software systems, which only handle a subset of the existing variety and new variety introduced by scaling. This makes the systems leaky with regards to unhandled variety and incredibly rigid (i.e., not robust) against exceptions, exceptions that could often be readily be handled by humans. By software being a snapshot of some small group of humans' knowledge of a system, it is completely unable to respond to live operational exceptions. Stafford Beer's work is interesting background on this type of thing.
> A much more efficient automation.
There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?".
> There are many, many examples where that is both not true and true. For the not true cases, just see literally any customer service interaction with a large company in the past half decade. I suppose this could be arguable though if one asks the question "efficient for who?".
Another example I love, that is even less ambiguous, is restaurants (traditional, not fast food) that require waiters to take orders on touchscreen PoS systems, instead of taking the orders on a notepad and inputting them later. The touch screens are incredibly slower on a bigger party ordering appetizers, a main course, drinks and desert - it takes forever to navigate back and forth and find the right item, and usually has to be done again and again for each diner. By comparison, a notepad allows the waiter to write down exactly what they hear, often just making small extra notes to indicate duplicate items.
Of course, this particular one could be improved by arranging the menus more naturally for someone taking orders, and several other conveniences (such as always keeping the current order on screen, to easily tap to add "oh, and another beer for me", "I'll have the same desert as her" etc).
A couple years ago, politicians in Poland wanted to do a quick change to the country's universal pension system. The change had to be quick to score some kind of political win. They wrote the new law and went on with quickly pushing it through parliament. To their surprise, sometime along the legislative process , the contractor responsible for maintenance and changes in the pension system informed them that they can indeed implement the change - by June next year. The politicians learned that, in the mordern computerized world, just passing a new law does not change how the state bureucracy operates.
Yes possibly, but the whole point was for people to get more money into their pockets ASAP (as opposed of just a promise of more money in the future, which has lesser impact), so that they feel gratitude come elections day.
As a developer, any day in which you don't see a national headline about how your f*#(up has just made a lot of other people sad and/or angry, is a day that could have been worse.
Sounds similar to a problem that happened a couple of weeks ago in Madrid, although that was not the whole network it pretty much shut down the trains for the day. That was an IT failure too.
https://www.20minutos.es/noticia/4973632/0/graves-retrasos-e... (in Spanish)
Wow, talk about a single point of failure when an issue in the systems of one station can stop all but one train line. Interestingly, it seems to be a station where tracks from six directions meet so a backup mode of trains simply passing through might be impossible: https://en.wikipedia.org/wiki/Madrid_Chamart%C3%ADn_railway_...
Maybe we have different definition of fewer trains, because I wouldn't expect more than 10% at most, realistically 2-3%. And those trains are probably already 2-3%.
Many international trains were actually not running. I haven't read up on the details yet as I was away this weekend, but I think it was only short cross-border routes that still worked, but not much more. The night trains only started up past midnight. And anyway, even if more than that had run, night trains and international trains only run on a small number of Dutch train routes. The three daytime long-distance international train routes are: Amsterdam-Rotterdam-Antwerpen and onwards, Amsterdam-Arnhem-Duisburg and onwards, Amsterdam-Hengelo-Osnabrück and onwards (several intermediate stations omitted). The only domestic night service is Utrecht-Amsterdam-The Hague-Rotterdam. That's it. You can't fit any serious amount of domestic travel into those more than they already normally carry.
There is a swiss company based in Zürich that develops this kind of software. I believe it was a c#, wpf application, the last time I spoke with them. The schedules that their software produces is calculated and set for several years ahead. That baffled me! The tracks and infrastructure is booked up so far ahead! I think they mentioned that the german railway uses their software, nut only the sbb.
Based in my knowledge as a public transportation enthusiastist in China, I got really surprised to see a IT problem bring the entire system down. There should usually be some "more manual" backup mechanism that allows downgrade and operation of trains without computerized signaling system. By kinds of token signaling it will still be possible to run trains as long as the physical infrastructure is still there. The efficiency will be much lower and more prone to (potentialy deadly) errors, but no more trains today sounds unlikely.
The fact that it happened on a Sunday suggests to me that this is the result of a service outage gone awry. Realize that the trains failing on a Sunday is really not a big deal. However, if the trains don't run for Monday morning rush hour that is a big deal. I bet they made a change Saturday night and something broke Sunday morning.
It's now Monday morning here and the trains are back to normal in time for rush hour. So it's not really a big deal.
I have to say that I do find this framing which says "well, it didn't affect business, so no big deal" to be somewhat distasteful. It's completely dehumanising. Why run the trains at all at the weekend, if a weekend outage is no big deal?
That's not the framing. For the majority of travelers, missing a weekend trip is an annoyance. In contrast, missing their work is a major headache, for them personally. It also cascades pretty badly if it happens on a lathe scale - meetings get pushed and overlap others, suplliers are not able to deliver, stalling production, financial operations get delayed, potentially affecting deadlines - it's a huge mess in everyone's life that won't just go away when trains come back. It's also as true for non-profits, state organizations and schools as it is for businesses.
I do hope they have a more detailed RCA at some point, right now this is the only helpful paragraph in the article about what happened:
> The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.
I was expecting more as well. Not much information to glean from a barebones press release.
If I had to guess, it probably was an issue with scheduling/timetable software rather than anything to do with the trains, rails, etc. Nothing exceedingly seriously or difficult to correct.
Keep in mind that train schedules control what trains are on what tracks when, so while their are literally fail-safes in the system, this is not exactly far removed from being "safety critical".
Isn’t this exactly what the press release states? “ if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure.”
No, it’s talking about crew scheduling/allocation , not signaling.
The signaling system is totally fail safe. Even if that system totally went out, drivers are required to know where all the signals are, and to treat an unlit signal as a red.
This post is a fascinating use of grammatic passive voice. The hallmark is gratuitous use of "being" verbs, i.e. "is,was". Passive voice is a way to avoid using active voice, which is an implicit way to avoid responsibility for an error.
"Due to the enormous impact of the failure in the IT system, it is unfortunately not possible to run any trains today."
// =>
"Due to the enormous impact of the failure in the IT system, NS cannot run trains today."
"Although the cause of the failure has now been resolved, the impact is considerable. To be able to start up reliably, systems must be updated and trains must be brought to the right place. That takes time. For our passengers, this is extremely unpleasant news."
// =>
"Although we have fixed the cause of the failure, the failure impacted most of our customers. To fix this failure so that trains would start reliably, we needed to return all trains to a central depot, which took nearly 12 hours. We apologize for the extremely unpleasant interruption."
"The expectation is that tomorrow morning the normal timetable can largely be resumed. The night trains can still run."
// =>
We expect to resume minor operation with night trains first and then major operations tomorrow morning.
"The IT failure occurred at the end of the morning. It affected the system that generates up-to-date schedules for trains and staff. This system is important for safe and scheduled operations: if there is an incident somewhere, the system adjusts itself accordingly. This was not possible due to the failure."
// =>
The IT failure occured at 11am in the scheduling system, a major requirement for safe operation; if train operations are interrupted at one location, the system normally reroutes other trains to prevent collisions. This system failed.
"The international trains are not affected by this failure. For information about the timetables of other transporters, passengers can consult the websites of these transporters."
// =>
The failure did not affect international trains. If you have an inquiry about other transportation services, please consult those services.
"The journey planner is updated."
// =>
We updated the journey planner accordingly to account for the interruptions.
// Generally the use of "passive voice" indicates poorly-taught grammar, however the use could also indicate desire to avoid responsibility.
"Who are we? NS is active in the public transportation sector. We encourage the use of public transportation and keep the Netherlands moving. Our travellers are our 1st, 2nd and 3rd priority in all of our activities, and we do our utmost to make their trip as pleasant and sustainable as possible from door to door."
This is also fascinating. I would expect to find "NS (Netherlands Service) provides rail services for the entire country". Instead they choose vague grammar, more the idea of NS rather than what NS actually is.
"I am a programmer" => "I encourage the use of computers to provide solutions for common business problems". Just a strange use of grammar.
I was walking up to Amsterdam Centraal when I noticed floods of people all exiting the building... within minutes an Uber was booked to Schiphol before prices rocketed due to demand.
For those outside of Europe, could you explain how this works? Are these Hungarian citizens living abroad who are able to somehow vote from polling places accessible by train within the Netherlands? Is this an EU organized thing?
Only in Amsterdam inside Netherlands after waiting a long line. At the same time Hungarians in Romania could vote by mail and the government was using Fidesz to collect the votes and threw out votes of the opposition.
The soft dictatorship works very well, Hungary will go even more down in the future. I'll probably leave Hungary before Hungary leaves EU/NATO and gets into a war.
Who exactly would Hungary get into a war with? It borders Slovakia (EU member), Austria (EU member), Croatia (EU member), Serbia, Romania (EU member), Ukraine, and Poland (EU member). Is it your contention that it's going to invade the EU? Or Serbia? Or Ukraine? Or is Hungary going to use its awesome force projection capabilities to get into a war with a country that it doesn't border?
It's not simply voting by mail, it's voting by mail by not even using official the post office in Romania.
I know that you're probably refering to US, but I don't make any comment about it, because I don't live there, so I don't know what really happened in US.
In Romania some people found partially burned ballots with only opposition votes next to a garbage bin before the elections ended. The proofs of voter fraud couldn't have been clearer. It's supposed to be a scandal, but nothing will happen about it.
Well that is just an isolated case. There is simply no evidence to support the baseless conspiracy theory that postal voting results in widespread fraud. This has been well established by the experts and journalists.
Clearly the quality of the audit matters and that differs massively between countries. Nobody disputes the validity and integrity of postal voting in Australia for example.
Sometimes even local stuff. U.S. states, our provincial government level, have processes in place to account for Americans living abroad to vote in state and local elections. It’s guaranteed by law, I think. Our states handle all voting on behalf of the federal government anyway (registration, voting day operations, counts, all often under the misleadingly named office of a Secretary of State), so it’s a natural consequence of our structure. It is nice to live abroad as if you were still at your last physical address for voting purposes, though, and usually not notice a big difference. I think it’s implied that those of us who have done it intend to come back, in the system’s thinking, but I think everyone understands the relevance of their vote according to their situation and beliefs.
We’re of course all too familiar with this process from several consequential court cases deliberating its efficacy and merit in generational memory. Whether it’s sensible has been argued with elections on the line.
A shocking thing for me is that if you're an American living abroad registered to vote in California you're allowed to vote by email (well technically fax, but there's a DoD email alias you can send your scanned ballot PDF to and someone will print it out and fax it, which counts). Oh, and if you vote any ballot this way other than a write-in for federal offices you might also get a residency audit and have to pay California's personal income tax (which doesn't recognize any US tax treaties). You're not allowed to vote by fax or email if you don't have a non-US address.
Were the elections illegitimate ? I'm a complete outsider to hungarian politics so I'm not sure if this is hyperbole... but talking about how a dictatorship has been reelected again doesn't make sense to me. Unless elections in Hungary are just theather, and results are just faked like in Syria for example.
Outright dictatorships like North Korea are getting rarer. It's more common nowadays for de-facto dictatorships to maintain an appearance of democracy by actually holding elections but either having no viable candidates other than the incumbent (e.g. Russia), or by systematically banning or even murdering all opposition (e.g. Belarus). Hungary isn't quite that far gone yet. It still has elections that technically meet the requirements of EU membership. I don't know any more details though.
It's a relatively free election, there's a strong incumbent bias that undoubtedly helps him win but there's no evidence the entire thing is rigged. Dictators can be popular, the best example is Napoleon.
Each opposition party gets 5 minutes per 4 year election cycle by the state controlled media (and a lot of other media was bought from stolen EU money).
I dare you to watch 5 minutes of news on the government media channel (the only channels that can be accessed in the free TV tear, that poor people are able to watch) and say that it's not rigged.
If Napoleon, who ruled hundreds of years ago is the best example you know, there must be something in not allowing any person to stay for 16 years in power (especially because he just had 9 years)
> I dare you to watch 5 minutes of news on the government media channel (the only channels that can be accessed in the free TV tear, that poor people are able to watch) and say that it's not rigged.
All your saying can be described as a "strong incumbent bias that undoubtedly helps him win." International poll watchers agree, the elections themselves are fair but the blurred separation of party and state leads to an undue advantage. 2014, but I don't see evidence of the situation having changed. http://www.osce.org/odihr/elections/hungary/121098?download=...
Maybe I should have said "fully rigged." Real opposition candidates do get to run, they just have almost no shot of victory. And it's possible that if they did have a shot, the elections would be outright rigged.
Napoleon was my example of a popular dictator, I didn't want to use other examples as people will debate whether they're dictators. I never said that's a good thing, Napoleon was horrible despite remaining popular in France.
The government made extra payments to their typical supporters like pensioners and rural areas. Economically strong regions and younger professionals pay the bill and are free to vote against, but they remain a minority.
Yes, but the question was about Hungary, not the UK.
Though since there's no election for a while and they don't apparently get elected based on promises delivered, just promised made, they haven't shied away from breaking the pensioner-bait promises anyway for this year and removing the triple lock (pensions rise at least as fast as the largest of inflation, earnings or 2.5%).
Defense in depth? With this safety feature inoperable, the margin of safety is reduced.
Other systems that might normally provide safety critical redundancy could be providing the sole measure of safety, with no other redundancy available in case one of those fails.
What's the point of having redundancy if you can't use it to avoid an impact to service?
Edit: and to be clear, we're not talking about holding down a resetable circuit breaker to avoid being late, we're talking about the rail network of an entire nation being inoperable for a day.
The point is to still have it when you didn’t know you needed it. That’s the purpose of all redundant safety systems. The redundancy isn’t there to keep the service operating; it’s to keep people from dying.
You can never know if the primary safety system is functioning perfectly, so you need other systems to be there to step in when the primary fails unexpectedly.
If you detect the primary system has failed, isn’t it reasonable that you should stop operation as quickly and safely as possible, and be thankful nothing bad happened while you lacked redundancy? Any SPoF could be fatal for hundreds of people.
I'm not going to write the wall of text that's required to properly describe the situation but that's just not how these systems work. Industrial automation has all sorts of feedback loops so they people operating it know if they can trust it. The people using it are pretty much always trained in how to run the various systems manually so that an errant sensor doesn't turn into a clogged up rail line or waste whatever is in your process equipment that happens to be mid-cycle.
This isn't some consumer appliance where you have to stupid proof every inch of it. These systems are bespoke and their architecture is mostly a matter of business decisions and not at all a matter of the internet peanut gallery trying to figure out how safe they can make it.
If you want to claim public transit operators are happy to take on liability for potential deaths by continuing to operate when they know about reduced (or no) safety margin, that’s an interesting idea.
Calling me a ”clipboard warrior” isn’t furthering your argument, though.
A passenger aircraft can still fly with a failed engine, but it must land at the nearest suitable airport. Because at that point, another failure could not be accomodated. This sort of redundancy is about preventing disaster, not about avoiding service disruptions or keeping to a schedule.
It sounds like it was a crew scheduling issue. So if you implement some manual replacement for the automated system, it's possible that train crews will "time out" before they reach the destination, stranding passengers in the middle of nowhere. Or, will make a tired crew operate the trains, and tired people aren't very good at safety sensitive tasks.
There was an incident a few years ago where a LIRR track crew member stepped in front of a train going 78mph and died.
The employee that died had been on duty for 38 of the 50 hours leading up to the fatality. One of the NTSB's recommendations is to implement software that avoids worker fatigue:
"The FRA encourages the use of certified biomathematical models, such as the Fatigue
Audit InterDyne Model and the Fatigue Avoidance Scheduling Tool (FAST) by railroads to help
them develop work schedules for safety-sensitive employees that align with healthy work-rest
scheduling practices; however, these safety measures do not apply to roadway workers.
14 The work
schedules developed through biomathematical models avoid many pitfalls causing worker fatigue
that arise from excessively long work hours, highly variable work shift times that disrupt human
circadian rhythms, and infringement on sleep opportunity times."
TL;DR: it's not about the trains crashing into each other. It's about sleepy people making bad decisions that gets them or their colleagues killed.
I'm not from the NL, but I have travelled there by train. My feeling is that they have very dense traffic, but in a relatively small country most trains don't travel more than 2 hours. That means each driver and each conductor needs to make 4 different trips a day. And because they need breaks they proably switch trains during a working day. If all this planning has been computerized for many years it just means that nobody knows what to do next if the computer screen remains blank. Probably they don't get a printout in the morning like it was some decades ago. No idea how adaptively the system would react during the day dependening on delays and disruptions.
20 years ago, the NS - the Dutch National Railroad company - tried to remove complication from the railroad schedule and got a lot of criticism from their employees because it would mean they would travel less far from their home stations. Train drivers said “so we will only be driving around our church”. It led to a lot of protests and eventually a lot of the plans were revoked.
Might not be your point, but I wouldn’t be surprised if having people stuck in wagons in the middle of a section for half an hour because of conflicting schedules would be seen as safety risks for the travellers.
Isn’t way too late if the conflict is detected at signal level ?
The worst will have been avoided, but at least one of the train will have to back down to the nearest junction, and depending on how packed it is they might have to back down a few more train as well. I’d assume in reality it would deadlock from there.
I suppose it relies on some slack in the system and everything else moving as expected. Which could also be the reason why they don't want to restart a schedule at mid-day with every active train in a non standard state.
It highly varies depending on POV, but for the railways the worse could be people piling up in stations with no idea if they'll ever get a train, and people piled up in trains, with no idea when they'll ever be able to get off.
Sure stopping all trains must be burning a crazy amount of money, but the above scenario would be burning the same amount of money (I assume they have refund policies or they'd be operating with no ticketing), employee sanity, and urgent care for anything that could happen during that time.
I have faith in Dutch people's kindness, the same happening in Paris would be riots and physical danger for the staff if they kept the trains running.
It still sometimes is. There was a big crash in Czech republic recently, where two passenger trains (motor units) collided head on between stations.
Some local routes with few trains a day, where there is no longer any track-side staff, operate in a simplified rules (there is a whole code for that). There, the train will arrive to a station and depending on a symbol in the timetable, it will either continue to the next station without any signals or permissions needed or will wait for crossing train in oposite direction (the train driver must also unlock the station office and call the dispatcher on phone in some cases). Switches are either operated by train crew or they are self-returning (one direction always go to track 1 in station, the other direction goes to track 2, and the switches can be travelled through in the oposite direction without setting them first (this is an accident on standard switches) and they return back.
So, the problem was that the crossing was there on some days and wasn't on some other days, driver forgot or misread the day and the train left into oncoming train...
Was a big discussion after that to do anything, even an uncertified mobile app to shout "train coming" at the driver... Otherwise Czech rep. has a very modern European rail infra, just some local branch routes run like this saving them from being cancelled completely, but the rules were simified maybe too much.
There is also the token-based control of bidirectional travel on single-track, in which physical posession of a token governs permission to enter a track block. Still used in various places.
Lets say there is a north-south train line that is shared.
The schedule can create a number of north slots and south slots into which a train of a given speed and length can fit. A variety of different trains could fit into the same slot.
If a train is late, that train is moved into a later slot. That leaves a free slot where the train was going to go. Perhaps every later train is then delayed into a later slot also. Lets call this a rippling delay.
You would probably schedule some spare slots to prevent delays rippling through the entire day's schedule. Some slots might be long enough to permit several trains going in the same direction to travel (it's more efficient if 10 north trains go then 10 south trains rather than 10 north/south interleaved).
There was a massive southwest airlines IT failure compounded by weather issues over the weekend. This cancelled numerous flights but also snowballed into most of the southwest flights being delayed. We were lucky to return home. The news is only reporting weather delays yet for those at the airport this weekend...it was clearly larger.
If it is true that Netherland's and Italy's transport infrastructure were also down I wonder if Russian supported hackers are to blame.
See also : assassinations - maybe the death of the top polish government & military and Total's CEO in plane crashes were just freak accidents, but a nagging suspicion that Russia was involved will always remain...
He said that when he had recently graduated and just moved from Norway to Sweden his first job here was as a technician for the local railway. One of his first assignments was a call where the computer controlling all the track switches had stopped working. Luckily there was a backup, but they needed him to fix it immediately.
He arrived, took a look at the computer and its backup running next to it. He started out by measuring voltages on both, comparing to see what the difference was. After a while some men in suits came in and asked how things were going. He said it was all good, they said it wasn't because now all trains had stopped. Apparently, he had short circuited the backup.
"And that's when I decided to go for a theoretical career!" My teacher happily concluded. The classroom was left in a stunned silence.