> It affected the system that generates up-to-date schedules for trains and staff.
...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole.
I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system.
> A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days.
Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables.
> In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently.
Cool stuff.
Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
Those are pretty standard terms here in .NL. 36 hours is considered fulltime by most employers, including government and semi-government. Pension is offered almost everywhere and at least 4 weeks vacation is the legal minimum.
Maybe except for the _unlimited_ train travel but most employers do offer free train travel between home and work.
This also means that if an IT system goes down on a Sunday, a lot of employees won’t even pick up the phone until Monday 9 AM.
Indeed pretty standard, even in non-government it would be 40 hours but all other terms the same and most likely a company car + private use of it instead of the train.
And I'm also sure they have a significant on-call setup of the IT teams for any out of office hours issues, just like every other large 24x7 kind of organization.
Note that a pension is essentially a legal requirement. There might be ways around this, but it would be really hard to offer employment without offering a pension in NL.
Only AOW (state pension) is a requirement by law and for full-time salaried employees there is no way around that. Private pension (what is meant here) is not required by law though financially very lucrative compared to doing it yourself. It can be made required by making a CAO (collective bargaining agreement) "algemeen bindend" for all employees in a sector/company but this is not always the case in all sectors (e.g. IT). Note that this is way more complex and this is just a simplification.
Really, I thought pensions were required. I guess it's just so fiscally attractive that essentially every employer offers it. Combined with the pervasiveness of CAOs, especially among the more stingy sectors.
I worked about a year at NS as a consultant. I wrote python software that broadcasts the audio messages in the passenger compartment, steers the outside lcd screens and such.
It is a great place to work. A nice atmosphere, good coffee and they embraced change (in this case Scaled Agile) better than most places I have seen. They are often listed as top-5 best places to work and I can see why.
>36 hours is considered fulltime by most employers
How standard is that in the private sector or non former state owned enterprises? I haven't looked at the NL market in a while but most tech jobs I saw back then were 40h/week.
It's hard to find actual numbers. Tech jobs, smaller companies ("MKB") and work in the agricultural sector tend to have a higher percentage of 40 hour work weeks than other sectors.
I have been working between 30 and 24 hours as a Senior Software Developer in the past 20 years. I have had several colleagues who work a similar amount of hours. I have been telling employers that I am only productive for about six hours per workng day on average and that never has been a deal breaker. The number of productive hours does vary from day to day, but I know that I push myself beyond that limit, I will run in trouble after some months. I have heard about studies that the average employee is only really productive for about two or three hours per day. With me working six hours per day, I might just as productive as someone being eight hours in the office.
Most private sector tech jobs in NL are 40 hours per week. But many private sector jobs are also flexible in ways that I've never seen in the USA. For example, allowing an employee to work 40 hours in 4 days instead of 5.
> everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
I have a reflex repulsion for off-the-shelf "enterprise" software. You pay millions of dollars to subscribe to a suite of (seemingly) bad, bloated software that takes more work to customize and integrate than it would to fully replace. An army of developers trying to keep a business running with the world's largest Swiss Army knife.
My gut tells me that big names like SAP and Sales Force are easier to sell to executives, and it makes a company seem like it knows what it's doing. And maybe feeling pressure to choose a "respected" brand leads to fewer alternatives? Or maybe all the alternatives are quickly bought out by the behemoths. I really don't know. Maybe someone can enlighten me. It just seems like an enormous turd of inefficiency to me.
Have done consulting for 15 years, understand your feelings. But I'd like to offer some alternative viewpoints.
Buying an off the shelve package also buys you an industry-standard business process. The idea is that part of value comes from the process that you have to shoe-horn into. Some integration is expected (and provided for) but if you have to modify heavily, You're Not Using It Correctly.
Also, using an OTS package means less development risk (not every IT dept has 10x developers coming out the wazoo) and it also means that you have a huge reservoir of skilled labour if you need it. Not just for development but also on the business side.
Finally, if you're a big, publicly owned company, you likely have many financial / reporting constraints. The scope of your non-negotiable requirements may be bigger than you imagine.
I've seen people implement SAP in a couple of months. It's possible. It's still expensive and clunky though :-)
> I've seen people implement SAP in a couple of months. It's possible. It's still expensive and clunky though :-)
Never worked with SAP, but still coming off my burn-out hangover from $OLD_JOB - worked to migrate the nexus of the firm's accounting systems from COBOL to MS Dynamics. Unto itself, I don't really have a problem with Dynamics - I do have a problem with MS and their piss-poor available materials on the topic. While I understand their business sells training and certifications, this does ass all for folks pulled in as a pinch hitter during a death march. Can scarcely fly out to Las Vegas for a week long vacation while I'm working 16 hour days to beat the clock on our last true COBOL coder's retirement date.
The greater issue I have in enterprise systems - often the consultant / contractor firms[0] involved in the process. The classic fly-out of classy sales-engineers replaced with low-rent off-shore coders for the actual work. Slick-as-shit documentation that does not match reality in even the simplest of cases. Counter-Party contacts that neither care nor hide their lack of caring for the outcome of the project. Pressing for aggressive timelines that do not meet reality
I would assume there is some correlation, however, between buying $ENTERPRISE_TOOL and an unwillingness to hold consultants feet to the fire on their own failings, structuring a contract that keeps them honest.
[0] - Not to state `ArnoVW is one, only stating a pattern I've run into all too often.
Thanks for the footnote =) And I concur. Outsourcing your IT replaces one type of work with another. If you can't manage your externalized projects, you're gonna get eaten alive by IT services suppliers.
In fact my line of work was managing 'complicated IT projects', generally involving formalizing things and chasing said suppliers (ie they outsourced the outsourcing)
You see, when your Foxpro guy retires in 10 years, either you've been training your new hires how to program Foxpro (a waste of time) or you'll pay a specialist to move all your business data and operation macros to a new platform (a waste of money).
SAP lets you stop kicking that can down the road by wasting your time and money today.
I’ve never seen a SAP install that wasn’t modified heavily, usually to the tune of millions of dollars of development - which can’t help but make me think that if nobody can use it “correctly”, then perhaps it can’t be used in its unmodified form.
In addition, a common occurrence seems to be businesses paying for extensive custom developments (such as an integration with a third party system), and footing the entire development cost - despite their “custom” development being 100% identical to that deployed at other businesses. They’re then trapped in a world where changing the URI for an API endpoint costs €150k.
Honestly, I see it as just being a big corrupt hole fuelled by interpersonal relationships rather than any actual rationale.
My previous boss recently tried to pick up an enterprise solution. He runs a non-software company that has a lot of custom internal apps - 90% built/maintained by interns. Of course the intern-built custom apps are super buggy but the bugs get fixed immediately, and the system is tailored to the company's way of doing things. The company spent an obscene amount of money just demoing a full-fledged SAP-type thing to potentially replace the custom stuff, but they ultimately found the rag-tag web apps better and cancelled the SAP contract.
That whole situation convinced me pretty thoroughly that "enterprise" software is indeed a form of large-scale snake oil. I think most businesses can go a long way with just spreadsheets and email.
Nothing gets me as excited as a lowly footsoldier in a company as when the c-suite decide to spend $100mil doing a custom install of a big name software stack that will "totally change the way we work and at the same time be fully customised to the way we work here".
This is one of the reasons I'm bullish on no-code doc-apps like Coda [0]. I recently started consulting with someone who calls himself "The Coda Guy" and pulls together truly wizard-like apps out of Coda [1]. (Coda is effectively a hybrid of Excel, Google Docs, and Notion.)
I think the "Excel+" category should receive a lot more attention than it does now. I can easily see "no-code" tools like Coda taking over Salesforce and effectively filling the niche you describe.
Agreed, except for the spreadsheets. Please get people to use server based databases instead. They are no longer difficult or expensive to install or use and make life so much better even for 'interns' and it will reduce their workload dramatically as well as increasing the quality of the solution..
Basically SAP is a large database/software package that runs all aspects of your business. From accounting to billing. Your business probably needs to change a lot if you’re going to use it. When I did a time card application for a big company we interfaced our app with a backend some sap engineers made available to us.
I have no idea how it works, but apparently it does.. sometimes…
i think this is under-emphasized for people unfamiliar with them.
I think SAP and related modules are "solidified business processes", and their consultants and sales persuade your CxO level people that switching business processes will improve overall efficiency (and somewhat hint at standardization, so that the CxO will have experience at that domain when switching jobs!).
But of course, it's not really true - best practice business processes are an illusion imho, and it will turn out that the subtle differences have to be implemented individually.
To be fair, that is the case for most ERPs. It's not really about the size of the product but rather the important role it plays in a company.
Most ERPs can be quite flexible too, there's a ton of customisation possible. And a whole cottage industry of addons and integrations for every imaginable integration or business model. But the problem is you really screw yourself with added complexity when it's time to update. Then you're really condemning yourself to spending millions on consultants at a time when you might not have the resources. And postponement leads to technical debt.
Hence the idea of adapting the business to the system. Making sense too if you think about it this way: you're not really just buying a system but a methodology.
And another thing is that it makes integration with other companies easier. Makes the company more easy to sell or too acquire other companies and integrate them.
They decided to modify a number of core functions to work the same way as their old terminal system instead of the default ERP process. Now our job for the next couple years is decoupling the custom add-ons so we can safely update to a modern ERP system.
This was true even for Royal Dutch Shell, one of the largest companies in the world. A few years before he retired, my dad led a team responsible for coordinating implementation of SAP and for training thousands or even tens of thousands of employees to work with the new system.
I read recently that not using SAP - or heavily customizing it - in markets where it’s prevalent can break big(ish) companies. Apparently the interface to other companies (from which to buy or sell) is so much easier to setup when all have the same system, that using any other system is prohibitive. If you need to interface with hundreds of enterprises that all implement standard SAP, you become the expensive outsider if you don’t. Integration projects can take years and then fail, with nothing to show for - which can threaten a company’s existence. Better to use standard SAP „off the shelf“ and do as little customization as possible.
Without necessarily wishing to call into question your source on this particular occasion, it is of course in SAP's interest that as many as possible believe this to be true.
One of my customers was taken over during their SAP implementation. The general feeling among management was the work had weakened them so much it made them vulnerable to takeover.
SAP is mainly an ERP system, but it has various modules dealing with stuff from Purchase Orders and Materials Management to HR. It can be very large and extremely expensive, like over $1 billion for one company in ~ 20 years. It can be customized in many ways, good consultants are very expensive and the cheap ones are usually a waste of money. Without customization very few companies can map their processes on out of the box SAP.
It was developed a very long time ago, it has a huge adoption in large companies and a few years ago when they migrated from R3 to Hana one of their own people told us in an architecture review meeting that they have over 300,000 built in reports to migrate.
In many cases companies are adopting SAP because "nobody was ever fired for buying IBM". Anecdotical, the company where my brother works is implementing SAP and changed all existing networking gear with Cisco because "this is what big companies do". They threw away brand new equipment and spent several millions on the SAP implementation that is still not working, forecasting they will pay even more millions to make it work then millions per year to use it.
Think Oracle, with business logic on top. (JDEdwards, Oracle, SAP) were the OGs of BI.
(and bloat, and McKinsey infiltration into many a large F100+)
--
They are old school, entrenched, but yu would jump off a cliff supporting any of this legacy stuff. It will persist as the inertia on huge systems is heavy.
But yeah - I wonder if they are signing any NEW business... aside from military bunker grade stuff (which will cost as much as the bunker to implement) I dont see a reason to implement (not because I am bashing, but because I dont know the modern business case reqs that would call for SAP)
I don't know about “McKinsey infiltration into many a large F100+” first-hand, but from context I'd guess it's simply getting consultants from McKinsey & Company[0] jobs in the ‘infiltrated’ firm, on the basis that it has no in-house employees capable of properly setting up / running the bizarrely complex and idiosyncratic SCM-ERP-HCM-BI-etc. software in question.
Basically: You have a problem; Oracle/SAP says they'll solve it for you. Now you have a second problem; they say McKinsey'll solve it for you. Now you have …
Its the cancer that permeates your C-Suite. Meta-stasize-ing,
God I love etymology:
"-Stasize"
**To stand on a hilltop once. To be, this time, on advance. To reach the highest clouds. To touch the limits of space. To out-grow skyscrapers. And make them look at my face.**
---
So FB is becoming a meta-stacious being..
That which wishes to control from a vantage, or t corrupt from within the minds... The light house.
I have zero knowledge of how the Dutch railway scheduling system works, nor of the nature of this failure, but SAP has been blogging recently about using Prolog for combinatorical optimization tasks such as the one that may be in use here or in SAP's Advanced Planning [1]. There are also well-known Prolog proponents from NL.
because accidents and delays happen anytime, and if the schedule can't adapt, you can't run the trains.
people seem to believe that running trains is some simple thing. it is the exact opposite of a simple thing, even for small systems with only a few dozen trains
1 hour more than the standard French contract, and there are also far more holidays here..
> free unlimited 2nd class + low cost 1st class train travel.
.. okay that's actually a great perk. IIRC the French national railway company, SNCF, provide only a limited number of tickets per employee (like 10 per year? someone please correct me if I'm wrong). Their UX is also atrocious though (websites are just shit at workflows, apps are weird and break easily, ticket machines are OK but with really bad touch screens, and train announcement screens are each running Windows XP(with literally hundreds, if not thousands of them on big train stations), while from what i recall NS was acceptable.
I work for the Belgian railways and I have a free unlimited first class pass for Benelux (Belgium, Netherlands and Luxembourg) + lots of international free tickets, also in 1st class. But no company car, obviously.
I love deep diving on public info to determine the internals (even, not only SOPs, policies, but also *governing culture* can be easily gleaned from reading what is placed in public view.
For example, The Salvation Army is one of the biggest jokes in all of charitable history.
I wont go into details here, but I would NEVER give a dime to the Salvation "army" (of free slave labor under the guise of "salvation" (its a tax and money fraud to its core (I went through a SHIT ton of financial documents to see what their true operatin is disguised as. AVOID)
TL;DR: TSA uses its place as a church-like service to get special tax incentives, even the buildings in SF were donated to them. They run a for-profit-grey-market of boutiques they supply with "donations" they have running trades with China-Town in SF. They take subsidies, donations and other benefits. They require all that they "help" sign up for state benefits, sign the benefits over to their org, take the benefits, then feed donated & expired processed fod to their labor force.
They only take in able bodied. Force them to go to church each sunday. and make them use external addiction help, such as AA and make it clear they are not a treatment organiztion.
They maintain a very small amount of actual employees, and play on the egos of those in their "program" using military style rankings to appease the egos of the ignorant...
They then profit like mad, corruption is rampant and they feed all their "good" donatins to their side hustle of boutiques and other grey market.
Its a fn racket.
=---
The point being that if you DD a company/org, you can often find out a lot from financial public info, comments from even just a few employees etc..
Please go into details, I always think it's important to call out charities that are fraudulent and going into details helps with detecting if other charities are fraudulent or not.
What lead you to look into this? How did you originally suspect them of being fraudulent? What clues did you see in their financial documents?
For example, I tend to look at Charity Navigator and for Salvation Army, they mostly get dinged by not having an independent board (which to be fair is a clear red flag) but not much more, so as someone who does want any contribution I make to charities to go toward their cause, I'm very interested in any methodology around researching a charity.
I worked in defence on systems that did similiar and i remmember the one my company inherited and wow and wow it was the most complex, nested peice of SQL stored procs i have ever witnessed. It did the job, but it took HOURS to run. We had to re-write it completely to make any changes. But thats what happens when you let companies like Accenture at a system of yours! You pay them the big bux to be sueable, and they milk you like a cow.
A lifetime ago I did a stint as an Oracle consultant and 100% this. A gigantic sql query was always the answer no matter the question. I remember using sql queries to generate pdf and ps file output that would stream directly to printer feeds. So many “concat”s in that massive blob and nearly impossible to maintain for the next poor soul who had to own the code.
Learned a lot about db best practices to make these systems resilient and reportable but the whole “the database is the system” angle was very much a “every problem is a nail to this hammer” situation.
> sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.
And we can assume that a position in the IT department just opened up...
> It affected the system that generates up-to-date schedules for trains and staff.
...boy oh boy did trying to look into what NS uses for crew scheduling ever send me down a rabbit hole.
I don't know if the systems have changed but while poking around online I found this[1] doc about the Netherlands' timetable revamp around 2006 and it talks about the complexity of TURNI-- their on-the-fly crew scheduling system.
> A typical workday at NS includes approximately 15,000 trips for drivers and 18,000 for conductors. The resulting number of duties is approximately 1,000 for drivers and 1,300 for conductors. This leads to extremely difficult crew scheduling instances. Nevertheless, because of the highly sophisticated applied algorithms, TURNI solves these cases in 24hours of computing time on a personal computer. Therefore, we can construct all crew schedules for all days of the week within just a few days.
Then I found more detail about TURNI's implementation in this[2] paper about optimizing crew scheduling for timetables.
> In the railway industry the sizes of the crew scheduling instances are, in general, a magnitude larger than in the airline industry. Moreover, crew can be relieved during the drive of a train resulting in much more trips per duty than typical in airlines. In other words, the combinatorial explosion is much higher. The latter has made the application of these models in the railway industry prohibitive until recently.
Cool stuff.
Finally, gleaning from ns.nl's careers page[3] everything else in their IT land outside this system runs off SAP (likely including the actual distribution of the output of crew scheduling) so if I had to gamble I'd say the failure happened somewhere in the integration between them.
[1] https://homepages.cwi.nl/~lex/files/Interfaces.pdf
[2] https://repub.eur.nl/pub/11701/ei200803.pdf
[3] https://werkenbijns.nl/werkgebieden/it/sap-specialist-bij-ns...
sidenote: If anyone out there is an SAP specialist ns.nl looks like a pretty great place to work: 36hr week, 5 weeks vacation, pension, and free unlimited 2nd class + low cost 1st class train travel.