My first job was as a COBOL programmer on an IBM mainframe (OS/370 I believe) for Xerox. The system was absolutely massive and incredibly difficult to navigate. There were thousands of files with names that only had 6 characters. So everything was a two-letter system code, followed by four digits. LP0456, OP0234, SB1245, etc.
Then, in each of those files the variables had similar names in order to cut down on unique identifiers. WP100, WP101, up to the max number you needed at any given time. It was a nightmare. There was as much external documentation to keep track of all of this stuff as there was code.
On top of that, you could only build programs overnight in a batch process with the rest of the system, which was run by a different team in a different country (Spain, I believe). So you would write your program in the day, phone up the guys in Spain and explain that there was a new program going in and you were responsible for it. Problem was, if your program failed to build it would hold up the whole system, which HAD to be up and running and live again by the morning. This meant that every time you made a change that went into the nightly build you also had to be on call.
I remember eagerly watching the build queue at 1AM, waiting for my file's turn. Then immediately shutting everything down and going to sleep the moment it was successful. If it wasn't successful, well, then you had to fix it on the spot and phone Spain again to tell them to give it another try. Unfortunately, you rarely knew why it was failing.
Writing COBOL on the UNIVAC 490 series (30 bit word, FIELDATA character set), our files were on tapes. A tape request was submitted to the tape library, they would pull the tape(s) containing your file(s), and deliver them to the operations floor.
The customer would submit hand-written transactions, which would be sent to the keypunchers, who would punch them onto cards.
The tapes and transaction cards would be sent to the ops floor (where there was the actual computer). The operator would load the first tape in each file onto a UNISERVO tape drive, and go to the patch panel to run wires to assign logical devices to physical devices.
( https://www.computerhistory.org/collections/catalog/10266686... see page 57)
Then they would put the job deck into the card reader and enter the "UR" (read the Unit Record device) command into the console, which had a spool of paper, not a screen. When the program was awaiting input, they would load the transaction cards and give the console command to continue the program.
We did have a fearsome beast of a FASTRAND drum system for random-access storage, but that was just for executable code and temporary files.
Then you might well get called in the middle of the night, because one of the programs in the job aborted. They called it "aborted" then. Or ABEND (ABnormal END), if you were an IBM person. You would come in, read the core dump (reading core dumps was a valuable skill), and determine that there was a data problem in a transaction (e.g. an alpha character in a numeric field). You would find the offending card, pull it, tear it in half and set it aside to be addressed in the morning as a missing transaction, and tell ops to rerun the job. With luck, there were no bad cards after that one.
Tell that to kids today, and they won't believe you.
> “Logic doesn't deteriorate,” said Bob England, a COBOL developer and president of England Technical Services. “Until the business rules change, that code is as good as the day it was born.”
True, but the business context always changes. I doubt most of the payroll logic from 50 years ago is still in use today. Therefore someone needs to write new code, or change the old code.
> Many COBOL applications are large; rewriting them isn’t cost-effective—or even necessary.
Figuring out the requirements is a research project unto itself. The churn of people over the years means there is no single source of truth for what the system does or why. I wrote an answer about this at https://softwareengineering.stackexchange.com/questions/4252...
You generally want to improve on what you have in some way (or else why do it?), but the line between the warts and requirements is blurry.
Incremental replacement is often a hard sell on the relationship side of things because the customer is seeing a reduction in functionality until everything is done. You can mitigate it and work around it, but it tends to excite waterfall urges which drives up risk.
You also have the fact that the organization keeps changing, so your target is also changing.
There is also an opportunity cost to the whole venture. All that time you spent rebuilding what you had could have been put into new ventures or back into core business.
If you look into modernization type efforts, you see a lot of failure and a lot of loss. That isn't to say it's impossible or that it isn't ever the right answer, but it definitely isn't safe or easy.
Also, sunk cost fallacy. Any reasonably complex COBOL application has lots of lines of code, which are easily counted and in some cases may have even been originally paid for per-line on expensive contracts still on file. That gives a lot of "weight" to that sunk cost on the books, even in timelines of half-centuries. "You want to rewrite a hundred thousand lines of once very expensive code?" you can hear the ghosts of bean counters past shouting in the corridors no matter how much you explain to them that they should have better amortized and depreciated those costs decades ago.
Rewriting software is a massive slog and is prone to all kinds of subtle, possibly very expensive bugs. Especially when the original software has no tests to verify expected behavior in edge cases.
If you hired and trained competent people to do the rewrite, sure. A lot of places will just outsource to a cheap offshore team and be surprised when nothing works.
Regulatory changes in banks kept accelerating in the last 50Y and I see no reason this will stop. It is harder and harder for them to keep up. Reasons for this are:
* Banks still have very old codebases and changing anything is fear inducing for everyone (automated testing is a new things for a lot of teams)
* Banks control very little of their core software. They have a lot of vendors for the same thing. You will see multiple vendors delivering basically the same thing in different parts of the bank, and at the same time multiple versions of those vendor applications in diferent parts of the bank. Integrating this is a nightmare (it can take more than 1Y to move a tag in a XML).
* IT is still mostly used for marketing purposes in banks, it's not a way of doing things. The design/integration is mostly discussed/designed by functional people. They are great, but it does happen that they don't see the full implications of their choices till it is too late. Also, because this is a design 'from above' changing anything takes a lot longer. For any change you have to convince a lot more people till you get to the actual person that will have to do it.
* A lot more complicated management reasons. Nobody wants to take a hit on a new IT project, and few people have the IT & banking knowledge to move such a project (why would you? they pay is average). You cannot progress with just one of them. Places where these projects happen is mostly close to pricing/trading as those are seen as profit centers and the bank is more willing to fail a few times till it can build something decent. As you get closer to the guts of the bank things get more and more...problematic.
Edit:
I had to add this. Central banks are also bad at IT, they do not set an example worth following. I consider this the main reason why all our interactions with banks are less than ideal online.
This reeks as PR for some ancient companies desperate enough to please please PLEASE find someone to keep their unholy mess alive. I've seen the COBOL world, and it wasn't pretty.
Each line on its own is indeed simple. But you have a few thousands or ten thousands of them in 1 transaction or batch process. And that batch process is just 1 in a long chain of batch processes. So you'll need to get a grip on 10-100K lines before you understand the whole process. Procedures are easily 1000 lines long, and any line in there can undo any other line if someone from 30 years ago did not find another way to fix a bug.
There is no abstraction, so code duplication is everywhere. 5 things that seem mostly identical, except in some edge cases means in reality 1 that is correct, 3 that were forgotten in the update process, and 1 that really is different.
Every company is different. As there is almost no human movement, and this is from before networking was widespread, knowledge from each company is an island on its own.
The thing is deeply legacy. Ideas like encryption, unicode, a hostile internet network, ... are alien to the COBOL world. The underlying assumption is that the central computer is holy, and humans need to adapt to its whims. You'd better change your name if it contains an accent not available in the local EBCDIC variant. Putting an € sign in a string? That newby char only exists for 2 decades, hope someone migrated to a code page containing it. On the pro side, Hacking attempts are only allowed in the business hours, as the thing switches to batch processing mode at night.
An organized testing methodology is rare. The cobol world depends on the knowledge of the programmer to remember the edge cases. No things like automated unit testing, maybe a private db with a few records.
The data model is horrible. Choose the size of a record and live with it. You want more chars? Find some spot that's still free and put it in there. The overlays, C unions on steroids, mean any location can contain anything, and any missing IF means you're overwriting someone else's data. I saw cobol devs argue for truncating email adresses. They'd done it with physical mail adresses too, and the mail still arrives. Those mail servers should learn to guess the missing chars at the end, just like the postal workers did.
You can use COBOL in a modern setting, but it is not COBOL itself that is the problem. It is the whole ecosystem around it.
And of course, they want you to have a 30% pay cut, as COBOL is easy.
Companies want to pay less to work with COBOL? I was always told you’d get a nice bump to work with COBOL because so few people are still around to maintain these critical systems?
Same reason big tech companies are so desperately funding "learn to code" programs, especially in high school.
Don't think for a second that the software engineering gravy train won't come to a screeching end as soon as supply catches up with demand.
I wouldn't even be surprised if us SE's eventually try to heavily license and gatekeep the industry like law and medicine, loading new workers up with 6 figures of debt and significant post bachelors requirements, in order to keep our salaries artificially high. Medicine in the US seems to be especially damaged by this.
Ugh - I never thought of this (I work in IT) but I unfortunately could see this happening in the future. Weird times - Your healthcare example is unfortunately completely spot on
American medicine is made increasingly worse by legal liability.
From what I've observed by my conversations with friends and family in medicine, actual physicians increasingly do very little hands on work (with the exception of surgeons) and instead essentially just manage lesser educated technicians, especially RNs and advanced PAs, NPs, and CRNAs, who do the actual work. The physician essentially just stands around and observes but importantly takes most of the legal liability when things inevitably go wrong.
I think this is a huge part of the reason healthcare in the US is so expensive.
Perhaps the future of software engineering will look similar, with principle engineers and architects being like physicians. Extremely well educated, but almost no hands on experience writing code.
Could you imagine if there actually was legal liability for software developers? Imagine a world where ethics considerations actually mattered. Maybe we'd have fewer evil software corporations?
I'm on the wrong continent for FANG/VC funded startups. COBOL was still considered a downgrade vs Java/PHP/NodeJS/... , with a paycut. Why would they pay junior you more than they pay their senior cobolists now?
Of course, the pensioning age of those cobolists might give them a reason in the extremely near future ;-)
I've recently started messing with a little cobol, the "display/accept" screen creation is really quite a nice method of developing a TUI. It can be made amazingly quickly and changed quicker than most apps (See https://wmealing.github.io/simple-cobol-query.html for example).
The GNUcobol implementation doesn't seem to support UTF-8 characters, that'd be awesome if it did, preventing even more awesome "screen/accept" input screens.
My big beef that I have with the language is that there is third party "extensions" built right into the language, which is very frustrating when trying to understand as a new user.
Very good. I've never had to use COBOL in anger in my career. I do have fond memories of a little DBASE work and building TUI screens that bind to data/procedures was really easy and productive.
Geez, I wish it was as easy to build web apps the same way.
The terminals originally used on these IBM mainframes were interesting too. Their hardware protocol let you define an entire screen with multiple input fields and would handle accepting all the input in those regions, buffering it all until you submitted the screen. It's very different from the line-oriented and character-oriented terminals we're used to in the UNIX/DOS world.
In the MS-DOS days, I once wrote a CRUD in Turbo Pascal that was part of an inspection tracking system. To make it easier on myself, I stole an idea from dBase (the PC program, not the huge IBM Mainframe thing) and wrote myself a library that did screen editing fields in much the same way. It's a good idea with wide applicability.
Nicely presented. It’s fun but the new-user learning curve is definitely frustrating (especially because most of the documentation is super dry). I ended up with this after a while of playing with the language https://github.com/ShaunLawrie/TicTacTOBOL
Some COBOL was fixed by ensuring the code assumed years after "50" is the 20th century and before "50" is the 21st century. This removed the need to modify the database.
I had COBOL as one of the subjects during my engineering course in the early '90s in India. Absolutely hated it.
We first learnt Pascal. There is a lot to love about Pascal and in those days the Borland TurboPascal compiler was a breeze to use.
Then they dropped the COBOL hammer. We learnt programming this on our comp lab's mini computer. This was the most excruciating class of all.
Finally, we moved to C, followed by Compiler construction, Operating systems etc., The trauma of COBOL still lingers.
Luckily, I never had to touch COBOL as a professional.
ps: In India, for Comp Sci & Engg in those days, we had a fixed set of subjects for the first, second and third years. The fourth year is when we had two electives in each semester. The programming languages were mandatory during the second/third years.
I’m in my early 40s. I know a guy from my college years who is now making a TON of money because he got into COBOL development early in his career and most of his peers are now in their 60s and 70s and retiring. I think he was among the people who helped New Jersey fix some of their COBOL problems a couple years ago [0].
I sometimes wonder what’s going to happen in about 40 years when some then Fortune 50 company who built their entire operation on Node suddenly can’t find Javascript programmers because they’re all retiring. :P
Wait don’t leave us hanging. What dialect? What OS, IDE, brand of COBOL, and learning resources are you getting? Also AFAIK you also need to know lots of environment specific things like job control languages?
Sure, I would like to reply to all but sadly we haven't moved to the host yet and are just developing on Windows machines using an old Cobol Compiler (1974 dialect).
The host system uses COBOL 1985 as far as I know and runs on the newest IBM hardware.
Learning resources: A script our teacher wrote and his exercises and one big IBM book as well as one German "Cobol insiders" book that is not available anymore.
As far as environment specific things: We have guidelines for our code of course but I am not aware of job control languages. But like I said, we aren't on the host system yet.
An instructor I had many years ago said, doing "boring
like COBOL pretty much ensures good Job Security". For me turned out true, I never had job security worries. And I was able to branch out to other things, thus was able to program in many environments.
Does anyone else ever wonder if the bus factor of the economies of some countries equals the number of cobol programmers that keep the biggest banks and similar institutions running? Probably (hopefully) that's a bit of an exaggeration but still
Presumably if trillions were literally on the line, a bank could buy some extremely highly-paid consultants who have been training for exactly this situation precisely so that they can get massively well-paid in just this scenario. Probably some kind of special ops unit inside Capgemini.
Eh, that might mitigate some of the damage, but the older the system, the more quirks and workarounds there are that are not documented anywhere and for which the required knowledge only exist in the minds of the current programming team. I don't see how any team of consultants can protect against that in a case of emergencies.
> “It does exactly what you tell it to do on a line [of code],” explains Stuart. ”Sometimes you get files that are 2,000 lines long, but they’re easy to understand because they simply do what they say they’re doing at that line.”
I don’t think COBOL is a good match for the programs I write in my spare time, but as a job it sounds nice. I guess the main problem if I were to find a job doing it would be the corporate culture at the kinds of companies that use it, but if I could do it remotely using my own workstation with any tools I choose I would do it.
> DNA is really simple. It just consists of small parts that are being used in a chemical reaction. It's very simple to understand the chemical reaction because it's only based on four chemical bases. Sometimes DNA is really long, but it's easy to understand what it does because it simply does what each of its parts does.
I guess it's easy to see the difference between "easy to read a line" and "easy to understand a whole program" now?
Hi, you seem confused. It’s not a binary choice: for a same program, the version where each line is easier to understand by itself is easier to understand as a whole.
That is not true, to the point it'd be fun to make a contest where each line has to look simple, but the result is hard to understand or easy to misunderstand.
Sort of like the underhanded C code contest, but optimizing for simple parts that create a complex whole.
The individual "files" are usually very simple. They are chained together into more complicated programs, but they rarely go much beyond driving a single screen or report.
I never did it myself, but a friend of mine compares writing code for CICS to AWS lambdas: small and self-contained programs, strung together into a transaction processing system.
I have had to deal with e.g. JCL and from my experience it is a PITA. It's like when you have a bunch of small bash tools, each doing their job well. Then you have a this big script, that calls them and has to glue everything together and is hard to test and understand even though the small bits are all easy.
I admit though that some things are easier - but this is mostly due to architecture. You rely on one big host and one big database. You don't have to deal with a distributed system then. However it also limits the range of business applications that you can build with such a system.
These feel-good statements ignore the fact that ancient COBOL was based entirely around global data and GOTOs. There were no block structures or loops. You kind of had procedures (I think), in which programs can call other programs. But you still ended up with thousand line files, all linked by GOTOs and GOSUB.
All GOTOs. Is GOTO not powerful enough? Good news, we have GOTO DEPENDING ON. The destination of your GOTO is determined by a variable.
Youngsters may not realize just how bad GOTO based code can be. This is where the term spaghetti-code comes from. I still remember as a teenager, trying to understand how a BASIC maze generator program worked. I printed out several hundred lines of source and proceeded to study it. And eventually gave up after an hour.
I'm sure later variants of COBOL introduced blocks, loops and procedures. But that 50+ year old system will be all GOTOs...
I see, even if clear unambiguous lines by themselves are a good thing, there might be plenty of reasons to stay away from a COBOL system in particular.
Things such as function overloading, operator overloading, dynamic dispatch, etc, are all conveinient but also obscure what is actually happening.
To take C++ as an example (I love C++ btw, this isn't another C++ bashing), a line as simple as "variable1 = variable2;", could be doing all kinds of things (including not even assigning) because of the ability to overload the asignment operator. As someone reading this line of code for the first time, you don't really know for sure what is actually happening unless you also go and read the code for "=".
I wonder if the banks are getting cold feet and pushing PR articles about how "kids this days" should get off their behinds and learn COBOL because "someone needs to work on this mess and we can't find anyone"
Why not? Good salary, no stress, nearly never deadlines, working with massive mainframes, no hw failures, no OS failures, and in bank/insurance pretty good bonuse's...but hey maybe you like systems like borg/kubernetes where everything is build on cheap stuff and your deployment is like a babylonian tower build on unstable ground.
Good salary is subjective. Tell me when these banks start paying 150-200k and I’ll consider it. Most offerings I’ve seen is between 75-90k.
Also there is no career growth with COBOL. Its a dead language. Learning it is as useful as learning a proprietary tool that is only used at one company. You’d be better off learning React, Rust, or some other language that can apply to several different companies.
Not only is there no career growth, you don't get to implement new features at all. Ancient software maintenance is soul-crushing, on a good day nothing happens, on a bad day you might have to do hours of digging to find the source of an error.
>Good salary is subjective. Tell me when these banks start paying 150-200k
Living just in Switzerland (remember 3rd world country), 150-200k is not just a good but really good salary, but maybe i should move to San Francisco to learn what a good salary is...maybe i am ignorant to think that someone with 100k has a good salary, but who knows.
This is the kind of vapid marketing nugget that doesn't impress anyone that doesn't suffer from oxygen deprivation due to neck decorations. Why in tarnation should I care for overpriced obsolete technology? Do you see Google or FB investing in this? No? Ok then
> where everything is build on cheap stuff
pulled straight out of the IBM sales book I see
I won't touch anything that came out of those 3 letters
This strikes me as just as much propaganda. I mean if it's so obsolete, why are "modern" server manufacturers still touting new features that have been in the IBM mainframes for decades?
>Do you see Google or FB investing in this? No? Ok then
As if AD-Company's are any measurement for innovation, you could have said Oxide or hell even Amazon...but NOOO you take the lamest two Firm's out of the "FAANG"-Gang.
You obviously never worked with mainframes....that's sad.
Did you also learn SQL? Most COBOL code I've seen contains SQL. I've always felt SQL statements fit well into COBOL code as the syntaxes are similar in their verboseness.
Yes, I've learned SQL to an introductory level. I have trouble with different joins and so on, but I haven't spent much time on it. Only studied it in the beginning of the year because I thought I could get a T-SQL job. It's going to be covered by this training anyway, it's used a lot here
It was extremely spread out timewise but otherwise very tolerant and pleasant. More potential based than experience based. I applied in like april, got a writing question and a coding question split in two parts in may (cobol snippet, loop with conditions to change variables until exit condition is met, answer is all values in order of appearance), then a logic test online. Then in June an on location interview with two people from the bank, hr and manager. Then I had to provide two references, preferably two bosses but I could give a boss and a coworker. Then sign a contract same month. Begin now in september.
Not a computer scientist, but surely COBOL could be machine translated to another more modern language - or even a dialect of a modern language specifically created to make such a translation verifiably correct?
COBOL was designed for financial analysts, accountants, etc to computerize analog processes. They took it literally sometimes.
My favorite example of this was a data entry routine that was insane - and lifted to a web interface circa 2004. “Turns out” when processing paper, several key transactions filtered through a gentleman, long since passed, who lost an arm in Korea. When paperwork was received, they were arranged and stapled in a way that accommodated his mobility limitations.
Every COBOL code base has stories like that, and there’s no functions or modern metaphors as it’s procedural. So your opportunities for code translation are really scoped to things that you can make a “black box”, never to be touched again.
Remember too this is shitty, unsexy, non-portable work. Top talent started looking elsewhere in the 90s, and often you’re left with a couple of people who understand how stuff works, then mostly folks who value stability over all else. Companies tend to pay for shiny skills, and the folks who really understand their business get second fiddle.
The article even mentions Visual COBOL which is a COBOL compiler to the .NET CLR so you can mix C# and COBOL in the same applications. You can look at the .NET IL disassembly with tools like ILSpy and even get a sense of what the equivalent C# code would look like (and my understanding is that it mostly looks like garbage and nothing like what a good C# library would look like).
One perspective on COBOL and why it is such a "write only" language is that it's barely a stone's throw from an Assembly Language with a verbose BASIC-like phrasing. (Well, historically BASIC derives from lessons learned from COBOL rather than the other way around, but from a modern perspective you get the gist.) Every single line has the readability of English, sort of, with that BASIC-like face. Every single line has just about the simplicity of your average basic processor op-codes, just like assembly. To do anything reasonably complex you need lots of lines building up layers of complexity from the simple instructions, just like assembly. Trying to figure out the high level control flow and analyze the overall logic, just like in any disassembly effort, becomes a mind's eye puzzle because there's few ways to visualize the abstractions in the code because the code is at too low of a level for good abstractions. There's no functions and classes, just lots and lots of little instructions. Endless instructions.
There are tools in the market to do this kind of thing, eg to Java. But the typical problem is that you then have a non-idiomatic code base and it’s harder to maintain and operate than the original. So unless it’s an intermediate step on a much larger and more expensive project to replace the whole system it’s too risky for benefit.
If you translate it so that it does exactly the same, the resulting code has a modern language but is even more horrible than the original COBOL code.
If you translate it so that is does effectively the same, the resulting code is nice.
Unfortunately only the former can be automated. Also, you need to understand the code to do the latter. But reading it is hard and the ones who wrote it are dead/pensioned.
Why? OK, sure, Shakespeare is better when it's been rewritten into modern hip hop rhymes and reinterpreted through lyrical dance but what's the advantage of rewriting something that's working and workable in some other form just because that form is not old?
Sure, there are such translators on the market. The problem is that if you translate COBOL to Java which is the most likely target, you won't end up with any nice OO abstractions but a ton of Java code that looks just like transliterated COBOL.
they already are, but actually most COBOL stuff is pretty safe as far as job security goes.
As with anything, its not about the programming language, its about the business context. COBOL outsourcing means having great documentation so they know what the fuck is actually going on in the codebase. Also, COBOL has less boilerplate than for example java. The boilerplate is not standardized and usually unique to each company, again highering the difficulty to outsource everything offshore.
Most succesfull approach seems to be to import 10 indian guys on premise, and have them work along your existing team.
Dont underestimate the effectiveness of a 59 year old cobol programmer who knows the program in and out, and knows exactly where to adapt something without breaking anything else. There are no dependency checks, linters or unit testing for most of these programs. You basically want 5 newbies working alongside one of these senior level programmers for a year or 2 before you confidently can let them retire.
The big downside is that the companies that use COBOL tend to think of IT as a cost rather than an investment. They don't expect to get any competitive advantages from their technological infrastructure.
I'm, in fact, surprised they never banded together into standardizing and sharing code and processes.
I'm not a commercial developer, and when I see something branded as an "investment," it usually translates into "money pit." The country that's spending more on software development than any other in the world is growing at a rate of roughly 2%/year.
Engineers tend to look askance at categorizing different kinds of money. Granted, if you work in a business that has "profit center" and "cost center" mentality, you want to be in the part of the business that the suits associate with the good kind of cost, rather than the bad, no disputing that.
One reason why companies have unique code is that in fact the code reflects a business process, and they are in fact trying to compete via having better processes.
>I'm not a commercial developer, and when I see something branded as an "investment," it usually translates into "money pit."
>The country that's spending more on software development than any other in the world is growing at a rate of roughly 2%/year.
What is the relationship here? Is this also the country that has the world’s biggest tech companies with the biggest growth in the world for the past 30 years?
Sometimes the user promote elsewhere and after a decade the only guy or ladies know what is going on is the IT lot. Have to advise them what to do, what not to and what is possible. They Ss try their new thing. Just break the system.
The only way out is to redo the whole system sometimes. It is not ease.
In 1998 I was on a contract at a telco and ended up getting drafted to help with their COBOL Y2k issues. I spent my days checking and documenting whether dates were being handled properly and my documentation was sent to India where it was fixed overnight while I slept. When I arrived the next morning, the fixed code was awaiting me to check and document again. Half the time the code still wasn't correct but the cost of Indian labor was low enough that it was still less expensive if they had to fix it two or three times.
Follow the article and some Googling, I found this VSCode extension. Idea like this might help people (and AI?) read, analyze and build some automation on old COBOL code with modern tool chains.
Then, in each of those files the variables had similar names in order to cut down on unique identifiers. WP100, WP101, up to the max number you needed at any given time. It was a nightmare. There was as much external documentation to keep track of all of this stuff as there was code.
On top of that, you could only build programs overnight in a batch process with the rest of the system, which was run by a different team in a different country (Spain, I believe). So you would write your program in the day, phone up the guys in Spain and explain that there was a new program going in and you were responsible for it. Problem was, if your program failed to build it would hold up the whole system, which HAD to be up and running and live again by the morning. This meant that every time you made a change that went into the nightly build you also had to be on call.
I remember eagerly watching the build queue at 1AM, waiting for my file's turn. Then immediately shutting everything down and going to sleep the moment it was successful. If it wasn't successful, well, then you had to fix it on the spot and phone Spain again to tell them to give it another try. Unfortunately, you rarely knew why it was failing.