Can anyone who works with big iron regale us with tales of modern mainframes? These days it seems the term "mainframe" is associated with the 70s and 80s (and COBOL!), but I'm pretty sure the contemporary mainframe market has a bunch of cool stuff that we mere mortals hardly get to hear about.
My father worked on a product called SnapVantage in the late 90s and early 2000s, which basically used the copy-on-write abilities of the virtualized mainframe disk in a StorageTek disk array combined with z/Linux and z/VM as a hypervisor to instantly provision and boot Linux machines.
It looked like a very, very toned down version of the AWS management console, with a near-identical ability to select a distribution, upload SSH keys, find an IP, and have a full, running VM in seconds... but this was in 1999, when watching a Linux box "copy," provision, and come up in just a few seconds was mind-blowing.
I used my nascent Perl skills (as a ten-year-old) to write the installer for the host software, which I believe actually shipped to customers nearly unmodified!
The zSeries still supports a lot of battle-hardened HA features like lock-step execution, hot CPU fail-over, and hot-swap memory and IO which are very rare, esoteric, and hence hard to support in the x86 world (and rare, esoteric, and untested is not what you want in your HA system).
Plus, the number of VMs that can run well on a z/Linux and z/VM environment is amazing - we had hundreds and hundreds of instances running on one zSeries.
There's a very full-featured zSeries emulator called Hercules which will allow you to run z/OS and z/VM (if you can find them, of course). There's also a distribution of MVS 3.8j, the 1980s System 370 / System 390 precursor to z/OS, available for Hercules for free if you want to get a taste of what administering such a system is like. A lot of obscure job-scripting languages like JCL and early scripting languages like REXX were developed for early mainframes and still live on today.
Ex-IBMer here. I worked with WebSphere (IBM's Java App Server) on zOS, and am by no means a Z-series expert.
My understanding is that these machines are amazing at IO, and handle virtualized environments incredibly well (and have so for decades) (e.g. thousands of Linux machines running).
If you need raw CPU performance then a mainframe is not for you. If you need competitive pricing then a mainframe is not for you. If you have a mainframe and need a have an expiring agreement then this machine may be for you. If you have legacy mainframe code then this machine may be for you.
I remember watching a financial update from Sam Palmisano (IBM's previous CEO). I swear he started to drool when referring to IBM's mainframe margins. From my experience mainframes are for legacy companies with legacy systems, as they are not price competitive with other markets that experience real market competition. But, it is interesting and incredibly reliable hardware.
I/O is about channels. Channels are fixed bandwidth queues which generally have a 'signaling' rate, and a 'bandwidth' the one that most people are familiar with is a SATA port.
A SATA port can do some number of IOs per second (this depends on how the controller is designed for the most part) and the SATA data pipe is 6 gigabits/second or 600 megabytes per second these days.
A SATA Disk (the spinning rust kind) is different, it can pump out some maximum amount of data per second (usually about 100 Mbytes/sec and has an average seek time of a few milleseconds. So if you are doing random i/os to the disk, so that every I/O needs a seek, and the seek takes 10 mS, you can do at most 100 I/O operations per second (IOPS) if you transfer 512 bytes with each IOP you get a throughput of 100* .5K or 50K bytes per second. If you transfer 100Kbytes per transaction you get 10 megabytes per second. Anyway the bottom line is that an I/O channel is bound both by its bandwidth and the number of IOPS it can do.
A PCIe 'lane' can do 250Mbytes per second and with Intel's 5500 chipset about 2 million IOPs. (Although those IOPs are spread across all PCIe channels) A "big" server chip set will give you 32 or maybe 48 PCIe "lanes" which can be 12 by 4 (or x4) lanes or 2x16 + 8x1, or 2x16, 1x4 and 4x1 (this is a common desktop config for two video cards and misc peripherals)
The bottle neck is exactly like networking, each I/O device can consume a certain amount of channel bandwidth, since they don't run 24/7 you can "oversubscribe" which is to say potentially have more I/O devices than you have bandwidth to service them (this happens a lot in like big disk arrays) and ultimately the transaction rate of the entire system is constrained by the cross sectional bandwidth of memory and I/O channels such that you get I/O limited in terms of things you can push around.
Anyway, a mainframe has thousands of channels. Lots and lots of them. And each of those channels has a lot of raw bandwidth and can sustain a high transaction rate. This makes a mainframe a transaction monster.
In the taxonomy of computer systems, 'supercomputers' have a monster amount of memory bandwidth and high transaction rates (low latency) between their compute nodes (or cores). Mainframes have a monster amount of I/O bandwidth with high transaction rates between their nodes and I/O devices. General purpose computers achieve good economics by limiting memory and I/O bandwidth to stuff that will fit in a single piece of silicon. (or at most a couple of pieces)
Mainframes tend to have prodigious amounts of expensive, unusually fast, specialised RAM. And then they have extremely smart controller hardware that the OSes can hand off almost everything off to.
And this is a theme replicated throughout mainframe design. Everything is replicated, all pathways are wide as possible, everything is integrated from the hardware up to the application layer.
Take virtualisation, for instance. IBM mainframes have PR/SM, a hardware-level capability to virtualise everything in the system down to the lowest level of granularity. And they've done this since 1985.
Plenty of other machines have Amazing IO in this day and age now. It is a fairly old notion now that Mainframes were much better at IO than other machines (this /was/ true).
That said, they do deal with IO pretty well. They have dedicated offload processors for transferring data, which means for the same (IO intensive) workload, the CP utilisation of a zSeries machine would be much less than that of an x86, pSeries (or other) machine. And in terms of the disk being a bottle neck... If you're pushing to a disk array that has 192GB (pretty standard) of write-back cache... then no, the disks are not a large concern.
But... At the end of the day, does this really weigh up to the cost of Big Iron, or for that matter the additional licensing of software on top of it?... Not really.. is my answer.
Back in the day mainframes had an I/O advantage because they offloaded a lot of I/O processing to coprocessors (e.g. a minicomputer would just have one processor while a mainframe would have one main processor and one or more I/O coprocessors). That advantage appears to be gone now because I think they are using normal PCIe cards and normal cores as I/O coprocessors (which you could also do on a non-mainframe server).
Of course, mainframes are also faster simply because they are more expensive (e.g. 100 PCIe slots are obviously faster than 10 slots), but that's not an intrinsic advantage (in theory you could develop a $10M x86 server — call SGI if you're interested).
Lots of cache memory and very fast buses to very large external disk arrays. An old mainframe trick was to use the previous generation hardware as the IO controller for the current generation...
I used to develop software for system-z(2 years ago). As someone else said, if you are already using mainframes, get another one. If you arent, pretty much you can ignore them. Their hardware is of higher quality, but I think their uptime has more to do with the hermetic seal of human process that exists around the installations. You can't get anything changed in production, ever. And thats fine, because banks have been doing the same thing for the last 30 years.
I don't actually think performance wise they have anything on a cluster of x86 servers. But its hard to benchmark things like that accurately.
From a developer point of view, its not a great technology stack to work on. Its old and out dated, and is pretty incompatible with other platforms. EBCDIC is a pain, 3270 terminal is a pain. Having to allocate files manually is a pain. Having actually about 7 different types of files is a pain.
Its really all a pain. Just dont go there.
Also, I found that guys who had been working on mainframes for a really long time have a pretty outdated view of what the "distributed" world has to offer. A lot of the time they dont understand what "web Scale" at the level of google, facebook et al are operating at.
As an additional funny annecdote, the I/O that a lot of people have been talking about is now conventional if high end disk array's AFAIK. But internally the disks are emulated for the OS to look like the disks it was using back in like 1975. Thats how it gets backwards compatibility, just dont change anything.
As a technologist I'm always excited about new stuff, but there's another part of me that thinks it's really neat that some of the world's most important software was written 30 - 50 years ago and still works just fine.
Can you imagine a time 1000 years into the future where software written today might still be in use? How cool would that be?!
And I've just finished the Iliad (in translation) which dates from about 800BC describing events of ~1250BC.
It's a bit like the descendants thing that was discussed a few days ago on here - either lots of stuff will survive the next 1000 years (and therefore lots of running code will) or almost nothing will.
Given I'm sure many people here have anecdotes about little snippets they wrote in a short amount of time that ended up in use for a decades, even though the code was crappy (someone tell me I'm not alone here...) - I'd be willing to bet something, somewhere will still be around a long time from now.. though a thousand years is a bit hard to comprehend in terms of technological change.
If banks still exist they'll still be emulating zOS on their quantum dimension-folded computers...
>Their hardware is of higher quality, but I think their uptime has more to do with the hermetic seal of human process that exists around the installations. You can't get anything changed in production, ever. And thats fine, because banks have been doing the same thing for the last 30 years.
You hit it. Things break when you change things. The high priesthood that surrounds all mainframes prevents that from happening.
Nowadays, if you have to change stuff, IBM sells you AIX boxes that cost nearly as much as mainframes which sort of integrate with them to support those changes.
A friend of mine worked for several years as a dinosaur herder and still does on-call work for his employer's data centre.
Basically it's a different universe. Everything that might be familiar to someone with a unix or windows background is ... different. More eldritch. More powerful and yet more obtuse.
Oh and JCL. JCL is the devil's work.
One thing they did that I thought was cool was to consolidate a bunch of Linux servers onto their mainframes. They replaced a few racks with z/VM instances. If they needed another server, they could spin it up in literally seconds if the template was already loaded. This was back in 2004, 2005.
You don't buy (actually: rent) a mainframe (actually: CPU cycles and RAM capacity) because it's cost-effective. You do so because the system absolutely must work and you have a mind-numbing number of transactions to process in a hurry.
If you need it done in a hurry, you don't use a mainframe. Their performance is actually quite poor compared to pretty much anything else, as is the cost per transaction. In general, if you need a machine where the hardware never fails, and you can't get the level you need with a cluster, you use a mainframe. Of course, it's still human error that bites you in the ass every time anyway. These days, you really only use a mainframe if that's all you have.
My father has worked for IBM, Airlines and Credit Card companies developing on their mainframes. I believe the the system is called z/TPF?
I can't recall any specific stats or stories right now but you can imagine the discussions between a father who develops primarily with Assembly and a son (me) who ended up being a .NET developer working on various startups. We've had a few talks about the 'cloud' mostly resulting in him laughing at what we call 'web scale'.
I had a number of mainframe mentors back in the 90s who taught me about TPF...
TPF is an OS, transaction processing & database system that was written in mainframe assembler and deployed in 1979 to run SABRE and a few other core transaction systems (VISA, AMEX, Marriott hotels, etc.) - maybe 100 customers still use it today - and it was a descendent of a previous assembler system developed in the 1960's.
It probably was (and is) the closest system to "web scale" through the 80's and 90s's in that it ran some of the highest volume workloads globally (e.g. 5000-10000+ tps back in the 80s / 90s, 25000+ tps today) at 99.99% availability. In context, Twitter hits 10-15k tweets per second during big sports events.
These days it's extremely expensive and slow to make changes to TPF relative to modern technology (as it traditionally required S/370 or S/390 assembler, though these days requires C). IBM has a newer version called z/TPF which makes it easier to code in C or C++ and communicate with the outside world via web services, etc.
Indeed it does seem like there are only a few companies that use this stuff. Like you said, you had to be a pretty big company with massive amounts of transactions to merit it. He's actually contracting with the IRS right working on some newer systems to handle your tax returns. (He tells me every year it's a miracle you get your tax returns every year on the old system)
Whenever he's looking for a new gig he basically has a set list of contacts he goes through. It seems like almost all the developers of these systems know each other.
Let me say that "web scale" to a startup is a different beast than "web scale" to big companies.
I remember being at a startup and being impressed that we were doing a million dynamic pages per hour. Then I got the chance to see a real system at a big company pumping out orders of magnitude more traffic than that per second. (I am not at liberty to share exact numbers.) And doing constant processing on that traffic.
I doubt your father would laugh at systems like that if he learned what they manage to handle.
Oh I'm sure! His examples usually revolved around his days working on credit card processing systems. But I can't even start to imagine the systems required to handle requests for VISA which can supply the needed response-time and up-time demands.
Stop and think about how much traffic you think that Google gets. Don't forget all of the AJAX requests that are sent back and forth, random ads on third party sites (many of whose pages auto-refresh while they are sitting in your browser, each time loading multiple ads), etc.
Let's just say that any semi-reasonable back of the envelope will give you some truly impressive numbers. I don't think that I should say any more than that.
Now consider that this data is constantly being processed to figure out things like click through rates, trending requests, and much, much more. And that data is then fed back into the live site.
The implied transaction rate should be enough to impress the average mainframe user.
Of course Google's scale is impressive but you're talking about datacenters upon datacenters of servers. I always get a chuckle when people talk about "old" mainframes taking up a whole datacenter. Now the Mainframe occupies a corner and the commodity servers take up the datacenter. :P
The Wikipedia article [1] is a pretty good summary. The main advantages are basically very high IO throughput, very good fault tolerance, and very good backwards compatibility.
The mainframe scheduler is fantastic. z/OS has the capability of limiting the resource usages in different dimensions, including memory use. You can give priorities to processes, which means that you can define what is business critical and what is not. If the workload goes too high for the machine, then low priority work just waits.
In any case, I'd advise anyone who can have a try at mainframe to give a look, even if just because it's one of the still used non-Unix based systems.
The trend at the moment is having java based web applications that interface with a mainframe. So we have a web front end that interfaces with a mainframe that processes/stores the data.
Most companies are trying to stop using the mainframe so much, it is laughable though because they are moving from mainframe based solutions, to mainframe based solutions.
When I was working for IBM (software group) I heard anecdotes that in case of accident power outage mainframes can continue execution exactly from the same CPU instruction where it stopped and absolutely no data is lost. I don't know if it's true or not but this sounds cool.
As someone who never has need to touch a mainframe, it's quite strange to look through this. It's not the 120 processor cores and 3 TB of RAM that surprise me; it's things like the cooling system and the I/O system that are so different from what I'm used to from working on a desktop system.
The funny thing is that IBM and other major DP firms have had cloud servers for years. Decades, even. You just can't get to it with a credit card.
I've sometimes thought that a cloud provider backed onto z/VMs could be a nifty business model. Sure, you'd be priced above even the "expensive" providers like AWS; but then ... it's a mainframe. At the very least your customers would get hipster cred, which tends to attract a massive markup.
Mainframes are very, very extended nowadays. Researchers use them, the government uses them, some big business use them.
From my point of view, there are four "mainframe" concepts, and one of them is not a monolithic mainframe per se:
1. Supercomputers: thousands of CPUs, each with access to a few gigabytes of RAM, which can run massively parallel jobs
2. Shared memory machines: 64-256 CPUs, each of them can access to 0.5-2TB of memory
3. GPUs: Hundreds of CPUs, each with access to 1-4 GPUs and a few gigabytes of RAM, to run highly parallel vector jobs
4. Clusters: Commodity hardware (either racked or workstations) connected by a network which uses a queue system
The first three belong to the modern implementation of the mainframe, the last one is a "supercomputing"-like facility, built with regular hardware.
This brings us to OP's article, and to my main point. There are many research centres, business and government which use them, but there is a trend towards using regular Intel processors instead of mainframe-like vector processors.
Why? Because they're easier to program for. Us at the Barcelona Supercomputing Center have worked with ppc, ppc64, cell, ia64, i686 and x86_64 architectures, and here's what happened.
Some of these architectures don't have debugging/profiling tools available.
I joke with my colleagues about how it has been 10 years from the first CPU with hyperthreading and we still don't have fully automated tools to take advantage of parallelism on legacy code. We know it's difficult to do, but that has to be taken into account when planning to buy a new machine. Old code might run slower on the new machine because of the highly parallelism but lower raw speed.
Others (cell) are just so difficult to optimize that we needed 2 postdocs working 2 years only to make a matrix multiplication take about 80% of the full computing power. I'll only say that our next machine was going to be a Cell supercomputer, but that idea was dropped after a scientific panel advised against it.
Some months ago I took a course on GPU programming, and for those of you who have never programmed on vector processors, it is like the Hello World in brainfuck. My CS degree+masters wasn't just enough. It is overwhelming. The guy teaching the course was a PhD with 3 year experience programming GPUs, and he admitted that most of the times, if you compare the time it takes for you to program on a GPU with the time you save running the software, it is just not worth it. Hey, sometimes it is, and you save a lot of time. But not for most cases. So we're back to regular Intel processors again.
What I'm trying to say is that nowadays we buy machines for different purposes (large number of cores, large amount of memory, large throughput for vector multiplication, etc) but most of them are x86_64. GPUs are an exception because we buy software which has already been adapted to vector operations, but somebody had to spend many days adapting the code.
We work with Intel and IBM and we have access to the most recent CPUs and tools. IBM makes great hardware but they don't have anybody writing a development framework for that hardware, so you never reach even 70% of its peak throughput.
TL;DR: Yes, many companies use mainframes, but most of them try to put x86_64 processors in there, because they are easier to parallelise and there are many tested programming tools available.
Unfortunately, none of these are mainframes. Mainframes are a fairly different architecture and usually run OS/390 or z/OS these days. They really aren't about performance computing and are more oriented towards databases and IO.
We also work with banks and process huge amount of data, but we don't use those kind of mainframes.
I'm not saying nobody uses them, but rather that there is a trend to use a NoSQL database on a regular Linux system rather than z/OS.
It's easier to find developers, more widespread support, and hopefully they will be more compatible in the future.
Edit: chuckMcM's comment [1] provides more insight on this. I must admit I've never worked with traditional mainframes myself, but I know of many research groups who are trying to develop new high-troughput data channels for commodity hardware (i.e. fibre-channel disks).
The current trend is going towards Intel+Nvidia+Linux. Anything else might be a good investment for some, but generally a bad idea for most.
Most of banks use linux boxes for downstream systems (reporting and so on) but a mainframe is the centrepiece of it all, executing end of day batch jobs (this can be anything from the feeds to the clearing house, or end of day reconciliation) and processing trades.
You will be surprised, but it is due to the fact that banks were some of the first people to start using computing decades ago, and some of the programs I have seen in use at banks were older than me. Moving everything to a Intel based platform removes that backwards compatability, but z/OS is made with this in mind.
> Most of banks use linux boxes for downstream systems (reporting and so on)
I don't know - for regular client-server computing, I've seen a mix at the 6 banks I've worked at - some Linux, some Windows Server, some Solaris. Most banks have a mix of all three, but have one that they "major" in.
Also, mainframes tend to be more of a Retail banking thing, for the core banking platform.Some Investment Banks use mainframes for back office stuff, but not many (in my experience).
Where I am working now, I have seen plaques in recognition for using a particular mainframe program from VISA for 20+ years. Also, the recently-retired core-banking platform from an acquired retail bank ran on mainframes, but was written in assembler.
What kinds of workloads though? I can see that "supercomputers" might be appropriate for modelling, analysis and reporting but that's a very different kind of scenario to highly available high throughput transaction processing that mainframes have traditionally excelled at.
Nice rant, but All 4 of your examples are supercomputers, not mainframes. A supercomputer is optimized for absolute performance. A mainframe is optimized for reliability and guaranteed performance. With a supercomputer if you get a soft error on a block, you retry the block. On a mainframe, it's too late.
More to the point, mainframes are designed for very good performance on commercial applications -- often database applications that aren't too different from what you could knock together with PHP and MySQL except the names of the tools are different.
It seems strange to say that it's so difficult to program for a GPU. It wasn't hard to learn or to do. The OpenCL specification struck me as quite accessible and straightforward; plus there are a plethora of SDK examples.
Sorry, I don't mean to detract from your excellent points. I was just wondering if there was some miscommunication. The idea of me (a self-taught 24 year old) being more competent than a postdoc feels absurd, so it probably isn't true. Yet GPU programming is very easy for me.
Just an interesting mystery. I would like to help people out if I can.
Learning the syntax of programming for GPU is easy. The problem is porting algorithms to utilize the GPU most efficiently. This means taking into account SIMD architecture, warps, different kinds of memory, ... It's easy to port code to run on the GPU, it's not easy to actually make it run faster than a general purpose CPU.
If you understood this and still think it's easy, then congratulations you'll be able to pick and choose your jobs :)
Can you give an example of an algorithm that you feel is difficult to port to a GPU in a way that takes advantage of its capabilities? I'd like to try my hand at porting one. It's a lot of fun and hasn't been difficult so far. I'm genuinely curious whether it's odd that "taking advantage of the characteristics of different kinds of memory" is quite natural to me, or if I have a distorted view of my own capabilities. Either way I'll learn something, though, which is the fun part.
(Obviously something like scrypt would be difficult to port to a GPU since by definition it's un-parallelizable, but other algorithms should be doable.)
If you have a simple, straight through algorithm running on a few million datapoints. GPU's are great. If it deviates even a little from there, gpu's start to fumble. Writing those two algorithms is an afternoons exercise in Mathematica or matlab, but several weeks adventure on a GPU.
And remember computers are cheaper than humans (Currently, $0.12/hr at amazon and $10/hr minimum wage).
Have you started benchmarking, yet? Just see if you a) can do way better than the CPU, and b) are competitive with other people porting to the GPU. Of course, you'll have to choose an algorithm that's already ported for the second comparison.
Regarding the second comparison, it's easy to do trivially better than someone else's work if you use their work as a starting point. But that type of optimization doesn't really matter as much as the first one you mentioned: the ability to "take that which would have run on a CPU, and parallelize it to run on a GPU.". It has always felt quite natural for me to do that, so it was strange to hear that it's so hard for others. Why is it difficult for them, but easy for me?
The easiest way for me to solve this mystery is to try to port an example someone considers difficult. Do you know of any?
try matrix multplication. I mean a real matrices, say 10000x10000. And once you're there, try a LU decomposition of matrix, with proper numerical erro handling. You'll soon get to the point where you fell it's harder than you tought. And BTW, matrix multiplication or LU decomposition are super fundamental stuff when solving equations (which in turn is super fundamental when you want to compute bridges, constructions, aerodynamics, etc...)
I don't know very much about this, but do you think that part of your success has just been the benefit you get from going from dynamic ram to static ram? Seems that alone would give you a baseline performance improvement?
GPUs use dynamic RAM too (though discrete cards usually have GDDR rather than DDR, which gives higher bandwidth). The most fundamental difference between a CPU and GPU isn't the memory hierarchy (modern GPUs are growing cache hierarchies as well) but the core architecture (vector/highly threaded "small cores", vs. out-of-order/superscalar "big cores").
Hey, maybe you are very good at address and data organizing.
I'm serious. It's just so difficult for the majority of us. As relix said, if you find that easy, you'll make a lot of money working as a GPU programmer.
That being said, there are many real world problems where there are so many data dependences that transforming the input into independent vectors is plain impossible. For those cases, letting the CPU optimize the datapath is better than programming a GPU version yourself.
Just watched the intro video for the water-cooled EC12. I know nothing about mainframes other than what I've learned reading this thread and scanning IBM's website, but man that thing looks wicked.
I really like the "RAIM" stuff that's apparently new. It seems like a similar idea not only to RAID, but also to this "lockstep execution" thing I just read about. I suppose the idea is that you can configure all or certain memory slots to be slaves such that all reads and writes are identical, or does this RAIM stuff also have RAIM with parity for reconstruction plus storage size increase?
Also, is the ability to install this thing on a normal (non-raised floor) pretty new, or has that been an optional feature of larger mainframes for a while?
Finally, the guy in the video mentioned some Java stuff. Do these things typically see lots of Java use or are they mostly programmed closer to the metal?
They used to be commonly programmed very close to the metal. The assembly languages for the System 360 and 370 machines (40 - 50 years ago) were well suited to processing records of data. These days, I believe using these as hypervisors to hundreds of Linux VMs is more common. IBM has their own Java implementation, but I don't know what it's adoption is like in the mainframe industry.
hey actually sell special processors that run Java for 'Free'. Most people don't know that IBM charges you for 'MIPS'. They actually charge you for using your computer. They have some other specialized processors as well ... all are very expensive.
Yes Java runs on the mainframe and runs quite well. CICS can run JVMs which can provide web services using Axis2 as well languages like PHP, Groovy, Javascript (via Rhino) or Java for CICS apps. Websphere runs on the mainframe but it's a bit different than what you find in the distributed world. The JVM that would normally run an application on x86 is broken into two parts the controller and the servants. The controller will work in conjunction with workload manager to deliver incoming work to a number (you configure) of servants which then process it. With this new mainframe it looks like it will have support in hardware and software for JVM's that can self reconfigure on the fly depending on what work needs to be done and what the current load is. I would like to read more on that.
