The reason that both Access and Excel use is so prevalent in corporate “shadow IT” land is because there are many parts of the business that have problems for which only a negative or marginal business-case can be made for IT to solve it (given the “get out of bed” costs of most IT departments). It’s a barrier-to-entry problem. Excel and Access are cheap enough and fly under the corporate IT radar (no involvement needed), that these marginal problems can be addressed in a self-serve manner.
I’ve worked in organisations that have tried to kill these tools off but unless you can lower the cost-to-play or offer a “better” self/serve alternative, you will fail.
Access is the perfect tool for an intelligent, technically-minded person with limited programming experience to create an application to replace spreadsheets.
There are a lot of those kinds of people out there, and they're extremely useful in introducing minor optimizations that other people wouldn't be able to find. Access is for that guy who says "I know there's a better way to do this," but doesn't have access (no pun intended) to a team of programmers and a project manager.
I didn't major in programming in undergrad but I've taken classes here and there, so in my first job out of college I replaced a really awful system of spreadsheet-jockeying with an Access DB.
I considered other options, but that it's self contained and NOT a web application is a feature, not a bug. I couldn't get access to the corporate database, so I just ran the Access DB on a network drive. It's still probably there, ten years later, running happily on its own.
Honestly, it seems like the fix for Access... is a better version of Access. It fills a very useful niche, between spreadsheets and full-fledged applications designed by programmers. It's so much easier to make a quick app that works for your organization than get an external team of programmers involved, who will probably tell you "no" or remain unconvinced that you're worth helping.
With Access, you don't need political clout, you don't need years of experience, you don't need a title. Just build an Access application and get kudos from everyone in your team for making their lives easier.
These are excellent points, I think it’s underestimated how difficult it is to get approval and budget for a small project that doesn’t directly contribute to the bottom line or have huge savings. These small but impactful solutions using Access act as both a stepping stone to greater skill sets, a way to build POCs for non-tech or low-tech people, and to avoid months of politics in big organizations.
I’ve had projects that were red-taped before getting off the ground due to resources being placed elsewhere. I then take those project ideas, and instead build them in Access as a POC using shared drives, splitting the database, etc. By the time I have 100 users saying how useful the application is in a couple months, other bigger projects haven’t even finished a project plan. Meanwhile we’re then ready to scale the solution using an appropriate stack and basically can just say “replicate our POC features but improve performance, security, accessibility, etc”. So far that’s worked pretty well for me.
In summary: find people’s Excel files with a mess of VBA and formulas —> see if the use case should be expanded —> build POC in Access without permission/budget from a bunch of people —> see how it goes and then plan to scale with the evidence you’ve gathered from your POC.
I taught myself Access 97 when I was about 10 years old and helped a teacher set up a small database to register library books. I couldn't have done that with a real SQL database, but Access was somewhat discoverable. Never underestimate the empowering power of tools that can be learned without any external help! I hate Access, but I love it too.
I know you say that being something other than a web application is a feature, but we have Microsoft's replacement for access in Office 365 and it's PowerApps combined with Flow. I haven't been able to dive too deeply, and it definitely has a learning curve, but so did access. Further, data is actually backed up and permissions can be applied.
I also have fondness for Access, but I think that we have a chance for something like that again.
O365 still gives you desktop versions of the Office software. It does makes me a little worried, but I'm hoping Microsoft isn't suicidal and won't pull the plug on desktop software, leaving us only with the half-featured web versions.
Flow is interesting, but AFAIR it requires extra clicking in the administrative interface to get running, so it might not work for shadow IT as well as Access does.
I am hopeful that between MS's financial commitment to the suite, improved JS performance in the browsers, and a set of practices coming around, it will reach parity in the next few years. I run Linux and while I miss features, I can do my job in an O365 org rather than having to have a mac or run a windows client for office suite.
On the one hand, I'd like it too. I use Linux as a daily driver - but I recently replaced my sidearm machine, and I keep it on Windows, so that I have a computer on which I can run desktop Office.
On the other hand, I'm convinced that once Office on-line reaches feature parity with the desktop, they'll quickly drop files, and turn Office into just another SaaS that holds my data for ransom. I don't want that.
I couldn’t agree with you more. I’m currently involved in a small startup where loads of administrative, transactional data is contained and manipulated by hand in an ever-growing Excel spreadsheet.
I’m a Mac user but there’s plenty of Windows machines with Office lying around, and my first gut instinct was to start throwing something together and importing the Excel file to populate the DB, and construct ‘robust’(er) queries & scripts for interacting with and getting reports from the data.
Perfect? No way. But within a couple of days I was able to get something working more reliably and (albeit temporarily) “shored up” a major calamity waiting to happen. Am I proud of my work? Absolutely not. But it was quick, cheap, and cheerful. Eventually when urgency has lessened and there’s more budget (in terms of mind-space, time, money) we can start looking for developers that will take my kludge and make it into something more bullet-proof.
For now, the leap from Excel to Access has been a huge leap forward (the kind that the physically illiterate offhandedly describe as a “quantum loop”, not realising that quantum leaps, though discrete in nature, are tiny).
I think Developers also underestimate IT Setup Costs for a DB since its so easy to do it locally. Setting up a simple RDBMS like MySql or Postgres and skinning with with a basic, out of the box UI is an afternoon of work if you need to look up commands, or minutes if you have done it recently.
But setting up something for a Team immediately opens questions like Access Rights (Can we limit this just to the team, or even have profiles?), Security (Needs to plan nice on the VPN/intranet), Scalability (in theory easy, but provisioning/managing VMs is hard), and Recoverability (where are the backups stored and who makes them). Those are all hard problems from not just a technical perspective, but also a business one.
But that is just the technical hurdles, not the process ones. In a well (overly?) managed system it can take a handful of people multiple meetings and possibly hundred of man hours to add a new field to a form. Adding the field is trivial, but deciding all the business rules around it is hard. If you run your own system, you can ignore any established process.
So its two fold. Its not just IT start up costs, its also getting everyone to agree on changes to an existing system, versus running your own.
I would argue many of the costs associated with migrating from Access to a "real" DB, is not the actual migration script, but the reassessment of all the business decisions that went into the design of the Access schema. Usually a committee evaluates if they are valid, and what to do with existing data to make it conform to a new schema.
I've worked in multiple organizations that run small Access apps here and there and for us the main cost is just sitting down and doing it.
My last job was at an HR company and things were historically done through Access and spreadsheets, until an IT guy came along and started writing C# web apps, rose in prominence and now runs a small internal software team. I worked for that guy.
It's not terribly politically difficult to convert existing applications, but it's mind numbing work and it's bottom priority. There's not a huge need to change something that already works, and the only reason you do it is you know it can't stay like that forever.
Yeah exactly. Working in an IT Dept when someone said "ah they've built a spreadsheet to do it, but now they want us to add that functionality to the main DB" we'd all roll our eyes because it was another lashed-together thing that we would have to unpick and re-implement. But the fact is, if they'd come straight to us and asked us to do it instead of doing it in Excel first, they would have had nothing, for years and years.
“But the fact is, if they'd come straight to us and asked us to do it instead of doing it in Excel first, they would have had nothing, for years and years.”
Very true. As soon as us professionals take over the dev speed drops drastically and costs go up.
I killed a project a few years ago that had spent about $8M on some project to implement a pretty straightforward business process. No end was in sight. The catalyst was Access 2003 going end of life, and this little app doesn't work.
I don't know why one of a 3 BAs couldn't figure out WTF was happening and fixing it. We had an intern futz around with Access for a few weeks and it's been running fine to this day.
Access, as concept, is great. Is exactly what many need.
But it have a lot of small deficiencies that make it disliked by developers. I have used FoxPro, and it have almost all the right things instead (Fox is Access, but goooood!... except a bit less user friendly).
I wish I could dedicate to build this. Is weird to me why this kind of tool have so little support?
I really like the idea of developing a modern Access alternative, but why are you trying to create a new language? Aren't QBE and SQL sufficient?
Why not something that is like a cloud SaaS version of Access, but can be extended with some procedures and later used as a GraphQL backend like Hasura?
Salesforce is (among other things) basically a cloud SaaS version of Access. And people pay a lot of money for it. See my other comment https://news.ycombinator.com/item?id=21403393
I don't disagree, but isn't it a sign of much deeper issues if this "Shadow IT" land exists at your company in the first place? Why is it that experienced programmers can't efficiently solve a problem that a business user can handle in Access?
Are they under-staffed? Neck-deep in spaghetti code? Lacking business knowledge and the opportunity to acquire it? Bottle-necked by a lack of business analysts or testers?
I hear about this too often and nobody seems to think it's a major problem. IT departments can have a queue that's 2 years deep and people don't even bat an eye, they just think "Oh, that's how IT works!" And that's not even taking into account all the requests people aren't submitting because they've given up hope of getting any dev time.
The problem is most of the people writing code in Access (or monstrous Excel macros or in apps like MatLab etc) are not trained programmers. I think the article sums it up well by using "power user" to describe them - technical people with no formal programming training. Think Engineers (the non software kind), Accountants, Plant Technicians these kinds of people.
In my org (I'm an Engineer at a manufacturing plant) a long time ago someone senior made the decision our company was not in business of hiring programmers so any coding is done by contractors.
So yes people have implemented things (Engineering models for example) in MatLab, or Access, or as an Excel spreadsheet etc as "prototypes" ostensibly intended to be rewritten by a real programmer. At some point these models invariably find their way into production environment and suddenly you have some kind of mission critical thing written in Access (or VB6 or Matlab or Excel).
Then IT has minor panic a contractor is given the job of trying to translate the "prototype" into "real" code and has a battle on their hands.
Often programming contractor doesn't have technical understanding of what model is doing, regardless of choice of language the code engineers write tends to look like Fortran (i.e it's essentially all math formulas), even comments probably don't help when they say stuff like "/* Ergun's formula to calc Delta P */" that doesn't mean a lot to someone without a chem eng background. I've seen contractors produce bugs by doing things like misreading a spec and adding a variable called Fe203 (i.e Fe "two hundred and three) instead of Fe2O3 (Fe "Two Oh three" - a chemical formula). You can't really blame contractor he doesn't know anything about chemistry - he was just hired to rewrite this monster in java based on a spec.
So I don't think its a simple as turning problem over to an experienced programmer and expecting success.
I have seen all of these symptoms at numerous large companies. The backlog is years deep and the costs associated with engaging the professional programmers could be staggering. The truth is, not all solutions need anything more robust than what Office can provide.
Agreed, I think it's because developers like to over engineer everything and the managers like to over engineer the processes.
So many of these access excel solutions should be a days worth of work, simple perl cgi scripts with a minimalist UI deployed by rsync. Instead we have to use our super "productive" modern frameworks, split everything into a thousand files (god forbid you embed an sql query in the only place it's called), add unit tests, etc. There's certainly times for the later approach, but most businesses need much more of the former.
So because developers don't have a reasonable platform to pinch off random little projects others step in.
"So many of these access excel solutions should be a days worth of work"
I don't know why you think a lot of VBA would be replaced by a little perl. Of course, my perspective has a lot to do with the fact that it was essentially impossible to get a new perl module installed where I used to work.
But developers often don't appreciate the importance of presentation (and other) details in reports for managers. Your reference to "a minimalist UI" is telling.
Even though Access and Excel can be buggy, unstable, and annoying, it doesn't make much sense to use anything else if you are automating a report that was previously assembled by hand in Excel, and needs to match precisely.
And often IT types like to exercise power by gatekeeping - if you aren't doing "real" programming, you don't need a Turing complete solution, so Office ends up being the only option. I've been told that if I can select a list of columns from a dataset, and some filters, by pointing and clicking, that's all I, or my managers, need for reports.
Honestly, I think a lot of people find fulfillment in their work through being the person who can say "no" to people, particularly managers that are theoretically higher ranking. And also by expressing themselves through creative decisions when others fail to specify details. I think that using Office/Access/VBA may be correlated to rejecting the value system of most developers, rather than a technical judgment.
> But developers often don't appreciate the importance of presentation (and other) details in reports for managers. Your reference to "a minimalist UI" is telling.
We're talking access here, at best it's ugly and at worst you've got a gaudy background image(1) and a color scheme that would give the disability compliance officer a stroke. Throwing in bootstrap or a more minimal css framework is a huge step up in terms of presentation.
> Even though Access and Excel can be buggy, unstable, and annoying, it doesn't make much sense to use anything else if you are automating a report that was previously assembled by hand in Excel, and needs to match precisely.
I'm thinking of scenarios a bit more complex than that. Access apps generally have a few data input screens, multiple users, etc. Not complicated but not as simple as reports.
I'll admit that I do run away from anything to do with reports, but usually that's because they've installed some "easy to use, no developers required" reporting system that the non-developers can't use and makes life 10 times harder for the developers. If I can just write sql to shove data into an html table or excel template (where we can have the best of both worlds) I'm more than happy too.
1. I actually think of some of these old access programs when I look at windows new built in mail app, who the hell adds a background image?
"We're talking access here, at best it's ugly and at worst you've got a gaudy background image(1) and a color scheme that would give the disability compliance officer a stroke. Throwing in bootstrap or a more minimal css framework is a huge step up in terms of presentation."
I'm talking about creating Excel reports based on pulling stuff from (possibly a random assortment of) databases. Where you can use any feature of Excel. Not that the Access application is distributed to people who care what it looks like.
It seems like you can't even imagine a complex report that isn't an interactive application. So I think we're just talking different languages.
Honestly, I was just talking to someone in the organization I work in with the same lack of understanding. He was like "you have a point and click interface that lets you choose some columns from a table and some filters using simple boolean criteria, what else could you (or your manager) want?"
I want the ability to define all the business rules to produce the formatting and munge the data, I guess. And to structure the code in such a way that it's flexible enough to handle major changes. I need regular expressions. I need to run a diff algorithm on text. I need to use XML and REST to talk to SharePoint. I need to scrape information from a system that I only have access to through a web browser.
Basically, I'm using Access/Excel to do what I used to use Qlikview for, or just plain Perl, and it seems to be less of an "impedance mismatch" as people like to say. Also it doesn't cost as much as a car as Qlik licenses did.
Re: Why is it that experienced programmers can't efficiently solve a problem that a business user can handle in Access?
Because existing web UI standards suck rotting ass to hell and back. The standard was not meant for CRUD and still isn't even if you add gajillion layers of (buggy) JavaScript to emulate a real GUI.
I agree that if you tune it for shop conventions and know its warts well, one can indeed be productive in it. But getting to that point is not necessarily short or easy.
Sometimes it is budget or just needing someone to drive it through a heavy process.
It is also not always worth it if you cannot show a real business case. The thing might help you and it might be worth it if you hack something together yourself but if IT is suppose to build something that is available, has support, has disaster recovery, is patched etc. it might no longer be easy to find the business case for it.
Exactly, to get professional IT resources you need to demonstrate a business case, go through many decision boards and budget allocations, defend it for months, and be ready to be pointed out if it doesn't reach its promises.
So in the end people hack it in Excel or Access. No need to ask permission, no budget issues, no blame if it's not great.
Also: little or no waste happened if it doesn't pan out. Your Access prototype doesn't seem to make you and your team work faster? Copy out the little data you've put in there to Excel, SHIFT+DEL the Access file, and you're done. You've wasted few hours. That beats wasting several man-months of IT time only to not use the result in the end.
Access is agile in the truest form. It lets you prototype and refine in an extremely tight feedback loop, until what you have is a working solution for your specific problems.
I’ve worked in organisations that have tried to kill these tools off but unless you can lower the cost-to-play or offer a “better” self/serve alternative, you will fail.