In defence of Excel, imagine it didn't exist. Then somebody says:
"I've invented a program that allows non-programmers to input loosely-typed semi-structured data without needing to define a schema. It is automatically human-readable, and data can be simple code that can act on other data. It doesn't scale too well, but it can export to CSV and other standard data formats, allowing you to build real systems with it once the non-technical people have finished their logical prototyping."
People would be singing its praises and proclaiming the death of SQL for umpteenth time.
Excel is good in some ways, not in all. We've always had this issue running reports on Excel instead of Oracle because it was much faster at that moment than waiting for IT to fix the bugs. Business processes aside, the results can be devastating once the data you wanted inside Oracle is still sitting on that Excel spreadsheet. Reports generated in Oracle become inaccurate, schedules get delayed, employees work late, everyone becomes miserable.
The one good thing why software and platforms are created is control. The only challenge is how can someone create a seamless web 2.0 spreadsheet almost similar to Google spreadsheets but definitely with more flexibility as Excel's and more security-enabled features like any other enterprise software.
Yes, I saw that startup idea below. If someone wants to do something about it, I'm in like Flynn.
Not only that, but I think a lot of people would be skeptical that your software could even do all those things. While Excel can be dangerous in the hand of the wrong user it is also pretty magical sometimes.
Excel is totally free-form, and for small-scale "databases", it's robust enough (until it isn't). This means the user can do pretty much what they want, without getting a programmer involved. And that's the killer feature, that more advanced or more specific solutions miss.
Let's take an example. If the user wants to stick some free-form text in between the end of sales records for one year and the start of the next, they are free to do that. In any less free-form application, they need to define a "comment record" or similar, and they probably need to get a programmer involved to do that. And although the SaaS web version of their "database" may have a better interface, in a lot of cases, having to get a tech involved to make that sort of change is not a compromise people want to make.
And they have a point. It's not a slight against programmers, it's just that when they need to make that change, you'll be 2 companies and 5 projects down the line, and it won't be possible. The article mentions a sheet that has been in use for 15 years - if that had been made as a proper program, at that time, it likely would have been done as, say, a VB application, with an MDB back-end, and it probably would have had purple buttons. The source code would now be lost, and if the business process changed at all, the choices would be a full re-code, or working around it. I would be surprised if in 15 years time, we don't look at today's pet technologies in the same way that you just did when you read VB and MDB.
For me, the direction that Excel, and other spreadsheet, need to take is the same route that browsers needed to take when IE6 ruled the world. We need standardisation, and innovation. I've written a couple of blogs on this, and for me, the way to go is a central repository of extensions (http://edparcell.posterous.com/how-about-an-app-store-for-ex... for more on that). In the case of Excel "databases", it might be sensible to create an extension to standardise the management and creation of such "databases". It could even allow features like sharing data, backing up etc, but for that to still happen where users are comfortable: within Excel.
See my proposal below. What I'm proposing would only require the installation of a Dropbox-like daemon on workstations and an Excel add-on that could make REST queries to a repository server. Otherwise, users of Excel just keep on using it, but their data is backed up, can be rolled back, and data in spreadsheets is available company-wide and can be integrated.
Seriously though, the problem is that any real solution needs to accommodate housing data internally that's accessible concurrently with backups. Once you do that you still need IT involvement, which still makes it 3 months slower than excel.
+ Most companies have share-point, and the newer versions allow users to define spreadsheets with rules that can be edited directly in excel yet still upload centrally for all users to access with backups.
I think you're trying to find a sweet spot where technology will overcome organizational bureaucracy. It's a tight spot.
I have a feeling the UX has a lot to do with it. Excel is just one big grid and all you need to do is move the cursor around using the arrow keys and key in data. Done.
All the 'Excel-replacement' solutions I've seen are forms based, so there's a lot of pointing and and dragging going on. That's so much slower than just using the keyboard.
Just observe how true Exel ninjas use it... the keyboard calisthenics is mind-blowing.
Perhaps an Excel replacement will need to have an Excel-like interface, or find a way to get rid of excessive pointing and clicking.
"Perhaps an Excel replacement will need to have an Excel-like interface..."
Perhaps an Excel replacement will basically be Excel, with a background script/plugin that scrapes that routinely scrapes the data and feeds it into a database. (Hey! Startup idea). The first entry in each column becomes the data field. The program is smart enough to take a sample data from each column, and use that to auto-pick the best data type. (Also re-checks a random sample periodically and re-defines data type if necessary). Bonus points for creating excel templates for consistency ('actions template', 'to-do list template', 'known bugs'....etc. Last one was a joke) and for easier definition of relationships between different spreadsheets (like for like data comparisons and relationships as long as the user enters the right info in the right cell). Now that Office has migrated to the web, this could be entirely web-based.
People's workflow visually to them remains exactly the same, which would overcome the biggest obstacle - unfamiliarity and resistance to change. Non-techies who need to make a list of customers, processes or actions don't know or think about headaches they are creating for themselves or others down the line. It's only the technically literate who would appreciate the problems of consistent, clean data, version control etc. This solution would begin to answer some of those problems.
I think people massively underestimate spreadsheets. I don't think its a matter of them being so dumb that businesses like them - I think the grid metaphor with dataflow processing of values and the integration of code and data make them unparalleled for business use.
Funny, just last month I sat in a meeting with the president of a $1 billion company to kick off the development of a mission critical app. When I asked him how to go about collecting requirements, he answered, "You don't have to. We're already doing this. Just see what everyone is doing in Excel. My people have to get the job done whether they have support from IT or not."
It's also widely used for issue tracking I bet. I almost said bug tracking, but those tracking bugs are it inclined enough to get away from spreadsheets.
I'd love to say I agree, but I can't count the number of times I've been asked to deliver a list of bugs/issues/vulnerabilities in an Excel spreadsheet. --shudders--
I am actually amazed at how heavily some fortune 500 companies I have worked for relied on Excel as both a database, and repository for what is sometimes the only copy of sensitive financial data...
How about a combined backup/sync solution for Excel spreadsheets? The sync portion could work like Dropbox, but the daemon would only pay attention to Excel files on a given machine, or in a folder, like a user's home directory. There would be a company account on a website or a locally running web appliance, where users could share spreadsheets, particular pages in a spreadsheet, or even particular cells. This would be combined with an Excel macro that can pull-down such shared data from the account website. Updates to data would only be sent when the spreadsheet is saved and uploaded by the local daemon.
This could even be automated by screen-scraping Excel running on virtualized Windows instances.
A automagically simple syncing solution for excel data ala dropbox would be great and a hell of a start. This might even be a nice app for the dropbox api.
It wouldn't only be a great place to start. Once everyone's spreadsheets are backed up, you automatically have a repository of all of their data, which can be federated, shared, synced, data-mined, displayed in a web app, etc...
Doing this with the new open source dropbox thingy Sparkle would be great. I like it because it doesn't force people to switch from Excel, but adds another awesome layer: web access, versioning, syncing,etc.
I think MS dropped the ball with Excel similar way as it did with IE.
Users of Excel see its great potential and its intuitive interface and use it for things that builders of this product have not anticipated.
What's wrong here is that builders do not follow users and don't try to improve their product to better fit what the users are using it for. Excel should turn into actual database many years ago.
How many people use access over excel when it comes to basic things? I don't know why the more advanced db like things in excel, yet on the simpler spectrum of access' features aren't baked into excel in a dabbledb type manner.
I just headed over to DabbleDB's site to check it out and I see that Twitter, of all companies, has acquired it (!) They're no longer accepting new sign-ups. It looked like a totally awesome service. I can't imagine how twitter is going to use this.
The same team also run another analytics app called Trendly. And the team will be joining the analytics team on twitter. So you can see where it's going :)
dabbledb was a great service. I think it was still a little more technical/large dataset feeling though. It's not accepting new signups or accounts now :(. Twitter bought them more for trendly and the team.
iFreeTools Creator, which I am working on, is moving in this direction.
Interestingly, I did not start out to build a replacement for Excel, but to offer customization features for my CRM app. Removing the default CRM entities (or "tables", if you would like to call it that), provides a slate wherein you can build your own custom database apps.
this is exactly the problem we are trying to solve at hypernumbers.com, the internet has been in programmers hands since its inception, wufoo does a better job than most at enabling ordinary people to do more and we are looking to do more of the same.
Ill chuck up a review my startup thread in the next day or so when we get the website / documentation cleaned up, but would be happy to hear what anyone thinks.
In the last office I worked in (mortgage processing place) they used Excel for everything and got no help from IT. One time I heard the general manager say 'I don't want to use Access, Joe Bloggs had everything running on Access a few years ago, and then he left, and then it broke, and nobody knew how to fix it.' In other words, Excel is a safe option because it's so ingrained and there's always Excel whizzes around.
Also, I never thought schools, PCs or technical manuals did an adequate job of explaining what the difference is between a spreadsheet and a database. Usually they just say 'database is for large amounts of data' but as we know businesses have enormous spreadsheets anyway. I'm not even sure how I would explain it without referring specifically to relational databases and saying something like 'think of a database as like a 3D spreadsheet...' and mumbling a lot
This is the problem I'm trying to solve with AppRabbit.com. DabbleDB is a sweet online database, and Wufoo has awesome custom forms and reporting, but for actually creating a database driven web app, well hopefully that's where AppRabbit comes in. An intuitive model builder, column and record level access controls based on group membership and customizable filters, relationships between models in different apps, workflow, etc.
Also, I'm building everything on top of Django with plans for an Export App feature that will give people portability. Build your app and tweak it with us, then export and host it on your own servers if you want (although you'll lose the ability to customize it without getting into the code).
All in all, I think there's room in the market for a lot of different approaches, since everyone's problem is a little different, and everybody's data is different.
I'm struggling to understand what AppRabit actually does. It claims to "give[s] you an easy way to create web based database applications" that is build on top of Django.
Django already is an easy way to create web based database applications so what does AppRabit add to this?
We have a very complex SQL database that we use to keep track of patient data (we are a healthcare company) but to access most of the data for reporting purposes we still have to use Excel and pivot tables.
I'd love to find a better solution but excel so far is still the cheapest/easiest to train.
My previous job was in the healthcare industry at one of the top two patient information companies. I left because we had spent the last 8 years working on a protocol that was extensible enough to handle all the different kinds of data available. At first it was interesting, but once we passed 100+ individual .xsd files, it got to be a bit unwieldy.
Accessing patient data is tough because it's tough to model in a database. It's tough to model in a database because every single vendor exposes data in a slightly different way. HL7, the protocol that is used as an industry standard for this sort of data, has several flavors and everyone implements it their own way.
Also, the industry as a whole has no incentive to move forward. Healthcare providers (aka hospital systems) don't really want your data to be portable because it makes it easier to keep you as a customer. On the other end of the spectrum, FDA regulation has scared the big players into spending 5 hours on documentation for every 1 hour of software written, meaning it costs $20-50,000 (depending on the salesperson you talk to) for a hospital to buy a server that handles a few hundred beds' worth of patient data.
I don't know but I'd love to hear any ideas the main problem is it such a niche market that few people have tried to tackle the problem on a micro level. Instead most companies come in and say let's change everything and lock you into our proprietary solution for the next five years.
I implemented a graphical drill-down on relational data with similar functionality to pivot tables. You could bring up an aggregate view on a table, like calls per day, then right-click on a bar in the bar-graph and choose an AND of a relationship. As an example, let's say it's "representative." Right-click choose "representative" and all of a sudden, you're looking at a graph of calls per day per rep.
The cool thing about that program, was that it was based on an OR framework, so all we had to do was to map it to a particular schema, and it could work with anything. It was implemented in Smalltalk running in a web plugin. It would be easy to do as a webapp in any dynamic language with a good OR framework and chart graphing framework.
The problem is that Excel already has pivot tables for relational data, and charting. The charting isn't interactive, so that would be cool. But you're not going to get people out of Excel just for that one feature. Because there are 50 other features in Excel that they use daily.
All the queries were dynamically built just by using the Object Relational metadata. The only setup needed was to map the schema, which there was a GUI tool for. (Not graphical, just GUI.) Do you think something like that could sell?
I'm pretty surprised no one has mentioned Google Spreadsheets. They are all over this problem - it already has pretty close to perfect interoperability with Excel, but also includes really nice web-friendly features (eg, forms for entry, an Atom interface etc etc)
From the Excel systems I have seen a lot of the business logic is coded at the VBA level. So without full VBA support you only have half the a solution.
Our take on this: users like spreadsheets. The spreadsheet interface is the goose that lays the golden eggs; it's what allows so many people to get so much done. Thus any replacement for Excel has got to preserve the spreadsheet UI and preserve it in a relatively familiar way. Those criteria narrow down the class of possible replacements.
We met with a group of spreadsheet support people once at a fairly large company where, as is typical, IT was trying to force the users off their spreadsheets. Here is how they said users were actually using the spreadsheet-replacing-proper-enterprise-apps that they had been mandated to use:
1. Open the enterprise app.
2. Paste the data into Excel.
3. Do whatever they want.
The problem isn't that the software to replace Excel and Access isn't out there. In fact, Alpha Five does a damn good job at it. I started doing market research for exactly this idea, and saw dozens of people who are trying to approach this market in slightly different ways.
Unfortunately, I'm away from my notes, but my latest thought was this:
1. Able to import basic excel data, even if an MVP can't deal with VBA. Support the basic types of Excel, and be able to interact with the data in a spreadsheet sort of way. (Much like Access.)
2. Don't make IT afraid of this app. Imagine the technical support requirements as people lose data, do something weird. It's a burden to IT. So, it has to be dead-easy to use (Read: like Excel) with incredible documentation (video, audio, tutorials, dead-tree), and has to give buy-in to IT that it will make their lives easier, not harder. Judging by my experience working in shops with small IT, this is really hard.
Or, it has to be for very-small businesses without a major IT presence. Then, it has to be cheap, because they already have excel. This looks to be the biggest barrier! You have to convince the client they have a problem. To me, this looks like the wrong market, though the one I started to look at first.
It's a sexy market. I'd still love in on it, and the people who finally figure it out will be a billion dollar business. However, more people than you know have tried it. It's a deceptively hard problem and I've been thinking about it off and on for over a year now.
I'm starting to think it should look a little different. Take care of an enterprise app framework for small businesses that takes care of single sign on with easy LDAP/AD integration, connections to multiple database, connections to legacy apps (simple links with option for single sign-on integration, with options for new windows or frames) and THEN have an Excel/Wufoo-crossbreed add-on for creating small applications. This would have IT buy-in, as they could use it themselves for small things, it would have built in transparent security, and they would be able to delegate to power users who could define the problem and would be able to work with IT for solutions.
This would be a big app. I haven't found the MVP to extract, save for the framework itself. I'm looking for partners on this idea currently.
I put it at the bottom of the post and if I decided to do something with it, I'd do a more focused ask hn post. Here's a video of what my last startup built, but never released. Ironically I think it solves some of the problems defined in the post. Would this be useful? http://www.youtube.com/watch?v=vNl4cJNdizA
Wow, that looks a lot (lot lot) like my current project, AppRabbit.com. Even the layout looks similar, although this is the first time I've seen your demo or heard about you guys, I swear! I love your approach, and your presentation and UX looks sweet! It's awesome to see other people dabbling in the same area I'm working in.
Ha we deadpooled in december 2008. This was a demo video from back then, but I did ressureect the code on my localhost last night for fun. Shoot me an email: j@jasonlbaptiste.com
A big barrier to something like your product is the notion of something replacing Excel spreadsheets. People love their Excel spreadsheets. They are used because there are a lot of perceived benefits. Such barriers could be avoided by solving the same problems as an adjunct to spreadsheets. (This makes sure that there's a backup of all your company's spreadsheets. You can roll back to any prior version. It also enables easier sharing of data...)
ObjectStudio Smalltalk could do interesting things with Excel spreadsheets through OLE. You could open a spreadsheet and pull data out of individual cells, etc... I'd see if there are other technologies in more popular languages with better licensing terms.
In much the same way that x86 is not the most used computer architecture, due to all of the embedded devices out there. Yes, SQLite is probably used by more people than Excel, but it's not used directly. Excel is used directly by non-programmers constantly.
Using a product that uses a database is not using a database.
If it were, then the winner would probably be Oracle since you are "using that database" every time you, e.g., make a purchase with your credit card. Probably multiple times since your purchase will end up in the databases of the credit card company, the bank, the merchant...
Probably is, but that's why I put the disclaimer in the first sentence. Would love to see if there was somehow a way to grt stats on this, even if it's random sampling. Updated: refer to daekns comment above. It says what I wanted to say.
The text is set to 22px, then that style is overwritten with '100%', meaning 22px. This is huge by desktop standards. 3% of people will be reading your words with an iPad. A much more common font size would be about 16px.
"I've invented a program that allows non-programmers to input loosely-typed semi-structured data without needing to define a schema. It is automatically human-readable, and data can be simple code that can act on other data. It doesn't scale too well, but it can export to CSV and other standard data formats, allowing you to build real systems with it once the non-technical people have finished their logical prototyping."
People would be singing its praises and proclaiming the death of SQL for umpteenth time.