You might want to adjust the pricing tiers. With $19/$79/$399 per month tiers, I have to guess that most people will start by trying the lowest tier; but that's limited to 2k visitors.
If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky.
You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around.
EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.
All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans.
We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
All our plans start with a 30-day free trial so anyone can start with Silver or Gold plan and at the end of the trial period switch plans.
True, but that doesn't help people who don't get that much traffic within a 30 day period.
We also offer 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason (including not reaching statistical significance), we'd be happy to offer you a full refund for the month.
Offering refunds is great, but it's even better if they're not needed. :-)
Why not just collect their credit card, but don't bill them initially so that the barrier to entry is low, then notify that they'll be charged at the end of the month. If they're pleased with the service, they'll probably ignore the email.
Requiring the user to remember to cancel or downgrade seems potentially sneaky.
If you intend to keep things this way, then don't call it a Money Back Guarantee. Instead, be really clear that users have a trial, and billing is in arrears. Something like:
YOUR CREDIT CARD WILL NOT BE CHARGED!
We collect credit card numbers during signup to
prevent fraud. But we won't take a cent until your free
thirty day trial is complete.
The trial is obligation-free; just visit our easy-to-use
cancellation page during the first thirty days and you'll
never be charged.
Consumers know that guarantees and rebates are intentionally painful to collect. They feel more confident and empowered when you don't have their money yet.
One caveat: I've seen successful trials that are a week or two long. At thirty days (and with users who aren't very committed), you might have an issue with users forgetting about the forthcoming charge, and then getting angry when it shows up on their bill next month.
It was three months ago to the day that we announced our private beta. We were thrilled to see thousands of people sign up for the beta. We've worked hard to improve the product over the last three months based on their feedback and we're now ready to come out of beta and launch to the public!
Our goal is to make it as easy as humanly possible for you to create and run A/B tests on your site. All you need to do is enter your website URL and point and click on what you want to change. Absolutely no coding or engineering required. You don't even need to create an account to get started.
We're working hard to improve the product every day based on your feedback-- just reply to this comment or shoot us an email at feedback at optimizely dot com and let us know what you think!
I actually got excited watching that video, and I don't even have something to use it for right now. Looks fantastic. I also really like how you have the plans setup to record stats from all visitors, but only unlock them if you buy the appropriate plan. I assume that means you can start out with the small plan, then upgrade and see stats that have already been recorded? If so, very cool.
Based on the results in the promo video, looks like you should have your video autoplay! ;-)
"Also on the front page: when you enter a URL and hit enter on your keyboard you are taken to some TechCrunch article :)" ... well scratch that, it does not happen any more.
Great job otherwise, I'm "selling" this to my boss tomorrow :)
We've actually just implemented Undo! It didn't make it into the demo video but if you click on the little upside triangle next to a variation's title (e.g. "Variation #1") you have an option to Undo or Redo. There are keyboard shortcuts as well!
As for the front page, is it possible you accidentally hit tab and then enter? If you hit enter right away, it works. If you hit tab, it changes focus to the first testimonial link on the page which is that TC article. We'll fix this so it doesn't happen in the future.
It's quite possible I hit tab first yes. As for the undo: great. To be honest, I haven't even noticed the triangles, so that's why I wasn't aware of the feature.
What's the USP on Optimzely when compared to the already quite formidably awesome Visual Website Optimizer?
Also, as someone that really wants to incorporate this tech into my larger scale marketing efforts I am put off by the pricing models to both of the above mentioned sources.
When only one of several ad campaigns are doing 100k visitors a day you can see how that pricing model does me no good. Particularly when you take into consideration that I never stop testing -something- on a campaigns landing page.
Also would love an API so that something like this could be tightly integrated into my custom conversion analytics solution.
Right now our biggest competitor is non-consumption. Businesses see the value in A/B testing but just aren't doing it. The pain points we are trying to solve are setup & implementation, easy-to-understand real-time results, and resolving the marketing & IT impasse within organizations when it comes to doing A/B testing: marketing wants to do it, IT doesn't have time to help. We want to enable marketers to run A/B tests without having to rely on engineering or IT.
Great feedback on the pricing model. I agree this isn't perfect. We do offer a 100% Money Back Satisfaction Guarantee so if at the end of any month you aren't satisfied for any reason, we'd be happy to offer you a full refund for the month.
DISCLAIMER: I'm the founder of Visual Website Optimizer but I will try to give a fair and honest comparision. Optimizely team, please feel free to correct me if I am wrong in my understanding.
What VWO has that Optimizely doesn't
- VWO has (JavaScript based) A/B and multivariate testing and split URL (based on traffic redirect) on while Optimizely is strictly (JavaScript based) A/B testing. You cannot do multivariate testing with it.
- VWO has heatmap and clickmap reports while Optimizely doesn't
- VWO has various advanced features such as visitor segmentation, cross-domain tracking, testing behind login wall pages, test results notifications, variation screen shot generation, etc.
- VWO has team collaboration such as multiple-permission based logins and subaccounts. Optimizely doesn't have that.
- In VWO, for all plans, CEO equally shares the support work ;)
What Optimizely has that VWO doesn't
- Changing layout of the page. Optimizely has a drag and drop feature in its test designer which is useful if you want to test different layouts (say sidebar to the right). To achieve the same in VWO, you need to enter some CSS (say float:right v/s float:left) as there is no drag-drop support (yet)
- Since Optimizely exclusively focuses on A/B testing (and not MVT), in its test designer you can change several elements of the page (say headline, image and text) at once to create a variation. In VWO (currently), you need to change one element at once for A/B test and multiple elements for multivariate test.
- In optimizely, results update on the page in real time. In VWO, results are also realtime but you have to refresh the report to see latest results (no auto-refreshing)
I hope this was a fair comparision. I wish Optimizely team great luck for their product. I hope A/B testing market is big enough for multiple players to survive!
One benefit you forgot to mention is that your cheapest plan ($49) comes out to $0.0049 per visitor. Optimizely's cheapest plan ($19) comes out to $0.0095 per visitor.
One thing I think we do well is allow you to try Optimizely and create your experiment without needing to create an account or give us your email address. The only time you need to create an account is when you want to save.
We also offer cross-browser testing and uptime monitoring and reporting.
One small correction: we do support testing behind login wall pages. You just need to put the embed code on your page and the tool will load your site even if it's behind a login wall or on your own local intranet behind a firewall.
Our biggest competitor is non-consumption. I'm glad there are other startups out there helping to educate the market about the benefits of A/B testing. A rising tide floats all boats!
Sorry you thought the video was dry. I'm not the best voice actor in the world. We were thinking of putting a song from the Inception soundtrack in the background or using a real voice actor from http://www.voices.com/
Do you think either would help?
Also, would love to share how we went about determining our various plans and rates. We did a lot of customer development on this and it might be interesting to other entrepreneurs. Let me know if you are interested and we'll blog about it.
I actually would be interested in seeing results of testing this video vs. a voice actor and maybe even against a set of videos that were split up and shorter.
Great idea! We're actually running an Optimizely experiment on the http://www.optimizely.com homepage right now. I won't give away what we're experimenting with but we'll blog about it once the experiment is over and share what we learned.
This post prompted me to have a look at optimizely's source. I won't give it away either, but I'll certainly be surprised if there's an appreciable difference between the 3 variations. In fact, if turns out there is, I'd be much more inclined to use A/B testing in general!
Now, a coding question. I'm wondering: For the all_experiments_json property of the main optimizely object, why is the value JSON-as-a-string, rather than just JSON? You're forcing yourself to parse it from a string to native JSON for no apparent reason. Just curious :)
As for your question, I think you're right that we could just write this as straight JSON without wrapping it in quotes. What are the benefits of doing this?
I tried just putting the all_experiments_json property in Firebug without wrapping it in quotes and I get a 'SyntaxError: invalid label' error. Looks like that is because the keys of the dicts are strings. Probably a way to generate this JSON server-side that makes Javascript happy.
One potential downside of making it just straight JSON is that this is generated by our Python template and if for whatever reason this isn't valid JSON it would throw a javascript exception when the file is loaded.
Right now our backend that generates this JSON does this:
First off, there aren't really any benefits, other than it being simpler and more elegant. Right now you're going from Python Dictionary -> JSON-like-string -> JSON. There should be a way to go Python Dictionary -> JSON. It just seems like a code smell. Really though, if it ain't broke, don't worry about it. I was just curious as to why there was this roundabout.
Removing the quotes won't fix the problem, because you've also altered the way characters are escaped via the .replace() calls.
One potential downside of making it just straight JSON is that this is generated by our Python template and if for whatever reason this isn't valid JSON it would throw a javascript exception when the file is loaded.
I'm not familiar with Python, but there shouldn't be any reason why simplejson would return invalid JSON. In the event that it does, your script would still break, anyway. Granted, it won't be a javascript exception, but it would certainly fail elsewhere.
Like I said before, it really doesn't matter because it works :)
Skip the soundtrack, but a professional voice (and more importantly professional recording) could do wonders for the perceived polishedness of the video. That being said, the content was enough to get me interested!
I don't understand people using a CMS thinking that Google Website Optimizer (GWO) for A/B tests is hard
I got myself a plugin for WordPress that lets you insert in any page the GWO javascripts
I created a variation page on WordPress, identified the original one and the conversion page.
On the site i created the test, started it and it's just working.
But even if you're not using a CMS, copy/paste some code at the beginning and end of a HTML page? how is that so hard?
The inspiration for Optimizely came from my experience during the Obama campaign where I was the Director of Analytics. We used Google Website Optimizer extensively and even tried Omniture Test & Target.
The biggest pain point for us was setup time & implementation. We were all pretty technical guys but we only had so many hours in the day. Having to go back to the code base and remove add noscript blocks everywhere was a huge tax on our resources. Omniture has a similar pain called "mBox hell" caused by having to keep going back to your CMS or code base to make changes every time you want to run a new experiment.
We probably only had the opportunity to run 10% of the experiments we wanted to during the campaign. In the end it meant an incremental $60 million we were able to raise but I'm convinced that if we had Optimizely during the campaign that number would have been a lot bigger.
Google Website Optimizer is a great product but if you are short on time and don't want to muck with your code give Optimizely a try. Certainly let us know if you disagree!
I see you're talking of A/B tests for BIG and medium sites then.
If you only do let's say < 10.000 Unique Visits a month you're probably gonna do 1, 2, heck even 5 A/B tests, not more, so that wouldn't be a big time tax, would it?
Also I'm not sure about other opensource CMSes but this one for wordpress you just have to copy the javascripts on a page and create the test on GWO.
Great question. Even when it's just copying (and later removing) several javascript snippets on a page this can be a large hassle for several reasons:
1. For anyone who is non-technical, this can be daunting. Are the snippets in the right places? Was this deployed correctly? Have I set up my conversion goals correctly (GWO only allows for a single goal)? More often than not, tasks like this are passed from marketing onto IT, who are usually overworked as it is.
2. Even for technical folks, deploying and verifying a new version of one's site just to enable (and subsequently disable) a test can be a big hassle.
In our experience, it's often the marketers, inside sales, and product managers in an organization who have the strongest interest in A/B testing. We're trying to eliminate the burden (and subsequent pain) that testing places on IT/engineering, freeing those stakeholders to test to their hearts' content! :)
Using variation pages causes SEO and site problems when the test ends unless you manage your 301s properly, and the marginal load for the redirect to B artificially biases test results. Also, I did a bit of an eye popper demo at a company I just consulted with: deploying an AB test live on my site took two minutes twenty six seconds while slowing my typing speed to explain every step, while the company reports their experiments with GWO require 15 minutes at a minimum. That gives the typical engineer 12 reasons not to AB test.
Hi Patrick, glad to hear that! Curious what you think about client-side A/B testing products in general (Optimizely being just one of them) vs. server-side products (like A/Bingo). Do you face pain points with A/Bingo that might be solved with a client-side javascript based solution?
In general, I feel server side is far superior for developers and client side with a WYSIWYG UI is a fabulous option for other stakeholders. I could talk on this subject for a while but my plane is about to take off. Suffice it to say you can't match power, flexibility, and speed of server side with client side testing. You can move away from just twiddling text to actually redoing workflows or business logic.
I didn't write ABingo with a pain point included, but for Appointment Reminder it will be helpful to do AB testing in non-Rails marketing pages, too. I wouldn't append too much effort optimizing your solution for the needs of people who have written AB test frameworks though.
I would like to see an A/B test where A is no test and B is a traditional A/B test. ;-)
My guess is, similar tests favor the "non-test" or quicker result/load, though I bet it would take A LOT of views to irk this out. Google always talk about these test on shades of blue, I imagine those tests are similar in the number of runs they require (hint: a lot).
I did this before, I was making the case that JavaScript run tests took extra time and that server side tests were superior for large companies because they didn't need to call out to a resource outside their network. Sure enough conversions were 9% lower (percent of a percent, so from 2% to 1.82%). The funny thing was that when we tested another one of these JavaScript based split testers it was coincidentally in the same server room as the hosting provider at the time, so it didn't negatively effect the conversion rate.
That being said, almost nobody split tests. It's a serious problem because marketing usually needs technical to split test, so even if Optimizely cuts your temporary conversion rate by 9% you will still make that back quickly after a couple of split test iterations.
My number one question for optimizely is when are they going to include some sort of call back api or something to track real conversions. The number one concern SaaS businesses have is visitor -> paid account not visitor -> free account or signup screen.
Your variation editor looks really awesome too. How much effort does it take create something like that? I mean just the HTML hover/manipulation etc. I am planning to write a scraper related tool similar to http://open.dapper.net/
Did you encounter any serious cross browser issues? Or any other notable info you would like to share with the HN community. Is there any open source basic implementation that can be used as a starting point? Do you preprocess the page on the server or is it all Javascript magic on the client side?
Dapper is pretty cool. Good luck in making a similar scraper tool.
As for doing the manipulation: They're using jQuery for all of that. The current hover/manipulation stuff they're doing seems pretty simple. So simple, that it makes me want to make a competing service.
For the hover effects, you can do something like:
<style>
.bordered {
border: 4px solid blue;
}
</style>
<script>
var curHltdObj = null;
$("div").live("mouseover mouseout", function(e){
if (curHltdObj) curHltdObj.removeClass("bordered");
if (e.type=="mouseover"){
$(this).addClass("bordered")
curHltdObj = $(this);
}
});
</script>
This is just a starting point. What I would do is put an overlay on top of the hovered item instead. This gives you move flexibility in terms of styling and event control. I just checked out optimizely, and this is actually what they do. Though, curiously, they put their overlay element in before the <body> of the iframe document, but nevertheless it works (in FF, at least).
That actual page editing is incredibly simple as well. For the most part, you position everything relative to where it was. For normal (non positioned) elements, you just set them to position:relative, and set their top/left to be however far the user dragged the item. For other elements, you tack on how the user moved the item to their already existent top/left coordinates. Again, I'm sure it's more complicated than that, but that's the gist of it.
Our biggest challenge in building this tool was compatibility. Every webpage is different and even seemingly innocuous things things like frame busters and javascript redirects presented quite a challenge for us. We were fortunate to have a great set of early beta customers that gave us feedback when things didn't work and we took it one website at a time.
We'd love to share the lessons we learned along the way with the HN community in a blog post. Let me know what you're most interested in and we'll share what we've learned.
This is frame busting. I'm looking for anti frame busting. There's one way which involves calling a page which returns a "nothing" header code when the iframe tries to bust out. Though some sites like stackoverflow even take this into account. I wonder if SO works with optimizely (I'm on iOS, can't test it ATM).
Since optimizely has access to the document, there might be a way to wrap the window.location object and prevent it from changing, before any experiment-site js is loaded.
http://stackoverflow.com/questions/958997/frame-buster-buste...
This keeps getting funnier and funnier :)... that must be the state of the art in recursive busting ;-)... And if someone is so obsessed that his page must not be framed... I guess it is best to leave such a guy alone...
In a couple of spots, the edit HTML dialog overlaid the items I was trying to edit. I attempted and expected to be able to move the edit dialog around the page and out of the way of what I was editing.
On my own site, I was particularly wanting to play with colors/themes. Not sure what the best way to implement this is, but the only way I could do that through this interface was with style tags.
Awesome product! I will likely use it in the near future.
This is a great suggestion. We try to automatically put the dialog in the right place depending on the action you are trying to perform. For example, if you use drag-and-drop and move something to the right-edge of the window, we move the instructions dialog to the left-hand side of the element.
This certainly isn't perfect and we'll work to improve it. Would you mind sending me the URL you were trying and some steps to repro the frustration you had? Feel free to email us at feedback@optimizely.com
This looks brilliant, you just need to nail your pricing structure.. perhaps consider per test or limit the number of tests and variations? Either way, congratulations, some serious skills on show form your engineer(s)!
Great stuff, the video made me want to jump right in and test stuff; too bad I don't have any medium/big sites to test :)
For us folks with small-traffic websites (ie. your Bronze plan), an useful addition would be a suggestion to people to not try too many variations at the same time. (I don't have the math, but I assume less variations means less datapoints until you get significant results)
Perhaps, as another commenter suggested, by estimating how long (in time or visitors) it will take - and by showing this estimate while adding variations.
You are correct, the more variations, the longer the experiment will need to run to reach statistical significance. Estimating how long it will take to finish is a great suggestion and something we plan on doing.
Disclaimer: I have watched the video but not tried the product, so some of these observations may not apply to the the product itself.
Overall, this looks like a great product. I know a number of people who find Google Website Optimizer complicated to use, and I would definitely recommend this to them as a simpler option. I love how slick the browser interface to edit pages is, and I think having the default 'engagement' metric so people can see results without having to set up a goal page is a brilliant idea.
There are a few things you have done which I would consider doing differently.
1. It looks like your mission is to make A/B testing really easy, but your pricing page at the moment doesn't really reflect that. Number of visitors tested is an easy metric, but one that it is hard for me to interpret without lots of knowledge of A/B testing. How many tests does this mean I can run how quickly?
I would also reconsider the additional features you offer in premium packages. Cross-browser testing sounds complex and makes me worry that your site edits will fail in IE6. I don't want to have to test it, I just want it to work. With uptime monitoring, what does this have to do with A/B testing? Bigger sites probably already have some form of monitoring already anyway, so it looks like they are going to pay for something they don't need. I think your core product is strong enough that you don't need to offer these.
2. Showing me the percentage significance level appeals to my inner stats nerd, but I suspect the sort of people I think will benefit most from Optimizely will have difficulty interpreting this number. What level is 'ok'? Having a rank out of 5 below doesn't really address this, is 4/5 ok or do I need 5/5? Google deal with this very well with their bars which turn red or green when they reach significance.
3. The 'select container' option to expand the selection seems non-obvious, and isn't how multi-select works in any other interface I've seen. Maybe allow people to select multiple components and then take their deepest common parent?
There are also some additional features I personally would like to see
1. It would be great if you gave an estimate for how long until my experiment will reach an appropriate significance level (obviously based on % change and traffic seen so far).
2. I would like to be able to choose my conversion action by clicking on a form button or link in your page editor.
3. It would be amazingly useful to have some automatic suggestions for how a page could be changed - on many occasions I've seen people resist A/B testing because the options are so wide and they don't know what to do. Doing this for some simple suggestions sounds possible - e.g. making key links bigger and moving them up the page. Doing anything more sophisticated could be a good challenge though :-)
On pricing page: I agree this isn't perfect. What metric would you like us to segment our plans by ideally? We want to make this as simple as possible so folks know how to budget for this and so they can easily know which plan makes the most sense for them.
On additional features: Are there any other "value-add" services you think bigger companies might want besides just more visitors?
On 'select container': we realize this is a bit confusing and we're working on implementing a multi-select almost exactly as you described.
On choosing conversion action: right now we automatically track all reasonable conversion events on a page (clicks, form submissions, subsequent pageviews, custom events) and we want to make this even easier by allow you to explicitly create a custom conversion event to track by specifying it when you are editing the experiment.
On automatic suggestions for how a page could be changed: this is a hard one and we hope to get there eventually. In the mean time we're going to try to do a better job blogging about best practices and lessons we've learned working with our customers.
Another possibility for pricing would be to simply buy a one time number of visitors for a test. I think that would be a much easier first time sell in my organization.
One test, 30k visitors, x statistical significance - $100.
Perhaps with the option of only deleting variations on the fly that don't work.
Do any of these A/B testing tools support offline conversions in any meaningful way? For us a phone call is a conversion. I'd love to be able to build the logic to support that once (give me an API call along with GET parameters to the page that I can post back to you), and then be able to use tools like this to actually try different tests.
I imagine it would be quite difficult to match up the phone call with what version of the website was being used. One idea: depending on how your incoming calls work you could try using different extensions on each version which would let you track it. A different phone number all together would work too, but might be confusing and bad for SEO.
Hi Enjo, great question. I'm curious--how would you use an API to track a phone call? If you can describe the process as you'd implement it in your org, we might be able to suggest (or build!) a solution..
nice idea. I tried to setup a new a/b test for a site and it became 'not easy' when I wanted to track an adsense click as a goal.
I wrote a manual a/b test for the same site about 7 years ago, and got adsense click goals working with 40 variations of the page, after about a week of working on it after work at night after my regular job.
There is some jquery code you can add to track javascript events which would probably track adsense clicks, but it's 'not easy', ie: I might as well not pay for it.
Cool idea though! I would pay for it if I could track adsense clicks without having to write code. Also, having an option to select multiple dom elements at once would be nice. In addition, logging into the web site and editing the html to add the code in the <head> tag would be nice too, especially since it's kind of a hassle for me to open eclipse and re-deploy to app engine at 1am when I am tired.
Certainly something we are well aware of and trying to improve! We'd love to automatically track ad clicks for you and provide the corresponding reporting to let you know which variation gets you the most clicks.
Please email us at feedback@optimizely.com and let us know if you're interested in this and I'd love to learn more about this use case.
I think it'd be amazing if you could work smething out with the ppl at mouseflow or thier ilk. They could track the clicks and mouse movements for you. Maybe offer it as an add on for a test.
Great job on the product and video - really easy and slick.
What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - http://news.ycombinator.com/item?id=1778104.
Evan Miller has a point, but not as good of one as he thinks.
It is true that multiple peeks mean that eventually any test will find
significance at any level you want. However in A/B tests the peeks
are not independent. This greatly weakens the effect he is talking
about.
Section 7 of my presentation, starting at
http://elem.com/~btilly/effective-ab-testing/#slide59, is about the
question of how long it takes for a test to complete. For that I ran
numerical experiments with constant peeking, literally every time you
add one to A and one to B you peek again. You can see graphs of how
many errors there were, and how long it takes to get an answer.
Here are key points:
- Be suspicious of tests that end quickly. Run them a bit longer on
general principal. (In general I'd call 500 people a very small
test.)
- Nobody can predict how long a test will take. Even if you know the
actual improvement, you still can't predict time to within an order of
magnitude.
- If a test has been running for a long time you know the true
difference is small, so there is no harm in accepting whatever answer
it gives.
Great point. Jumping to pre-mature conclusions is certainly an easy mistake to make.
During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days.
Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.
Whoaa why isn't there a crazy big testimonial screaming that on the homepage?
[EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.]
I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand.
The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.
This website is great. I was able to jump in and mess around with my website's home page within seconds after watching the video. I'm not a developer and being able to help out with A/B testing will be a huge help and one less thing our developers will have to deal with. Looking forward to using it!
This is great, just in time for my testing :), I'll definitely give it a try! I would love for one of your pricing packages to include speaking directly with an optimization consultant, or perhaps receiving a detailed report that includes optimization recommendations.
Why do all of these companies create cookie cutter names like 'Optimizely' is 'ly' the next .com?
To me ly says 'We have no track record, use with caution' I would feel more comfortable using something with a name that isn't following some online fashion.
Optimizely sounds like a great service but after entering my URL it took a long time to load. It would be great if you could speed that up... If I hadn't heard of the link from Hacker News, I would have closed the tab after a few seconds.
I get the impression that home page has been optimised to within an inch of it's life and converts like crazy. A very impressive landing page, I had the overwhelming urge to put in my Url and try it out. Well done.
I didn't believe it could possibly be that easy until I watched the video. Subsequently, I think they should add text to that effect near the video. After all the positive talk on HN, I was expecting something clever but was still surprised by the implementation (single line to paste in!).
I doubt I'm the only one who doesn't always jump to watch demo videos (unless referred by HN) or throw my URL into a strange form. For Ask HN's, I usually just check out the design and try and get a rough idea of the concept, FWIW.
This is a company of really smart and talented people. I have had limited experience with the beta, but have been following the project with much interest. I expect great things from Optimizely.
Just gave it a spin and it's a really cool product - feels a bit buggy at the moment but for the right application this would be a really valuable tool to have. Great job, guys.
Hi Kevin, great to hear! I'm curious what felt buggy. Feel free to email us at support@optimizely.com and give us a heads up. We'd love to look into it!
Hi Reedlaw--I'm not able to reproduce this. I'm assuming you mean you created an experiment on dropbox.com and that the play button somehow gets larger every time you hover your mouse over it. What browser are you using? Thanks!
To clarify: If you want to run one experiment on a single dynamic page (not multiple dynamic pages) you can do that as long as the structure of the page does not change. The best way to find out is to just try it. Load one of these dynamic pages in the tool and create your variations. Then put the embed code on your site and click "Preview variation" from the variation drop-down menu (you can get there by clicking the small upside down triangle next to the variation title, e.g. "Variation #1"). If you have the embed code on your site this will show you the page on your live site. You do NOT need to start the experiment to do this.
Great question! Visitors are bucketed randomly into each variation. Once they have been bucketed they are cookied and will see the same variation over and over again if they reload the page
You specify how much traffic you want to allocate to each bucket. You can do this when editing an experiment and going to "Advanced Settings" :: "Percentage of Traffic to Variations"
Great idea! I'll try to convince my co-founder to write a blog post about his experience going from product manager for Google App Engine to coder using GAE every day as a customer!
Yes! https pages are a great place to start experimenting because they are so close to the conversion event most businesses care about: making money. Optimizely works seamlessly on https and http pages alike.
If you're doing an A/B test with 2k visitors and you've got an "A" conversion rate of 1%, you need to see a 60% improvement (16 conversions vs. 10) in order to have statistical significance. That's a huge improvement, and I doubt many people will be that lucky.
You'll probably be better off setting the limit for the lowest tier at 10k (and adjusting the others upwards too); that should dramatically increase the number of new users who find your service useful and stick around.
EDIT: Or another option would be to have a "free 10,000 visitors" trial rather than a "free 30 days" trial.