Do acceleration products really work for every request, 24x7?

Hi

My company has been investigating possible acceleration products after attending the velocity conf this year. We have identified a top 3 in this domain of Strangeloopnetworks, Aptimize, and Acceloweb. So my first question is are we missing any key players?

My first impressions of the product demonstrations are “it’s too good to be true”. After hunting around a little I’m starting to think this might be the case. Autoanything, a site who used Strangeloop’s acceleration product was really touted as a huge success at the conference this year. Yet when I test that site, I see mixed results. There were several tests run a few weeks back during the conference, and several completed afterwards. I see these have been removed from public viewing now?

Likewise for customers of the Aptimize product. Inconsistent performance. I’m still not sure who Acceloweb’s reference customers are, but I’d really like to hear from anyone who knows.

So, my question to the community is do these products deliver all they promise? Are they actually too good to be true or are my expectations too high? Perhaps my view on what they offer is somewhat distorted. Can anyone give me a real success story? Or can anyone offer a horror story?

We are deciding how to implement several web performance optimisation best practices. The site has been around a long time, and performance has never really been considered with regards to front end changes. Given this situation, we’re anticipating significant cost in doing this, let alone maintaining the changes and standards we introduce. Thus the option to use an acceleration product has potential…

Cheers
stig

I would love to see some websites that use the acceleration products.

It would be nice if those companies had some sort of webpage setup where you could pass a link to it
and it would send back an accelerated webpage so we could see how fast our pages could be.

They all appear to be offering “in the cloud” solutions. I’m willing to bet that you could get each of them to set up a trial where you can test through their cloud to see how it performs before committing.

From what I have seen of all 3 solutions, they should be able to deliver on most of their promises as most of the things they are doing are fairly straightforward (combining scripts and inlining images are the really huge wins assuming your hosting isn’t completely broken and you already have gzip and keep-alives enabled).

Beyond the basics is where you will start to see differences with the products and how well they work is going to be pretty specific to your site.

All the companies I listed above will do a free demo for you. You send them the link, and a few days later they will send you an old vs. new waterfall and video from web page test. The videos look convincing! But is this realistic of final performance users will receive or is this “lab” performance levels which are ultimately unobtainable in the real world?

What concerns me is I never see prices advertised on their websites. I got contacted by one of the companies and it said the starting price was a few thousand and then there is a yearly maintenance fee; most small businesses and start-up companies simply can not afford that. I do not know if all the companies are that expensive. Another concern is I have asked for website urls that use their services so I can run it through Pat’s pagetest but never get any responses. As far as two videos are concerned, would the companies send back average time or their fastest time and if it is average time how many tests were done to get the average?

Was that for their cloud offering or one of the software/device installs? I would have hoped that the cloud offerings were structured so that they were more reasonable for small companies.

There are some free software-based solutions but most of the ones I know about are for php and aren’t NEARLY as comprehensive.

Hopefully more competition in the space will bring the costs down.

The quote I got was from Aptimize.

"The Aptimize Website Accelerator (WAX) starts at $3k - that increases depending on your environment.

After year 1 there is a annual 20% (of your original costs) maintainence fee."

Perhaps that is for the software/device install?

If anybody has any price quotes on cloud offerings, please let us know :slight_smile:

I’m president of Strangeloop. Hopefully I can answer your questions about our products and service.

To address the issue of inconsistent results, there are a few reasons why you might get these:

  1. We are A/B testing the site all the time so that we can quantify performance improvements on an ongoing basis. This means you have a 50% chance of seeing an unaccelerated page.
  2. In some of the A/B tests, we turn off some of the product features to measure and compare the impact of different feature combinations.
  3. Site Optimizer is always learning, and to learn it might send pages back that are not treated.
  4. Optimal Yslow/Page Speed scores are not always the best for overall conversion (especially when it comes to roundtrip reduction).

To respond to your other question: I’m not sure what you mean when you say that AutoAnything tests have been removed. Where did you see the tests originally?

We can do this. Contact me at joshua [dot] bixby [at] strangeloopnetworks [dot] com and let’s talk more.

Here’s how we do our performance tests (not to be confused with running a trial of our service, which follows a different process):

  1. We run the URL you provide through Web Page Test, which is housed on a server in our datacenter in Vancouver. The page test uses a real Internet Explorer 7 browser and a simulated DSL line.
  2. One of our Performance Experts reviews the results to ensure that it accurately reflects your site’s content and contains no anomalies. Then he or she configures our Site Optimizer service to accelerate the page. In order to get an accurate report, we must configure the system for your site’s specific needs. Then we run the page test again.
  3. After the report is generated, it is reviewed by one of our Performance Experts to ensure that it accurately reflects what Site Optimizer can do for the page.
  4. Once we are satisfied with the results, we generate the final report and send it to you.

The reason we use Web Page Test is because it’s the only third-party diagnostic tool that realistically simulates real-world user environments and browser behavior. We do this specifically to ensure that we don’t provide unrealistic, unattainable lab performance levels.

That’s partly because the prices range depending on a bunch of variables that we need to discuss with you before giving you an estimate. And it’s partly because we’re all cagey about revealing our pricing to our competitors. :wink: But I’m always happy to have a no-pressure chat. Please feel free to get in touch with me at the email address above.

Thanks Joshua, I’ve admit I’ve made a mistake on the Autoanything tests being removed. I failed to check show tests from all users, my apologies. A/B testing makes sense, but eventually I’d expect to see all traffic optimised. A faster web for all :). Thanks for replying.[hr]
OK, so we’ve weighed up our two options, Strangeloop and Aptimize. Both offering similar benefits of web page optimisation but with very different approaches. It really depends on the nature of your environment and how quickly you can implement changes to get this stuff up and running.

We’re in touch with representatives in Europe for both companies (I’m based in Holland) and have been given quotes for trial and implementations (can’t share those details sorry). Either way we will be doing a trial with at least one of the products. I’ll let you know how we get on.

Time will tell!

[quote=“astigley, post:1, topic:247”]

You can check out the results from sharepoint.microsoft.com as Aptimize has been running on that site for almost a year now.

http://www.aptimize.com/blog/sharepoint2010

No smoke and mirrors here. Just run the test yourself with WAX OFF and you can see the impact of less is more!

Mike Iem
Aptimize

Hey There,

It is nice to see the details of a test done by Aptimize in the post above.

I was a bit surprised some of the css and javascript were not combined near the top. However, any program that can reduce the number of resources and make the page faster without code changes is a big plus. I would be interested to see how the service stacks up against a website that has already been manually optimized. Perhaps in a few weeks I make a few more tweaks I will request a trial. I am having a pure css menu created to replace my javascript one and I plan to move all javascript to the bottom of the body.

If there are a bunch of variables that go into a price estimate, it would be nice to have some sort of price calculator that customers could take those variables into consideration. I do not understand the whole concept of wanting to hide the prices from competitors. Any consumer that is going to spend some serious money on a product is going to shop around and ask questions; having the prices available just makes it easier for them to find exactly what they are looking for. Any company that wants to hide prices makes me feel that they want to drive up the price rather than giving fair market value based on supply and demand. I am really hoping there is or will be some sort of cloud based service that small business owners can afford.

Wish both of your companies the best of luck and it will be interesting to see what the future will bring.

Sincerely,
Travis Walters

One of the biggest benefits I can think of for the automated solutions over hand-optimizing (other than not having to hand optimize) is that they can do browser-specific optimizations that you wouldn’t want to do by hand (like changing the domain sharding by browser, different methods for inlining images based on browser support, even changing the amount of blank data that needs to be included in an early flush depending on what the browser needs).

For most dev houses the other big win comes in the automatic versioning of resources and long expiration so you don’t have to worry about doing it by hand.

From our research we are aware that most solutions provide significant performance enhancements. However in order to get the most out of them I think you’ll always need to consider making some manual changes to your code.

We will be implementing one of the products (decision tomorrow), and will also be prepared to make some manual code changes in order to get the absolute most out the product. The cost is considerably cheaper than implementing best practice manually (by an order of magnitude).

Tomorrow I’ll let you know which vendor we will select, and also share the site which the product will be trialled on. Looking forward to finding out for ourselves what this can do.

@astigley: Glad I could help. Feel free to get in touch with me if you have any other questions!

I’d have to echo this very strongly. Hand-optimizing sites for every release is complicated and time-consuming enough that it often just doesn’t happen (or starts to regress with new releases). There’s something to be said for having the developers focus on the features without having to worry about the mechanics of sharding, etc.

It is not so bad to manually optimize the template on which websites are based upon.

I love using dreamweaver (.dwt) templates. It makes changes much easier.

Also with dynamic languages having the header and footer in two separate template files makes optimizing a bit easier.

All the stuff in the middle is time consuming though :slight_smile:

We will be trialling the Aptimze product in September. The integration of the Strangeloop product in our environment was little too complicated. A specific problem to us, rather than the product not being up to scratch. So we’ve opted for the Aptimize apache mod for a small part of our site.

The trial will be run on http://www.vodafone.nl/shop (Yeah we know, there is work to be done…) Will certainly be sharing the targets and results here for you to see the before and after results.

Hi,
This is Leonid and I’m the CTO of AcceloWeb.

Both Strangeloop and Aptimize (both very good products) were considered here, but you neglected the third player - us! :).

Feel free to contact sales [at] acceloweb [dot] com to discuss the details with our sales people. Also, I suggest to compare the actual speed benefits (by doing real life testing with actual users) of the different products before reaching a decision, as the 3 products provide different acceleration levels for different web sites.

I’d be happy to assist with any question.

Leonid Fainberg
CTO & Co-Founder
AcceloWeb

The folks at Aptimize are great. Best of luck with your trial, and definitely stay in touch if you’re still in decision-making mode after September.