Newest Industry

Why GrabPERF?

About four years ago, I had a bright idea that I would like to learn more about how to build and scale a small Web performance measurement platform. I’ve worked in the Web performance industry for nearly a decade now, and this was an experimental platform for me to examine and encounter many of the challenges that I see on a daily basis.

The effort was so successful and garnered enough attention during the initial blogging boom that I was able to sell the whole platform for a tiny (that is not a typo) sum to Technorati.

The name is taken from another experimental tool I wrote called GrabIT2 which uses the PHP cURL libraries to capture timings and HTML data for HTTP requests. It is an extension of my articles and writings on Web performance that started at Webperformance.org, and that have since moved to this blog.

What is GrabERF?

GrabPERF is a multi-location measurement platform, based on PERL, cURL, PHP, and MySQL that is designed to

• Measure the base HTML or a single-object target using HTTP or HTTPS
• Report the data to a central database (located in the San Francisco Area)
• Report the data using a GUI or through text based download

Why not Full Pages with all Objects?

Reason 1: I work for a company that already does that. Lawyers and MBAs among you, do the math.

Reason 2: I am an analyst, not a programmer. The best I can say about my measurement script is hack job.

Why is the GrabPERF interface so clunky?

See reason 2 above.

If you want to write your own interface to the data, let me know.

Why has the interface not changed in nearly three years?

The current interface works. It’s simple, clean, and delivers the data that I and the regular users need to analyze performance issues. If there is something more that you would like to see, let me know!

I like what I see. How can I host a measurement location?

Just contact me, and I can provide you with a list of PERL modules you will need to install on your linux server. In return, I need a static IP address of the machine hosting the measurement agent.

How stable is GrabPERF?

Most of the time, I forget it’s even running. I have logged onto the servers and typed in uptime and discovered that it’s been 6 months or more since the servers have been re-booted.

It was designed to be simple, because that’s all I know how to do. The lack of complexity makes it effectively self-managing.

Shouldn’t all systems be that way?

What if my question isn’t asked / answered here?

Your should know the answer to this by now: contact me.

Tags: experimental tool, GrabIT2, GrabPERF, GrabPERF.org, GUI, HTML, HTTP, Linux, Linux: Server, Perl, PHP, Web Performance, Web performance industry, Web performance measurement platform

Introduction to the Series

This is the first of a four-part series focusing on the reasons why companies measure their Web performance. This perspective is substantially different than ones posited by others in the field as they focus on the meat and potatoes reasons, rather than the sometimes more difficult to imagine future effects that measurement will bring.

Reason One: Customer Generation

It is critical that a company be able to show that their Web services are superior to others, especially in the third-party services and delivery sectors of the Web. In this area, Web performance measurement is key to demonstrating the value and advantage of a service versus the option of self-delivering or using another competitor’s service.

Comparative benchmarking that clearly demonstrates the performance of each of the competitive services in the geographic regions that are of greatest interest to the prospect is key to these Web performance measurements. To achieve truly competitive benchmarks and prove the value of a service, measurements must be realistic and flexible.

In the CDN field, a one object fits all approach is no longer valid. CDNs are responsible for delivering not just images or static objects, but may also host an entire application on their edge servers, serving both HTTP and HTTPS content. In other cases, the application may not be hosted at the edge, but the edge server may act a s a proxy for the application, using advancing routing algorithms to deliver the visitor the requested dynamic content more quickly (in theory) than the origin server alone.

This complex range of services means that a CDN has to be willing to demonstrate effective and efficient service delivery before the sale is complete. A CDN has to be willing to expose their system not just to the backbone-based measurements offered in a traditional customer generation process, but to take measurements from the real-user perspective.

Ad-providers have to be willing to show that their service does not affect the overall performance of the site they are trying to place their content on. Web analytics firms face the same challenge. Web analytics firms have one advantage: if their object doesn’t load properly, it may not effect the visitor experience. However, neither ad-providers nor Web-analytics providers can hide frow Web measurement collection methods that show all of the bling and the blemishes.

Using Web performance measurements to generate customers is a way that a firm can clearly show that they have faith enough in their service to openly compare it to other providers and to the status quo.

But woe be the firm who uses Web performance metrics in a way that tries to show only their good side. Prospects become former prospects very quickly if a firm using Web performance data to generate new business is found to be gaming the system to their advantage. And it will happen.

Customer Generation is a key method that Web performance measurements are used by firms to clearly show how their service is superior to what a prospect currently has, or is also considering. However, this method does come with substantial caveats, including

• The need to measure what is relevant
• The need to measure from where the prospect has the greatest interest
• The need to consider that gaming the system to show advantage will cost a firm in the end.

Tags: firm using Web performance data, frow Web measurement collection methods, HTTP, origin server, Web analytics firms, Web Performance, Web performance measurement, Web performance measurements, Web performance metrics, Web services, Web-analytics providers

The GrabPERF Black Friday Dashboard is done for another year and there were two performance victims that suffered the most at the hands of the onslaught of bargain-hunters in the area of Web performance.

Some caveats that I need to mention about the GrabPERF measurement methodology.

1. Only the base HTML file of each site is measured.
2. Only the base HTML of the homepage is measured. This means that any issues that arose in the shopping process were not captured.

All of the sites in the GrabPERF Holiday Retail Measurement Index can be continually monitored on the GrabPERF Black Friday Dashboard. This page will be available until January 1 2009.

That said, the two primary performance victims this year are HP Shopping and Sears. We focus here on those that did not do that well because sites who have met the Web performance challenge and survived to fight another year are not as interesting from a learning perspective.

HP Shopping

HP Suffered the greatest response time problems, by effectively becoming unresponsive as of 09:00 EST. The greates affect on overall response time came as a result of the First Byte time metric which is a solid proxy for measuring the server or application load, as it is the time between the initial client HTTP request and the server’s HTTP response.

Factored into the poor performance analysis is the fact that GrabPERF only captures data for the base HTML object. If the performance seen here is carried over to the download of all of the graphical content on the page, I would be surprised if anyone was able to make any kind of purchases on the HP web site on Black Friday.

Today, performance has returned to substantially lower levels, indicating that this application was simply not ready for the amount of traffic it received, or ran into a completely unexpected issue when the load increased.

Recommendation for 2009: Load Test the application using this year’s traffic metrics as a baseline for validating the scalability of the application.

Sears

Sears is a returning visitor from last year’s Black Friday measurements. Unfortunately, they return for exactly the same reason that they were on last year - scaling/capacity issues that appear as errors.

And these are the worst kind of errors. As can be seen in the graphic below, the Sears Web site announced to the whole world that they had over-reached and that they could not handle the incoming volume of traffic.

What is interesting is that Sears owns properties that survived the day very well, namely Lands End. The question that must be posed is why does the parent site fail so badly when the child sites handle the traffic without difficulty?

Recommendation for 2009: Load testing for capacity, and meeting with the Lands End team to understand what they are doing to handle the load.

Tags: Black Friday, Black Friday 2008, GrabPERF, HP Shopping, HTML, HTTP, Lands End, Sears, Web Performance, Web performance challenge, Web performance measurement

Once again, the Grabperf Black Friday Dashboard is up and running!

GrabPERF Black Friday Online Retailer Performance Dashboard

Check out your favorite retailer or suggest one we missed!

Tags: Black Friday, Black Friday 2008, GrabPERF, Online Retailer Performance Dashboard, Retailers, Web Performance

The term consultant is bandied about so much in this new economy that it has lost it’s meaning. Wikipedia defines a consultant as

A consultant (from the Latin consultare means “to discuss” from which we also derive words such as consul and counsel) is a professional who provides advice in a particular area of expertise….

A consultant is usually an expert or a professional in a specific field and has a wide knowledge of the subject matter. A consultant usually works for a consultancy firm or is self-employed, and engages with multiple and changing clients. Thus, clients have access to deeper levels of expertise than would be feasible for them to retain in-house, and to purchase only as much service from the outside consultant as desired.

http://en.wikipedia.org/wiki/Consultant

What this definition misses is that a good consultant, especially in a small firm, is not simply a person with specific subject-matter expertise and therefore a subject-matter expert (SME), a consultant is a jack-of-all-trades.

A simple list of skills needed by a good consultant include:

• Sales
• Project Management
• Product Management
• Educator
• Trainer
• Mentor and Coach
• Business Manager
• Subject-Matter Expert

In large consulting organizations, these functions are broken out into specific team members. In a small consultancy, everyone has to be able to manage all of these items.

And then there is another leap: How does a consultant move to being an evangelist? These two roles are substantially different.

While both are SMEs, an evangelist takes that one final step from being a functional expert who is able to make things happen and work in a product to a place where they can stand in front of any audience and make the product sing. It is not just able the abilty to do anymore; it is about the ability to show.

Go through the list of people that you or your organization work with. Do you work with true consultants, SMEs, or evangelists? Which group is most effective in helping your organization get better?  Are you using consultants as expert problem-solvers, or are you simply using them as staff augmentation?

To draw on my experience, I am learning to be a better small-firm consultant. I have developed my skills as a SME and Evangelist over the last decade, but I have not had to be worried about any of the things listed above until the last two years when I started working in a more structured consulting/Professional Services environment.

What has your experience been? Did you start as a SME and become a consultant? Or did you come out of B-school and then develop into a SME?

How has your development as a consultant affected the clients you have worked with and experiences you have had?

Tags: business manager, career, career development, career path, coach, consultant, educator, evangelist, mentor, product manager, project manager, sales, subject-matter expert, trainer

Flying back from SFO after a long and frustrating week introduced me to a new rule that UAL gate staff have been asked to start enforcing. Apparently, my Premier status, which I realize is the lowest of the frequent-flyer levels, means even less now than it did in the past.

Over my career, I had settled on UAL as my carrier of choice. Flying out of SFO for the first 4.5 years in the US meant that UAL was the primary choice to get anywhere. After a while, I became a devoted UAL fan when I realized that in this day of limited overhead bin room having Premier status got you the vaunted 1 on your boarding pass.

I could accept that First-Class and 1K flyers got to board ahead of me - hell, they’re on a first-name basis with most of the flight crews. This didn’t bother me because I knew that I got to board next.

Friday, that changed.

Apparently, the rule is that Premier Executive now rates between the Red Doormat Club and the Premier status flyers.

I have commented in the past about how people who travel a great deal assume too much from their airline frequent-flyer plans. I do not want to become one of these people. All I ask is that this single privilege I had grown accustomed to having be re-instated. I know my travel money doesn’t have a huge effect on your bottom-line, but I stuck with you through thick and thin.

But now this is a really thick move, and my patience has grown thin.

Tags: airline, airline frequent-flyer plans, customer service, frequent flyer, Hierarchy of assumed privilege, Perks, Red Doormat Club, travel money, ual, UAL 1K, UAL Corporation, UAL Premier, UAL Premier Executive, united airlines, United States

It’s November 14 2008. Charles Windsor is 60 today.
Forty - Oh yeah?

Tags: 40, Charles Windsor, Happy Birthday, Prince Charles, Stephen Pierzchala

I fired up the Boston FIoS measurement location today after a couple of days off, and found that suddenly FIoS doesn’t like the BitTorrent.

The line of purple dots all indicate measurements that reported an error code. All of those measurements come from Boston FiOS. See the real-time graph here.

Accident? Design? That I cannot comment on. I simply report on what I see.

Tags: bittorrent, FIoS, GrabPERF, Verizon, Verizon FIoS, Web Performance, Web performance measurement

In the last 24 hours, thanks to the help of some willing volunteers, GrabPERF has seen the addition of three new measurement locations:

• Dallas, TX (USA)
• Virginia (USA)
• London, UK

All of these location have been graciously provided by the team at e-planning.

Thanks to all of you who volunteer your machines and bandwidth for this project.

As always, we are looking for as more measurement locations. It would be great if we could get some data from the Asia-Pacific region.

Tags: Asia-Pacific, Dallas, e-planning, e-planning.net, External Web Performance, GrabPERF, London, United Kingdom, United States, Virginia, Web Performance, Web performance measurement, West Virginia

My new MacBook arrived two weeks ago, and I felt that I had spent enough time with it to actually make some useful comments on the good, the bad, and the headbangingly frustrating.

The Finder

Dear Apple: Shoot the Finder development team. Thanks.

I have switched to Path Finder as a Finder replacement. Truly the finder is one of the most debilitating pieces of software I have ever used. Nautilus on Gnome is a far superior file management system.

Software, in general

On the whole, I have found replacements for most of the Windows tools I use on a regular basis. But, as I am not made of money, I am using GIMP for Mac, and that is just clunky in the X11 environment.

Living in the browser makes my life much more tolerable than those who require the Windows environment. I haven’t got the money to buy Parallels or VMWare Fusion right now, so I am using RDC to connect to my Windows box. Slap Windows in Space #3 Fullscreen, and no one would know the difference.

Haven’t found a good Mind Map tool. And BBEdit is also muchos dineros. So Smultron is the text editor.

Usability

I rate this very high. Other than adjusting to the lack of certain keys (DEL, Pg up/dn, etc), the transition has been seamless. The trackpad is a dream and I miss being able to throw stuff around on my Dell laptops’ trackpads like I can using the one on the MacBook. I do find I leave apps hanging, as I am still adjusting to CMD-q closing the app.

Dashboard. What can I say? It’s what I need - high-level data at a glance, including the Prem Tables!

Overall

After four years waiting for a MacBook, I can say that it has been worth the wait. Solid, dependable, and slowly becoming my primary computer.

The only concern that I have is the aluminum case. I have an aluminum sensitivity, and if my hands start to peel and otherwise be in bad shape, I will have to determine a solution to that issue.

Tags: dashboard, Dell, Dell Inc., GIMP, Mac, Mac Convert, macbook, Parallels, Parallels Inc, Photoshop, Windows