Pontifications on Agile ALM, Continuous Integration and Software Quality...

I recently was invited to participate in Speaker Series program hosted by Morgan Stanley on the topic of Software Quality. It was a fun talk to put together and I was honored to have the chance to present to the global IT and Quality organization at Morgan Stanley.

The invite originated from an article I wrote on Automated Testing that was published by StickyMinds, a great resource for software best-practices.

Here's the presentation... look out for 'The Donald' easter egg in the slide deck!

If you have any questions or would like to discuss please get in touch with me!

Even Intel struggles with this stuff…

For the past few years Intel has been steadily working on bringing an exceptional step forward in processing capability to market. They internally called it Larrabee and it is a chip that is widely rumored to have as many as 32 or more cores on the chip. This would bring multi-core parallel computing to the consumer market in an unprecedented way.

Recently, however, news came out that at least the first iteration of this chip was being officially canned. Although it’s not unheard of for a big company like Intel to sack a project, in this case the number of man years, resources and buzz built up around Larrabee represented a substantial investment on Intel’s part.

This news for me really drove home the challenges that the computing and software industry is facing as we continue to attempt to scale out as opposed to up. To build more processors on a chip instead of faster ones. To fundamentally change the way we design and implement hardware and the applications that run on them.

This stuff is hard! Really hard, in fact. And although the concept of putting a bunch of processors together on a chip is simple, actually harnessing that power to do real work is anything but. There are many reasons for this, but one is that writing multi-threaded software that actually can tap into this power is exceptionally more challenging that just waiting for Intel or AMD to produce another CPU that is 50% faster than the last.

It’s hard to conject exactly what stopped Larrabee in its tracks as it was seemingly so close to launch. What we do know is there was a tremendous amount of horsepower built into the chip, but tapping that potential apparently proved elusive. A public demonstration of Larrabee at a recent trade event was met with a relative ‘Hmmm…where’s the beef’ kind of reaction from the press and analysts.

As the hardware scales out with multi-core and the applications get more complex, the tools and infrastructure needed to support them, including debugging tools and defect resolution solutions, need to grow and evolve as well.

I will be anxious to see what AMD and NVIDIA are working on in their respective labs as each race against each other and Intel to keep up with Mr. Moore. It’s an exciting time as the industry looks for innovation from our hardware and software vendors to keep up the pace of advancement that we’ve grown to expect…

Onward and outward!

Thanks Apple!

On a previous blog posting about TiVo Everywhere I was wondering why the market had yet to produce a good TiVo-like device for radio. Well, finally Apple stepped up and did so. The new Nano packs in quite an impressive bunch of features, including a radio with pause and replay features.

Probably the most impressive is the packing of a 640x480 pixel, 30 frame-per-second video camera into that tiny case giving up to 16+ hours of video on your nano. I would be nervous if I had recently invested in these guys.

The incredible advancement of these recording devices has truly started the democratization of information around the world. Recent events in Iran, Israel and Palestine have been a great examples of that. Every individual now has the ability to transmit exactly what is happening in their world almost instantly. It's impossible to hide, and there are no more information barriers.

So far, this has had a relatively positive impact on the world, in my view. It seems the most powerful change agent is the free flow of information and devices like these are providing the tools.

Technology is great, ain't it?

Ramblings on Starting a Company and Getting Funded...

I recently gave a short interview with TalkTech where I was asked about my thoughts on getting a start-up off the ground, and the attitude needed to persevere through the challenges of growing a business. I make no claims to be an authority on the subject, and if you choose to listen to anything I have to say you do so at your own risk, but here are a few things I will recommend you do before trying anything crazy like starting a company:

1. Make sure your wife/husband/partner is incredibly supportive and ready for the fun that lies ahead...
2. Choose your business partner(s) carefully, you'll need them to be on your side 100% of the time.
3. Choose your employees extremely carefully, and your investors even more so.
   
4. Fix your mistakes as fast as you can and move on... including your hiring mistakes!
5. And probably most importantly: Don't ever accept that something can't be done just because noone else has figured it out yet.

Oh, and don't get too attached to sleep, you may not see much of it for a while.

In any case, if you're interested in a few minutes of my rambling, here's the interview.

Ebay Does It, Why Can't We?

It occurs to me that as web applications have taken over the world over the past decade, what’s become clear is that we have a heck of a lot to learn about managing them.

Complexity in the software industry has sky-rocketed, and applications that once were versioned and released every six months are now being upgraded with new production code every couple of weeks. EBay is a prime example of this model. It is widely known that EBay has become so adept and addressing issues and deploying changes to their servers, that every two weeks a whole new version of Ebay is up and running in their multitude of data centers around the world.

Now, Ebay is not a simple application, and downtime at Ebay can cost millions of dollars, not to mention generate an angry mob of users who are not only desperately trying to buy the latest iPhone 3G, but some of whom are also trying to make their living. This is serious stuff. Downtime at Ebay is front page news.

It’s redundant to point out dangers of our new Software-as-a-Service software paradigm. However, the one-to-many relationship between application server and end-users does present a great deal of pitfalls that did not exist in Bill Gates’ world. When Microsoft Outlook crashes, you may be upset, say something not very nice, then restart it. When your application servers crash, money starts flowing straight out the back door as your customers all collectively say something not very nice about you.

So why aren’t more companies able to follow the Ebay model? What does Ebay know that they don’t?

The answer may lie not just in how they deploy new changes, but how they handle and resolve issues quickly when they do occur. Is this truly one the great remaining challenges in the realm of software? Some would say so, and they have good reasons to back them up.

Multi-tier applications represent some of the greatest levels of complexity ever seen in the software industry. With pieces of your application running on many heterogeneous, physically dispersed servers and environments, understanding what went wrong in these environments can be next to impossible. When issues occur, most often the only hope a team has is to attempt to reproduce the same conditions that caused the error, and hope it happens again. This means that to understand the root-cause of issues, recreating the environment, re-populating the database, and generating the required load on the servers is the only solution. Frequently, the pain of going through this effort is too great, and the issues lie dormant… until the next time something bad happens!

What the software industry is screaming out for is the ability to quickly capture, reproduce, and isolate issues as they occur. What we need is something like ‘TiVo™ for Software’.

One solution that has finally emerged from the chaos introduces the concept of recording and replaying software execution. This technology revolves around the core ability to not only record an application’s execution, but just as importantly, the complex environment in which the application ran.

With this new ability, teams can dispense with massive amounts of inefficient workflows that have traditionally been manual, iterative and error prone.

Imagine this common scenario: Your newly out-sourced team in India is handling QA for your complex, multi-tier application. They’re doing a great job and have found over 100 issues with your application. You’ve got your problem reports, log files, and the very large database datasets that your application was using when the bad things happened.

Next comes the fun part.

Now it’s your turn to bring up the same environment that your Indian team was running. I hope you’re using virtual servers! Finally, let’s take a shot at generating the same load on our application that existed when the problem occurred. Hopefully, the moons have aligned, and your fingers are crossed…

Now let’s fast-forward to 2008. Your Indian team is using your recording system. You arrive in the morning, logon to your defect tracking system, load the recording of an issue they found, and press ‘play’.

This time, every event that affected your application in that complex environment, including output from your authentication, LDAP, caching and e-commerce servers, has all been recorded and stored. Even the database and its dataset are no longer required. Most importantly, the end-user traffic that ultimately triggered the problem to occur has been recorded as well. All of these elements are perfectly reproduced allowing you to focus on the most important thing: What went wrong.

Anyone who has been involved in software development can relate to the age-old conundrum of trying to reproduce an issue that simply doesn’t appear to exist. At least not on your machine. Too many sleepless nights have been wasted chasing down phantom bugs. It’s time for the madness to stop.

The problems we’re facing are only getting more and more complex as new technologies are brought to market. This new software paradigm is here to stay. Luckily, I believe new technologies such as record and replay will help control the chaos.

Where are the clouds?

Why isn't everything in the cloud these days? Where is the promised land of SaaS?

It feels like SaaS has been a story waiting to happen since around 1999. The network is the computer, thin clients with fat pipes, scaling servers, clustering, virtualization... Seems like all the pieces are in place, the cloud is up there. But still there are only a handful of winners that have really figured out the SaaS model, and a sea of also rans that got run over along the way.

Why?

Well, some of them were just plain bad ideas. Irrational exuberance and all that. But I think there is another big factor at play here. When you have these giant, sea change moments in the way that software is designed, built, shipped and supported, not to mention sold, you'd better have the tools and technology to support you along the way, otherwise it's not going to be easy! In fact, it's going to be hard. Really hard.

I could point you to a room full of ex-CEO's who will attest to this fact. Whipping up your latest Web 2.0 mashup and putting it online is usually about 3% of the challenge. What happens when people actually start using it? Here's where the rubber meets the road.

One of the essential elements of success is getting a solid, scalable application online and running smoothly and securely. But there just hasn't been a lot of innovation here.

Being able to quickly identify, respond to and resolve issues in a SaaS application is critical, because if one server has a bad day, it's not one customer that feels pain, it's hundreds or thousands. And that's bad. SaaS acts like a big hairy amplifier on any defect or scalability issue that might be lurking in your app.

Technologies like Introscope, Patrol, Vantage, Snort and my company Replay are starting to address the needs, but our customers are still pioneering and forging the landscape as they increasingly feel the pains of this new software paradigm we find ourselves in.

So great job, VMware, Amazon and Linus for getting us to this point where we finally can explore the dream of The Cloud. Now it's up to companies like us to make SaaS applications manageable, cost effective, and safe as we keep things running up there 24/7 with less 'unscheduled maintenance'!

TiVo : Why isn't it everywhere?

I love TiVo.

There is no question that I am horribly guilty of being one of those annoying TiVo-heads who goes around expounding upon how life-changing their technology is to anyone who will listen. In fact, truth be told, TiVo was the main inspiration behind my company four years ago.

The question then, and still to this day for me is: Why isn't TiVo everywhere in my life?

The obvious one that comes up all the time is radio. How many times have you been in the car listening to a great tune and wondered, "What was that lyric?", then found yourself looking for the remote to jump back 7 seconds (boop-boop!). Why this hasn't materialized isn't entirely clear to me considering all it would require is a few hundred KB of RAM in the radio to keep the rolling window. Heck, most car decks already do this when they buffer up the last 30 seconds of a CD track for when you hit that bump and the CD skips. Perhaps there are issues around copyrights that I am blissfully unaware of...

But the real question I'm wondering about is TiVo for your life. It's a fact that digital cameras have exploded in power and capability, and digitial media capacity has more than kept up. For $250 bones today you can get yourself a 7 megapixel camera with a 4 GB memory card. It will take a couple hours of good quality video, and it probably fits in your shirt pocket.

So how far away are we from a device that can record your life?

Something interesting just happen to you? Boop-boop! Let's see that again. Heck, save it for later. Forget about all this fumbling around with your camera annoying your friends. I'm talking about Justin.tv for the common man, but without the backpack, and the live-streaming video to the world, and the annoying camera on the baseball cap. ;)

This should be very possible with today's technology. A simple pair of spectacles with a decent CCD in the frame, some bluetooth technology, the PLR (Personal Life Recorder) sitting in your pocket storing the audio/video stream, and you're done. As you travel through the myriad of times and places that make up your so-called life, your experiences, your memories, and those precious moments, all there when you want them.

Rutger Hauer summed up the problem we're solving quite nicely in my favourite film of all time. At the end of it all, he laments, "...all those moments... lost in time... like tears... in rain. Time to die.".

Hang on Roy, help should be on the way shortly... Sadly, his four years were up.