The MurkyGrey blog

Millions of people have already heard this story: Dave Carroll, a talented musician, flew with United Airlines. He checked in his guitar as luggage and then watched helplessly as United baggage handlers tossed his precious instrument around. Reunited with his guitar upon landing he discovered that as he feared, it had been broken. Dave tried in vain to get United to cover the cost of repair, but after a few months he received a final negative response from the airline stating that since he did not report his loss within 24 hours of collecting his luggage he is ineligible for any compensation. In response Dave wrote a song titled “United Breaks Guitars” and published it on YouTube:

The song became an instant hit on YouTube and on the iTunes music store; it quickly proved to be a PR nightmare for United. In addition to the sweet taste of revenge, it landed Dave a nice chunk of change from iTunes sales, two new guitars from the manufacturer, a serious boost to his popularity and a second career as a sought-after public speaker. It’s a great song: well-written, nicely performed, funny and catchy. Give it a click every time you feel like poking fun at United, the airline industry at large or poor Ms. Irlweg, but if we actually want to learn something from Dave’s guitar’s misfortune, let’s try to unravel the secret to the song’s popularity and the underpinning of the guitar incident.

A mountain of frustration

The millions of listeners who fueled the song’s popularity don’t just enjoy a catchy tune, they also relate to the songwriter’s pain and frustration on a very personal level. It is more than a song — it is a battle hymn for the hoards of frustrated airline passengers and for anyone who ever struggled with an uncaring and indifferent big-company “service” organization. Indeed, BoingBoing dubbed it “the complaint anthem”. Admittedly, airlines are the worst: you trust them with your life and your property while they seal you up in a metal tube and instruct you to sit down quietly while they abuse your trust. There is a lot of frustration built up there, and the song easily taps into it.

Who’s at fault?

Pointing fingers is fun, who should we blame for the guitar’s demise? Obviously the baggage handlers who tossed the guitar bear the most direct responsibility, they should have been much more careful. The trouble with accidents though, is that they tend to happen no matter how careful we are. Sure, baggage handlers can significantly reduce the rate at which baggage is damaged, but ensuring the complete safety of each and every checked item is entirely impractical.
When accidents happen, customer service should pick up the pieces. Three Chicago service agents (the ones “who showed complete indifference”) were perfectly positioned to avert the catastrophe and they dropped the ball gloriously. These service agents could have been more sympathetic, they could have asked for the claim check, tracked the item and told the concerned passenger where his luggage was, but most importantly they could have explained the process for reporting damaged luggage and emphasized that it must be done within 24 hours of landing. Just explaining the process would have prevented the PR disaster. Dave probably would not have become a fan of the airline that broke his beloved guitar, but he would have been compensated for the damage and likely would not have written “United Breaks Guitars”.

Who’s not at fault?

Ironically, the one person that Dave chose to name and shame in front of millions of YouTube viewers is hardly at fault. Poor Ms. Irlweg, the bearer of bad news, was wrongfully singled out. You see, second only to big-company customers in their helplessness are big-company employees, often rendered equally powerless by rules and procedures. By the time Ms. Irlweg received the claim the only thing she could do was note that the claim was not submitted within the 24-hour window and reject it.
This notion is not lost on Dave Carrol himself or on United. Dave all but acknowledged Ms. Irlweg’s innocence in his sequel song and United spokesperson Robin Urbanski stated that one of the steps they took following the incident was to “provide agents with a better way to escalate and respond to special situations”.

It’s all about process

Guitar tossing and careless service agents aside, what made this incident truly torturous is rules and procedures; a passenger who was not made aware of the rules in time and an employee who was rendered powerless by process.

And a lesson for acquired startups

Every now and then, a perfectly good startup turns around and gets itself acquired by an industry giant. When this happens, you (a perfectly happy startup employee) might wake up the next morning to find out that you have become… Ms. Irlweg; you try to get things done but find yourself suddenly bound by rules and procedures to the point of complete helplessness. It’s frustrating, and to make things worse all around you long-time employees of the industry giant seem to get their way with ease by invoking just the right rule every time.
Simply put, if you want to be successful in your new big-corporate job, you are going to have to take some time and learn the rules. You might want to:

• Start by finding reliable and current information about rules and procedures. HR should be able to help you with that, if only you could figure out the steps to reach the rights person in HR…
• Take the time to familiarize yourself with the rules (this may take a while).
• As an interim solution, align yourself with someone who’s been around for a while (if you can find such an ally).

Do you have a hair-raising tale of horrific process or terrifying big-company acquisitions? Leave a comment

As I’m sure you’ve heard, there are far-away lands where the best and brightest youths flock to science and technology schools. They emerge a few years later in hoards of brilliant and eager software developers. Why not put them to work for you? They are smart, talented, easy to find and hire and they work for very reasonable pay. The big guys have been doing it for years and recently, smaller and smaller startups are incorporating offshore development into their plans.
As a development manager I have, over the years, utilized development teams in India, China and Russia. Here are some lethal mistakes to avoid:

Downplaying the culture gap

Communicating with your offshore developers may seem easy; in all likelihood they speak decent English. A friend of mine, a European microbiologist who hosts in his research lab science grad students from all over the world, commented that in recent years the level of English spoken by young foreigners has improved dramatically. I think he’s right, and I believe that what we’re seeing is a generation of young people, now in their twenties, who speak good English because they grew up with Internet access and were exposed to a lot of English content.
But beware: culture is much broader than language and communication has a lot more to it than words alone. Intonations, gestures, idioms, slang, vocabulary, unspoken assumptions, acceptable and unacceptable behavior vary wildly across the globe and these variations can lead to catastrophic failures in communication. The fact that good English is spoken on both sides might lull you into thinking that you understand each other when in fact you do not.

To avoid this mistake:

Encourage offshore participants to explain back to you their understanding of a conversation (especially if the conversation is one-sided in nature, e.g. you are assigning tasks or briefing them on a new initiative).
Document and share meeting minutes; writing gets the non-verbals out of the way.
Mind the gap; keep reminding yourself of the cultural differences that you must bridge.

Managing offshore developers remotely

The whole point in offshoring is to save money, right? Excellent offshore developers are sometimes easy to come by, but hiring good managers is never easy and not nearly as cheap. You might be tempted to skip hiring a manager altogether, tap into the startup spirit and “do it yourself” (that is: manage your offshore developers yourself, remotely). My advice: don’t!
Managing people is hard, managing developers is harder. A software development team’s manager needs to be there with the developers, meet with each one on a regular basis, get a first-hand account of their activity, successes and challenges. Unless you are an expatriate offshoring back home you already have a cultural gap standing between you and your remote team. Add to that the time zone differences and the loss of non-verbal cues that come with long-distance telecommunication and you have yourself a recipe for failure.

To avoid this mistake:

Hire a local manager for your offshore team. If the team is small (less than 5-10 developers) you can hire a “working manager”: a senior developer that also happens to have management skills. If your team is larger than that, hire a full-time manager. In any case you must have a single point of contact and accountability on the ground, close to your developers and on the other side of the geographical and cultural gaps.

Skimping on travel

International travel is expensive, time-consuming and uncomfortable. With video conferencing and web meetings readily available you feel like you are virtually there, but “virtually” does not cut it. If you don’t travel to visit your offshore team you might never interact directly with some team members (usually the more junior ones), you will never get a feel for the atmosphere in their office, you might never know what is working well and what isn’t. If you never visit your offshore team you are relying completely and blindly on your remote manager, creating a dangerous single point of failure in your organization.

To avoid this mistake:

Every six months or so clear some time on your calendar, make sure your passport and visas are in order, get a plane ticket and pack your bags.
To make things a bit easier, alternate between traveling to visit your offshore team and inviting your offshore counterpart to visit you. This will give your remote manager an opportunity to interact with other members of your local team. Travel can be a burden, but in some cases your remote manager may see these trips as an added benefit.

Failing to protect Intellectual Property

Coders handle one of your company’s most valuable assets: the code. At home you are protected from code theft by non-disclosure and confidentiality agreements, ethical norms and the legal system. Personally, I never encountered a developer that stole source code and sold it to a competitor. Both the developers and your competitors have too much to lose in terms of reputation and legal liability (both civil and criminal).
In a foreign country, do you know what the social norms are? Will the social norms protect you? Can you enforce a non-disclosure or confidentiality agreement? Can you count on the legal system? If you fail to answer these questions before trusting your offshore developers with critical parts of your code you might find yourself up the creek without a paddle.

To avoid this mistake:

Get sound advice, specific to the location of your offshore team. Do not share critical portions of your code with developers in locations with a questionable confidentiality record.

Placing marketing functions away from the target market

This mistake is common among foreign companies that start out with a development center in a low-cost locale and develop a product for the US market. Once product development is complete they would open a sales office in the US and leave product development,QA, tech writing and other functions back home where things are more reasonably priced. It hurts to spend big bucks on a US sales force when you have grown accustomed to much lower costs, but you know there is no escape from it.
But what about marketing functions? Do you really have to hire super expensive US-based product managers? Lead generation is all done via telecommunication, it’s not quite the same as sales, is it? And what about collateral writing, does it really have to be done at $80 and more per hour when your brightest developers work for less than a third of that?
The answer is yes, you have to spend the money and keep your marketing functions close to the target market. If you don’t, you are wasting your money on your US-based sales team. Instead of results, they will deliver a constant flow of complaints about the quality of leads, marketing materials and product features.

To avoid this mistake:

Once you commit to the US market, hire a local sales and marketing team, don’t blunt your own competitive edge with a marketing team that is not in touch with its own market.

Need Help?

Do you need help defining, building, marketing or selling your software product? drop me a note, I’d be happy to help.

I recently commented on a blog post that started with a tale of a lousy supervisor doing a truly horrific job of relaying negative feedback to a subordinate. It got me thinking about some of the miserable bosses I’ve had over the years. Not wanting to linger too much in negativity, I tried to conjure up some of the better managers I reported to. A good manager is hard to find, but when you have one, you often get to learn valuable lessons. Here are some things I have learned from one good boss.

Maintain a blame-free work environment

Finger-pointing is a toxic, counter-productive behavior, but there is more to it; in a software startup it is essential to move forward at a fast and efficient pace. A no-mistakes pace is simply too slow. To survive, we have to move at a speed that guarantees a certain rate of error. We must accept the fact that mistakes will be made and corrected on the fly, simply because a pace that yields no mistakes will not bring us to takeoff before the end of the runway. I once spent a few weeks at a client site in a foreign country where a culture of “no mistakes” prevails; as you may expect, everything was gold-plated to death. Time was regularly wasted on unnecessarily perfect performances, and on cover-ups when things didn’t go according to plan.
“No blame” does not mean “no accountability”. Anyone who’s ever worked for me has heard me say: “It’s OK to make a mistake and you will not be judged for it. Making a mistake and not learning from it is a different story”.

Don’t confuse “urgent” with “important”

You plan your day, week and month. You focus your efforts in a calculated effort to achieve very specific goals. Then someone rushes in screaming that the sky is falling and all progress is put on hold until the oh-so-urgent issue is resolved. There’s a hero’s aura about riding to the rescue and saving the day, but when the day is done, you are still a day (or a week, or a month) behind your schedule. The fires you are putting out may be real, or they may be artificial emergencies conceived to manipulate your priority list. Don’t let the moment’s glory distract you from executing your plan for too long. It may not be as urgent, but it is far more important.

Manage your personal productivity

Developers’ productivity is a complicated issue. First off, the disparity between individuals is huge. It is not uncommon for a star developer to be ten times as productive as an average, good developer. You will not find this kind of distribution with say, athletes or steel mill workers. On top of that, there are many subjective and even random factors at play; the estimates that we use to measure productivity are always partially subjective, and if Alice took a day to resolve one specific bug, while Bob took a week to resolve another, can you really say that Alice is more productive? Or was she just lucky?
Improving your personal productivity is a great way to get better at what you do, but as a developer you are the only one who can tell how productive your day is. You can expect your manager to measure your productivity over time, but when it comes to your day-to-day personal productivity — you’re on your own. The CEO of a company I worked for used to say: “It’s tough with you developers, even if I see you wandering around aimlessly all day, as long as you have that thinking expression on your faces, you could be working”. So watch your own personal productivity every day. You’re the only one who can do it, and you’re the one to benefit from it.

Have you had a boss worth remembering? Leave a comment.

* In case you were wondering: the mythical boss I am referring to is Alex Shapira. Alex has done wonders for many software companies, he’s an executive consultant now.

Like any other tragic comedy, a software startup is not worth much without a good cast of characters. Here, for your reading pleasure, are some of them:

The Entrepreneur

The founding father (or mother). The whole thing was his idea, he got it started from nothing, it is his baby and he will see it to maturity. Naturally, he feels like he owns the place.
Strength: Relentless faith in a bright future
Weakness: Relentless faith in a bright future
In his defense: The whole thing really was his idea

The sales guy

He’s the one reeling in the big deals. He’s slick, he’s smart, he’s the cool kid. He can demo a product that requires five massive production servers on his laptop in mid-flight from a middle seat. He can drink any prospective customer under the table. He appears utterly unperturbed no matter how desperate the situation (perhaps because he knows that there will be plenty of time for him to cry alone, in his hotel room, when there’s no one around). His sales are the only source of future revenue so naturally, he feels like he owns the place.

Strength: Handles rejection very well
Weakness: His geek-cred leaves a lot to be desired
In his defense: Perhaps more than anyone else he’s the one assuming a significant personal financial risk. His base pay is way lower than it would have been in an established company and his commission hinges on an unproven product.

The coder

Super-geek is here to build the most amazing product the world has ever seen. Can solve any problem given sufficient quantities of time, coffee and Fritos (in warm climates may substitute Mountain Dew for coffee). The coder believes that the company starts and ends with the code, all of the other characters are just milling about and basking in the glory of the incredible product. Without a doubt the work-horse in the house, nobody puts in more hours than the coder and nobody has less of a social life outside of work. The code is the company’s main asset so naturally, the coder owns the place.
Strength: Can make the impossible possible
Weakness: Does not understand people, may need gentle reminders concerning personal hygiene
In the coder’s defense: It really is a complicated job and not very many can do it well

The VC folks

While the rest of us are busy with important tasks the VC folks take care of such minutia as, you know, paying the bills. They put their money where our mouths are and if they don’t see that money back with a nice return they will not be pleased. Unless you attend board meetings they will, for the most part, stay out of your way. Feel free to whisper a “thank you” every time you get a paycheck. Oh and also, they have a piece of paper that says they own the place.
Strength: The ability to turn promises into funds
Weakness: Do not seem to maintain a steady core temperature like other mammals
In their defense: Have we mentioned that they pay the bills?

Marketing

Nobody can really define what their job is, but it is an important job and they do it well. They write collateral and debate the weight and finish of brochure paper. They design, build and staff trade show booths. Sometimes they do market research and even contribute to product design, yet the ambiguity that surrounds their job robs them of any glory. They try to comfort themselves by acting like they own the place.
Strengths: Versatility and flexibility
Weakness: As bright as they may be they can’t shake a bit of an inferiority complex
In their defense: Everyone else has a well-defined job, why can’t Marketing?

Which is your favorite character? Leave a comment.

The difference between amateur and professional software development is, I am told, process. Tedious, endless, creativity-stifling, headache-inducing process. A bunch of rules that developers hate and ignore, while development managers bite their lips and try to enforce.

Software development, like many other technical undertakings, requires a combination of creativity and discipline and it is rarely successful unless both are present. You may choose to only hire highly creative and extremely self-disciplined developers. If you do, please let me know how that is working out for you… I’m not saying that these people do not exist, they do, and they are spectacular, but they are few and far between and if you hold out for them your staffing efforts may take longer than expected. For most of us there is a conflict between being creative and being disciplined and we struggle for balance. So there you are with a team of creative developers who are not crazy about rules.

Early in my programming career I was part of a maintenance team. Our job was to fix bugs in the live product while the product development team worked on future releases. I learned a lot about good and bad code, and about how good code can be turned bad by a series of undocumented changes. When it’s new, good code is coherent and consistent, it is readable and it makes sense. But code that has been around the block and back often becomes a jumble of changes made by different people unaware of each other. This degradation can be mitigated by documenting changes. Back in 1993, the rules stated clearly that each change must be documented in the source file’s header. It’s good process, but it was almost never followed. Enforcement (via code reviews) helped some, but the team did not seem to have the degree of self-discipline required to eliminate the problem completely. Then one day the problem was eliminated, not by better enforcement but by the introduction of a tool: a new source control system was put in place that required a comment on each change and inserted the comment into the source header. People still could write a non-comment such as “NA”, but nobody did. Instead, every change was accompanied by a good, descriptive, useful comment.

Process was followed at last. Changes were documented and code integrity was protected. Best of all, our manager didn’t have to harass us to do it anymore; we felt more independent and he had no complaints about the new order of things either. Code reviews, now free from process enforcing micro-management became positive educational experiences allowing us to learn from each other.

The tool (in this case the new source control system) did two things:

1. It provided people with a ‘just-in-time’ reminder to follow process. When you were done making your change and you wanted to push it to integration testing you had to check it into the source control system. That was part of your job, you had to do it anyway. While you were there, the dialog popped up and asked you to put in a comment and you did. The burden of remembering the rule had been lifted from your shoulders.

2. It took away the tedious, mindless task of taking your comment, adding date and time and pasting it into the right place in the headers of several source files. You were left with the meaningful task of writing a paragraph to describe your change, the rest was taken care of automatically.

I’ve seen the same thing happen with other process-related tasks. Collecting timesheets was always a painful and frustrating task until I had a system in place that sent email reminders with a “report your hours” link in the message. Logging hours was never hard, the painful part was remembering to do it. With an automated reminder that puts you a click away from completing your report, the pain is gone and process is followed smoothly and effortlessly. Similarly, testing and verification procedures were a real PITA until issue tracking systems with built-in workflow management came along.

No matter how lacking in discipline you think we developers are, what we really hate is having to remember tasks that are not directly related to coding. We also dislike mindless repetitive tasks that we know should be automated. Micro-management is bad for us and bad for our managers, give us a tool for every rule and process nirvana will follow.

Do rules drive you crazy? What do you do to cope with process? Leave a comment.

Suppose we arrange the world’s software along an axis. On the one end of the axis is the ultimate shrink-wrapped product; sitting on the shelf, waiting to be picked up by a user who will rip it open, install and use it happily, never knowing (or caring) who built it or how it was built. On the other end is the definitive tailor-made solution; written from scratch to suit the needs of an individual user, commissioned to a master developer of the user’s choosing who must have spent many hours with the user to learn of his pain and design a unique custom-made solution.

It is this axis that yields the renowned divide between product companies and professional services companies. Product companies sell software, and they sell the same software product to as many clients as they can. Professional services companies sell their time and expertise developing or customizing software for specific clients, one project at a time.

Having started my career in product companies I grew accustomed to the many advantages that a product environment provides for software developers. In a product environment you can always bring an exciting technological idea to the table, a discussion of potential markets for the new idea will follow and if it makes business sense you get to build it. Product companies take technological risks and innovate; the rewards associated with bringing a new technology to a broad market justify the risk. If you are ever asked to build something that does not make sense to you (and we developers always know best…), you can put your negotiation skills to the test in a discussion with product management. With a reasonable product manager, you can impact product direction quite nicely.

In a professional services environment, a new business idea usually starts with a new project for a specific client. The project managers rule the roost, and each project is expected to be profitable or it is not worth undertaking. Much care is taken in gathering and refining requirements and design, but this process is centered around the client, not around technology. The entire project is funded by the client and sometimes the client owns the work-product completely. Naturally it is the client that impacts requirements the most, not the developer, not even the project manager. In a professional services environment developers have much less of a say, or at least so I thought.

If all had gone according to plan I would have spent my entire career working for product companies, but then there’s this pesky little thing called “opportunity” that always seems to get in plan’s way. When I made my opportunity-driven move to professional services I wasn’t sure what to expect. What I found surprised me, and eventually helped me become a better product developer than I ever was. The first thing I learned was that professional services companies are not as bad as I thought they were. They learned a long time ago that their profitability soars with repeatable projects. They don’t really take it one project at a time. If a process or a tool can be developed to serve multiple projects they dive right in and build it, in essence developing their own mini-products. The next thing I learned was quite annoying at first. It turns out that clients who fund a custom development effort are not inclined to keep paying for the promise of a product in the making. Instead, they want tangible deliverables for their continued support. Those tangible deliverables are artifacts that we are all know and love: requirement and design documents, project plans, test plans, and other such paperwork.

I was never opposed to reasonable documentation. In fact, I can rightfully be accused of having been a bit too fanatic about internal and external documentation of code, but I couldn’t help but feel that creating the level of documentation clients required was a waste of time. We have a perfectly good design that the developers understand very well, why spend more time polishing it so that some bureaucrat can read it and stamp an invoice? We test thoroughly and efficiently, why publish an executed test plan? We already know that the product is tested. These artifacts do not help us build the product, and they do not help the end-user use it, they are simply irrelevant. I was exhibiting a bias common in the product world: The product is important, anything outside the bounds of the shrink-wrap (or the tarball for that matter) is not important. Despite this minor annoyance, life in professional services was fun and rewarding and some three years later I returned to the warm embrace of a small product company.

Guess what? I’ve come to appreciate those annoying little artifacts. You see, even in a product environment you probably have important stakeholders outside the development team, prime examples are: non-technical founders, sales teams and marketing people. These stakeholders might not sign your invoices, but it is in your best interest to keep them engaged. Just a bit of effort to gather high-level requirements and design and turn it into a human-readable document will go a long way in sharing your vision of a future product or feature. Throw together an executed test plan or other evidence of testing and their confidence in your latest release will skyrocket. My foray into professional services turned out to be a rounding experience. I learned a thing or two about engaging non-technical stakeholders and lost some old biases.

Which environment do you prefer: product or services? What do you do to engage your non-technical stakeholders? Leave a comment.

Related Link: Giff Constable, When it comes to startups, products and services don’t mix