I’m surprised when people think Agile is perfect and if there are any shortcomings, its not the problem with Agile, instead, it is the person/team/org’s understanding or implementation issue. Some where along the way, the aspect that “We are uncovering better ways of developing software” was lost and agile became this static, rule-based prescriptive and dogmatic cargo-cult thing.

IMHO Agile has made a significant difference (some of it a a placebo effect as well) to the software industry however it has some serious limitations when you try to apply in beyond simple CRUD based applications:

• Agile works well in linear or organised complexity domains where the problem is fairly well understood (static) and we need to find/evolve the solution iteratively and incrementally. But in domains, where:
  • the problem itself is unknown or constantly shifting,
  • the problem has a dozen or so variables that interact non-linearly. For ex:
    • in life sciences where we’re trying to understanding ageing/growth
    • in anti-terrorism where we have to deal with a crisis situation
    • when simulating chaotic systems like Indian traffic system
    • trying to predict outcomes in systems with distributed intelligence

applying agile values, principles and practices is not the best approach in these cases. We often find ourselves lacking the right kind of thought process and tools to be able to manage such project.

• Event though the Agile luminaries claim that Agile treats software development as a Complex Adaptive System, they actually try to apply techniques that work in a Complicated Domain.
  • For example, given a problem, we analyse the problem, figure our a best-bet solution (set of practices), apply the solution, see what happens, do a retrospective and tweak the solution (inspect and adapt). This is how you work in a complicated domain. In a complex adaptive domain, we try a few independent safe-fail experiments to solve the problem, but most importantly we do all those experiments in parallel (set-based development approach), so we can really amplify good patterns and dampen bad patterns. We manage the emergence of beneficial patterns with attractors within boundaries. Its like running 5 parallel A/B tests and then coming up with a solution.
• Agile folks seems to claim that distributed development is hard and you should always prefer collocation. But what about thousands of successful open source projects built by people who’ve never met each other? We seem to be missing something here. Open source project model seems to be way better at motivating people by giving them autonomy, master and sense of purpose. Most agile projects are not able to match this.
• Today velocity and bunch of other vanity metric is killing agility. There seems to be so much focus on output and very little focus on outcome and learning. Agile has very little to offer in the space of customer development, business model validation, User experience and other important aspects required for a successful product launch. Which is what Lean-Startup movement is trying to address. This is clearly a limitation of Agile methods.

What’s your take?

Over the last year, I’ve been helping (part-time) Freeset build their ecommerce website. David Hussman introduced me to folks from Freeset.

Following is a list of random topics (most of them are Agile/XP practices) about this project:

• Project Inception: We started off with a couple of meetings with folks from Freeset to understand their needs. David quickly created an initial vision document with User Personas and their use cases (about 2 page long on Google Docs). Naomi and John from Freeset, quickly created some screen mock-ups in Photoshop to show user interaction. I don’t think we spent more than a week on all of this. This helped us get started.
• Technology Choice: When we started we had to decide what platform are we going to use to build the site. We had to choose between customer site using Rails v/s using CMS. I think David was leaning towards RoR. I talked to folks at Directi (Sandeep, Jinesh, Latesh, etc) and we thought instead of building a custom website from scratch, we should use a CMS. After a bit of research, we settled on CMS Made Simple, for the following reasons
  • We needed different templates for different pages on the site.
  • PHP: Easiest to set up a PHP site with MySQL on any Shared Host Service Provider
• Planning: We started off with an hour long, bi-weekly planning meetings (conf calls on Skype) on every Saturday morning (India time). We had a massively distributed team. John was in New Zealand. David and Deborah (from BestBuy) were in US. Kerry was in UK for a short while. Naomi, Kelsea and other were in Kolkatta and I was based out of Mumbai. Because of the time zone difference and because we’re all working on this part time, the whole bi-weekly planning meeting felt awkward and heavy weight. So after about 3 such meetings we abandoned it. We created a spreadsheet on Google Docs, added all the items that had high priority and started signing up for tasks. Whenever anyone updated an item on the sheet, everyone would be notified about the change.
• User Stories: We started off with User Persona and Stories, but soon we just fell back to simple tasks on a shared spreadsheet. We had quite a few user related tasks, but just one liner in the spread sheet was more than sufficient. We used this spreadsheet as a sudo-backlog. (by no means we had the rigor to try and build a proper backlog).
• Short Releases: We (were) only working on production environment. Every change made by a developer was immediately live. Only recently we created a development environment (replica of production), on which we do all our development. (I asked John from Freeset, if this change helped him, he had mixed feelings. Recently he did a large website restructuring (added some new section and moved some pages around), and he found the development environment useful for that. But for other things, when he wants to make some small changes, he finds it an over kill to make changes to dev and then sync it up with production. There are also things like news, which makes sense to do on the production server. Now he has to do in both places). So I’m thinking may be, we move back to just production environment and then create a prod on demand if we are plan to make big changes.
• Testing: Original we had plans of at least recording or scripting some Selenium tests to make sure the site is behaving the way we expected it to. This kind of took a back seat and never really became an issue. Recently we had a slight set back when we moved a whole bunch of pages around and their link from other parts of the site were broken. Other than that, so far, its just been fine.
• Evolutionary Design: Always believed in and continue to believe in “Do the Simplest, Dumbest, thing that could Possibly work“. Since we started, the project had taken interesting turns, we used quite a lot of different JavaScript libraries, hacked a bit of PHP code here and there. All of this is evolving and is working fine.
• Usability: We still have lots of usability and optimization issues on our site. Since we don’t have an expert with us and we can’t afford one, we are doing the best we can with what we have on hand. We are hoping we’ll find a volunteer some day soon to help us on this front.
• Versioning: We explored various options for versioning, but as of today we don’t have any repository under which we version our site (content and code). This is a drawback of using an online CMS. Having said that so far (been over a year), we did not really find the need for versioning. As of now we have 4 people working on this site and it just seems to work fine. Reminds me of YAGNI. (May be in future when we have more collaborators, we might need this).
• Continuous Integration: With out Versioning and Testing, CI is out of question.
• Automated Deployment: Until recently we only had one server (production) so there was no need for deployment. Since now we have a dev and a prod environment, Devdas and I quickly hacked a simple shell scrip (with mysqldump & rsync) that does automated deployment. It can’t get simpler than this.
• Hosting: We talked about hosting the site on its own slice v/s using an existing shared host account. We could always move the site to another location when our existing, cheap hosting option will not suit our needs. So as of today, I’m hosting the site under one of my shared host account.
• Rich Media Content: We questioned serving & hosting rich media content like videos from our site or using YouTube to host them. We went with YouTube for the following reasons
  • We wanted to redirect any possible traffic to other sites which are more tuned to catering high bandwidth content
  • We wanted to use YouTube’s existing customer base to attract traffic to our site
  • Since we knew we’ll be moving to another hosting service, we did not want to keep all those videos on the server which then will have to be moved to the new server
• Customer Feedback: So far we have received great feedback from users of this site. We’ve also seen a huge growth in traffic to our site. Currently hovering around 1500 hits per day. Other than getting feedback from users. We also look at Google Analytics to see how users are responding to changes we’ve made and so on.
• We don’t really have/need a System Metaphor and we are not paying as much attention to refactoring. We have some light conventions but we don’t really have any coding standards. Nor do we have the luxury to pair program.
• Distributed/Virtual Team: Since all of us are distributed and traveling, we don’t really have the concept of site. Forget on-site customer or product owner.
• Since all of this is voluntary work, Sustainable pace takes a very different meaning. Sometimes what we do is not sustainable, but that’s the need of the hour. However all of us really like and want to work on this project. We have a sense of ownership. (collective ownership)
• We’ve never really sat down and done a retrospective. May be once in a while we ask a couple of questions regarding how something were going.

Overall, I’ve been extremely happy with the choices we’ve made. I’m not suggesting every project should be run this way. I’m trying to highlight an example of what being agile really means.

It appears to me that the Agile Community is falling behind the innovation curve. At conferences, user groups, mailing list, etc, we see the same old same old stuff (may be I’m missing something). So where is the real innovation happening? What space should I be watching?

These were the questions I posed to the group @ the SDTConf 2009. Later, during our discussion at the conference we tried answering them. After a wonderful discussion we come up with some suggestions:

• Web 2.0
  • Highly Scalability, Performance and Operations space
  • No SQL
  • Continuous Deployment and Monitoring space – Owen’s Slides are a good starting point
  • Watch out for conferences like O’Reilly’s Velocity
• Alternative Language (non-mainstream languages) space. Lot of interesting experiments going on in
  • Dynamic language space
  • Functional language space
  • Hybrid language space
• Domain Specific Language space
• Could Computing, Parallel Computing (Grid Computing), Virtualization space
• Code Harvesting Space – Check out Test Driven Code Search and Code Genie as a starting point
• Complex Adaptive Systems and its implication on our social interactions space. Dave Snowden’s work is a good starting point
• eLearning and visual assessments (feedback) of a programming session. Check out Visualizing Proficiency
• Polyglot Programming space
• With Google Apps, people are able to build 100s of Apps each month and get instant feedback on their ideas
• Social Networking and Second Life space
• Conference: Lot of interesting experiments are been conducted in the conference space. Conferences have evolved to something very different from before.
• Distributed Development and Remote Pairing space

If you would like to contribute to this list, please add your point on the SDTConf Wiki.

Lot of organizations in India are considering Agile.

• Agile Adoption in Services Companies is mostly driven by customers and competition.
• For product companies its all about
  • Improving productivity,
  • Reducing time to market and
  • Better quality.

While these companies are adopting agile, they have lots of concerns and questions. One recurring theme I’ve seen, is all these companies are really really afraid of failure. IMHO, this fear leads to a very dysfunctional agile adoption. Teams pick what is easy to do and what fits into their existing model, in the name of reducing risk, call it fail-safe 😉 . With this approach individuals and companies fail to see the real benefit of Agile.

This issue is compounded by the fact that a lot of companies are selling substandard certification, tools and consulting.

Also most Indian companies work in a distributed model. Distributed development is another challenge teams are facing and they don’t really see how to fit Agile in their distributed context.

All these points put together makes it very difficult for team to succeed.

These are my biggest concerns today with Agile adoption in India. Does this ring a bell? Can you share your thoughts?

I would be really happy if you want to share your agile adoption story with the agile software community of India. If you can address some/all of the following questions it would be great:

• Why did your company decide to consider Agile?
• How did you build a business case for going Agile? Can you share some artifacts from this original business case?
• Who all was driving this change?
• What measures were taken to set realistic expectations with both Management and Development teams?
• Did you start with a Pilot project? If yes,
  • What were the criteria for picking a pilot project?
  • What was required to get the pilot project up and running?
  • Was the pilot project a success? What were your success criteria?
  • Post the pilot project what did your organization do?
  • How did you roll out Agile to rest of your organization based on your learning from the Pilot project?
• From when you started to now, can you give us important milestones and some artifacts with regards to the same?
• Was there a key turning point in this journey? If yes, what was it?
• Looking back, what mistakes you think could/should have been avoided? And what mistakes you think were worth committing?
• Where is your journey with Agile heading? What are your future plans?
• What does a typical day in life of each team member (developer, tester, manager, etc) look? Any pictures/artifacts you can share?
• What impact did Agile have on your organization structure?
• What mechanisms did you use to guide your teams if they were going down the right direction?
• After having gone through it, do you think it was worth it?
• Any thing else you would like to share?

Today Distributed Development is unavoidable and is the need of the hour.

My belief is that if you have the “right” set of people, motivated and happy on your team, it will be a huge contribution to the success of the project. But the problem is most often “right” people are distributed all over the world and not located in the same city. A lot of organizations have tried to move people to the same city so that they can be collocated. While this might work to some extent, this model cannot scale and is not sustainable. All of us would like to settle down in the location of our choice. We don’t want our work to dictate where we live and what lifestyle we choose. For example, I like to live in India close to friends and family. So far I have traveled quite a bit, but once my daughter starts going to school it would be very difficult for me to relocate to another city.

Also if you consider a lot of leading companies they are loosing a huge amount of people, because most of those employees don’t want to travel/commute any more and want to settle down in one place. While tele-porting is a few centuries away, employees have to live with stressful commute everyday. Not only this is a huge wastage of time, it also affects your productivity and makes your family life very stressful. Currently most of the successful companies have offices in multiple cities. One of the main drivers for having multiple office in different cities is to attract local talent. If your company does not plan to have a office in location X, guess what, your competitors will open an office and you’ll loose some of the smartest folks you could have other wise hired. And when you have multiple office, guess what, people in different offices have to work together. This is another reason why distributed development is becoming more and more important.

In Agile we talk about having development and business teams together. It is very critical to have constant business involvement. But what do you do when your Business is distributed? Because of the global economy most businesses have global presence.  Also each region might have slightly different business rules or market. Hence it is very important to have development teams close to local businesses and distributed. This again leads to distributed development.

Another point to be aware of is, if we consider what industrialization has done by building power centers in industrial cities leaving the rest of the country really backward. We don’t really want IT to do the same thing to the country’s economy.

If you consider these practical issue, collocated team model turns out to be unrealistic in the long run. The chances are you will have to compromise on the quality of people or have the team members travel a lot if you want all of them to be collocated. The alternative is to build a model where the team members can be distributed across space and time.

Unfortunately so far, we’ve not had great results with distributed development. A lot of people have published their experiences trying to effectively work in distributed teams. For some it has worked or some it has not. But for most of them its not as effective as collocated teams.

What has worked best of me so far is to start off a team by having them collocated for a couple of week to a month. Then created cross-functional self sufficient teams in each distributed location. Once this model seems to work, then try to fully distribute your team with team members spread all the world. This turns out to be a long and painful process. But this is the best I know of today. Check out my presentation on Distributed Agile which I gave at the IV Agile Gathering in Ukraine.

Next year I plan to kick off a few experimental distributed projects to try out different techniques to get Distributed Development work as effectively or may be even better than collocated teams.