Why do we see new process or methodology or movement every 10 years or so?

Using the Adaptive Change Cycle we can easily explain the rationale behind it.

Often people tell you that “You should think about the long-term.”

Sometimes people tell you, “Forget long-term, its too vague, but you should at least think beyond the near-term.”

Really?

Unfortunately, part of my brain (prefrontal cortex), which can see and analyze the future, has failed to develop compared to the other smart beings.

At times, I try to fool myself saying I can anticipate the future, but usually when I get there (future) its quite different. I realize that the way I think about the future is fundamentally flawed. I take the present and fill it with random guesses about something that might happen. But I always miss things that I’m not aware of or not exposed to.

In today’s world, when there are a lot of new ideas/stuff going around us, I’m amazed how others can project themselves into the future and plan their long-terms?

Imagine a tech-company planning their long-term plan, 5-years ago, when there were no iPads/tablets. They all must have guessed a tablet revolution and accounted that in their long-term plans. Or even if they did not, it would have been easy for them to embrace it right?

You could argue that the tablet revolution is a one-off phenomenon or an outlier. Generally things around here are very predictable and we can plan our long-term without an issue. Global economics, stability of government, rules and regulations, emergence of new technologies, new markets, movement of people, changes in their aspirations, environmental issues, none of these impact us in any way.

Good for you! Unfortunately I don’t live in a world like that (or at least don’t fool myself thinking about the world that way.)

By now, you must be aware that we live in a complex adaptive world and we humans ourselves are complex adaptive system. In complex adaptive system, the causality is retrospectively coherent. .i.e. hindsight does not lead to foresight.

10 years ago, when I first read about YAGNI and DTSTTCPW, I thought that was profound. It was against the common wisdom of how people designed software. Software development has come a long way since then. But is XP the best way to build software? Don’t know. Question is, people who used these principles did they build great systems? Answer is: Yes, quite a few of them.

In essence, I think one should certainly think about the future, make reasonable (quick) guesses and move on. Key is to always keep an open mind and listen to subtle changes around us.

One cannot solely rely on their “intuition” about long term. Arguing on things you’ve all only guessed seems like a huge waste of time and effort. Remember there is a point of diminishing returns and more time you spend thinking long-term, less your chances are of getting it right.

I’m a strong believer of “Action Precedes Clarity!”

Update: In response to many comments: When the Future is Uncertain, How Important is A Long-Term Plan?

It appears to me that the Agile Community is falling behind the innovation curve. At conferences, user groups, mailing list, etc, we see the same old same old stuff (may be I’m missing something). So where is the real innovation happening? What space should I be watching?

These were the questions I posed to the group @ the SDTConf 2009. Later, during our discussion at the conference we tried answering them. After a wonderful discussion we come up with some suggestions:

• Web 2.0
  • Highly Scalability, Performance and Operations space
  • No SQL
  • Continuous Deployment and Monitoring space – Owen’s Slides are a good starting point
  • Watch out for conferences like O’Reilly’s Velocity
• Alternative Language (non-mainstream languages) space. Lot of interesting experiments going on in
  • Dynamic language space
  • Functional language space
  • Hybrid language space
• Domain Specific Language space
• Could Computing, Parallel Computing (Grid Computing), Virtualization space
• Code Harvesting Space – Check out Test Driven Code Search and Code Genie as a starting point
• Complex Adaptive Systems and its implication on our social interactions space. Dave Snowden’s work is a good starting point
• eLearning and visual assessments (feedback) of a programming session. Check out Visualizing Proficiency
• Polyglot Programming space
• With Google Apps, people are able to build 100s of Apps each month and get instant feedback on their ideas
• Social Networking and Second Life space
• Conference: Lot of interesting experiments are been conducted in the conference space. Conferences have evolved to something very different from before.
• Distributed Development and Remote Pairing space

If you would like to contribute to this list, please add your point on the SDTConf Wiki.

Programming is “the action or process of writing computer programs”.

Programming by definition encompasses analysis, design , coding, testing, debugging, profiling and a whole lot of other activities. Beware Coding is NOT Programming. Depending on which school of thought you belong to, you will define the relationship and boundaries between these various activities.

For Example:

• In a waterfall world, each activity is a phase and you want a clear sign-off between each phases. Also these phases are sequential by nature with very limited or no feedback. Hence you are expected to have the full design in place before you can code. Else, what do you code?
• In RUP (so-called Iterative and Incremental model) even though it follows a spiral model with some feedback cycle every 3 months or so, one is expected to have the overall architecture of the project and a documented design (in UML notation) of the subset of use cases planned for the current spiral ready before the construction (coding) phase.
• In the unconventional model (where we don’t have process & tool servants and team members can do what they think is most appropriate in the given context), we fail to understand these sequential, rigid processes. We have burnt our fingers way too many times trying to retrofit ourselves into this sequential, well-defined process boundaries guarded by process police. So we have given up the hope that we’ll ever be as smart as the rest of the “coding community” and have chosen a different route.

So how do we design systems then?

• Some of us start with a test (not all, but just one) to understand/clarify what we are trying to build.
• While others might write some prototype code (read it as throw away code) to understand what needs to be build.
• Some teams also start by building a paper prototype (low-fidelity prototype) of what they plan to build and jump straight to the keyboard to validate their thought process (at least once very few hours).
• Yet some others use plain old index cards to model the system and start writing a test to put their thoughts in an assertive medium.

This is just the tip of the iceberg. There are a million ways people program systems. We seem to use a lot of these practices in conjunction (because they are not mutually exclusive practices and can actually be done in parallel).

People who are successful in this model have recognized that they are dealing with a complex adaptive system (CAS) and not a complicated system, where you can define rigid boundaries and be successful. In a CAS, there are multiple ways to do something and if someone makes a claim that you always have to do X before Y, we can sense the desire of putting rigid constraints which by nature are fragile. This is the same reason why there is no such thing called Best Practices in our dictionary. Instead we keep an eye on emerging patterns. If we want to see a particular pattern impact the system, we introduce attractors. But if we don’t want a pattern to impact our system we disrupt that pattern. (rip-off from Dave Snowden, creator of the Cynefin model and leading personality in Knowledge Management Community)

The open source community in general, is yet another classic example which fits into the unconventional category. I’ve never been on an open source project where we had a design phase. People live and breath evolutionary design. At best you might have a simple wiki defining some guidelines.

Anyway, I’m not saying that upfront design is bad. All I’m saying is, don’t tell me that one always has to design first. In CAS, you tend to “Probe-Sense-Respond” and not “Analyze-Respond”. In software development “Action precedes Clarity”, almost always.