“Release Early, Release Often” is a proven mantra, but what happens when you push this practice to it’s limits? .i.e. deploying latest code changes to the production servers every time a developer checks-in code?

At Industrial Logic, developers are deploying code dozens of times a day, rapidly responding to their customers and reducing their “code inventory”.

This talk will demonstrate our approach, deployment architecture, tools and culture needed for CD and how at Industrial Logic, we gradually got there.

Process/Mechanics

This will be a 60 mins interactive talk with a demo. Also has a small group activity as an icebreaker.

Key takeaway: When we started about 2 years ago, it felt like it was a huge step to achieve CD. Almost a all or nothing. Over the next 6 months we were able to break down the problem and achieve CD in baby steps. I think that approach we took to CD is a key take away from this session.

Talk Outline

1. Context Setting: Need for Continuous Integration (3 mins)
2. Next steps to CI (2 mins)
3. Intro to Continuous Deployment (5 mins)
4. Demo of CD at Freeset (for Content Delivery on Web) (10 mins) – a quick, live walk thru of how the deployment and servers are set up
5. Benefits of CD (5 mins)
6. Demo of CD for Industrial Logic’s eLearning (15 mins) – a detailed walk thru of our evolution and live demo of the steps that take place during our CD process
7. Zero Downtime deployment (10 mins)
8. CD’s Impact on Team Culture (5 mins)
9. Q&A (5 mins)

Target Audience

• CTO
• Architect
• Tech Lead
• Developers
• Operations

Context

Industrial Logic’s eLearning context? number of changes, developers, customers , etc…?

Industrial Logic’s eLearning has rich multi-media interactive content delivered over the web. Our eLearning modules (called Albums) has pictures & text, videos, quizes, programming exercises (labs) in 5 different programming languages, packing system to validate & produce the labs, plugins for different IDEs on different platforms to record programming sessions, analysis engine to score student’s lab work in different languages, commenting system, reporting system to generate different kind of student reports, etc.

We have 2 kinds of changes, eLearning platform changes (requires updating code or configuration) or content changes (either code or any other multi-media changes.) This is managed by 5 distributed contributors.

On an average we’ve seen about 12 check-ins per day.

Our customers are developers, managers and L&D teams from companies like Google, GE Energy, HP, EMC, Philips, and many other fortune 100 companies. Our customers have very high expectations from our side. We have to demonstrate what we preach.

Learning outcomes

• General Architectural considerations for CD
• Tools and Cultural change required to embrace CD
• How to achieve Zero-downtime deploys (including databases)
• How to slice work (stories) such that something is deployable and usable very early on
• How to build different visibility levels such that new/experimental features are only visible to subset of users
• What Delivery tests do
• You should walk away with some good ideas of how your company can practice CD

Slides from Previous Talks

For Freeset, I’ve always been in the quest of Simplest Thing that Could Possibly Work. In a previous post, I explained how we’ve embraced an ultra-light process (call it lean, if you like) to build their e-commerce site.

In that post, I’ve talked about our wish to create a Selenium test suite for regression testing. But it never got high enough on our priority list. (esp. coz we mostly have static content served from a CMS as of now).

While that is something I wanted to tackle, last night, when I was moving Industrial Logic and Industrial XP‘s site over to a new server hardware, I wanted some quick way to test if all the pages were correctly displayed after the move. This was important since we switched from Apache to Nginx. Nginx has slightly different way to handle secure pages, etc.

So I asked on Twitter, if anyone knew of a tool that could compare 2 deployments of the same website. Few people responding saying I could use curl/wget with diff recursively. That seemed like the simplest thing that could work for now. So this morning I wrote a script.

rm -Rf * && mkdir live && cd live && wget -rkp -l5 -q -np -nH http://freesetglobal.com && cd .. && mkdir dev && cd dev && wget -rkp -l5 -q -np -nH http://dev.freesetglobal.com && cd .. && for i in `grep -l dev.freesetglobal.com \`find ./dev -name '*'\`` ; do sed -e 's/dev.freesetglobal.com/freesetglobal.com/g' $i > $i.xx && mv $i.xx $i; done && diff -r -y --suppress-common-lines -w -I '^.*' dev live

I’m planning to use this script to do simple regression test of our Freeset site. We have a live and a dev environment. We make changes on dev and frequently sync it up with live. I’m thinking before we sync up, we can check if we’ve made the correct changes to the intended pages. If some other pages show up in this diff that we did not expect, it’s a good way to catch such issue before the sync.

Note: One could also use diff with -q option, if all they are interested to know is which pages changes. Also note that under Mac, the sed command’s -i (inline edit) option is broken. It simply does not work as explained. If you give sed -i -e …., it ends up creating backup files with -e extension. #fail.

At Freeset, we have two CMS Made Simple instances set up. One for development and another one for live environment. We do all our stuff on dev and then run the following shell script to deploy the latest changes from dev to live. This is a great example of “Do the Simplest Thing that could Possibly Work“.

#!/bin/bash
mysqldump --no-create-db --add-drop-table --skip-add-locks --skip-disable-keys --quick 
-u{db.user.name} -p{db.password} {live.db.name} --tables cms_module_blogs_commentprops 
cms_module_blogs_comments cms_module_blogs_comments_seq
 | mysql -u{db.user.name} -p{db.password} {dev.db.name} && mysqldump --opt -u{db.user.name} 
-p{db.password} {dev.db.name} | mysql -u{db.user.name} -p{db.password} {live.db.name}
 && rsync -rpl --delete --exclude=config.php {dev.deployment.dir}/ {live.deployment.dir}/

Over the last year, I’ve been helping (part-time) Freeset build their ecommerce website. David Hussman introduced me to folks from Freeset.

Following is a list of random topics (most of them are Agile/XP practices) about this project:

• Project Inception: We started off with a couple of meetings with folks from Freeset to understand their needs. David quickly created an initial vision document with User Personas and their use cases (about 2 page long on Google Docs). Naomi and John from Freeset, quickly created some screen mock-ups in Photoshop to show user interaction. I don’t think we spent more than a week on all of this. This helped us get started.
• Technology Choice: When we started we had to decide what platform are we going to use to build the site. We had to choose between customer site using Rails v/s using CMS. I think David was leaning towards RoR. I talked to folks at Directi (Sandeep, Jinesh, Latesh, etc) and we thought instead of building a custom website from scratch, we should use a CMS. After a bit of research, we settled on CMS Made Simple, for the following reasons
  • We needed different templates for different pages on the site.
  • PHP: Easiest to set up a PHP site with MySQL on any Shared Host Service Provider
• Planning: We started off with an hour long, bi-weekly planning meetings (conf calls on Skype) on every Saturday morning (India time). We had a massively distributed team. John was in New Zealand. David and Deborah (from BestBuy) were in US. Kerry was in UK for a short while. Naomi, Kelsea and other were in Kolkatta and I was based out of Mumbai. Because of the time zone difference and because we’re all working on this part time, the whole bi-weekly planning meeting felt awkward and heavy weight. So after about 3 such meetings we abandoned it. We created a spreadsheet on Google Docs, added all the items that had high priority and started signing up for tasks. Whenever anyone updated an item on the sheet, everyone would be notified about the change.
• User Stories: We started off with User Persona and Stories, but soon we just fell back to simple tasks on a shared spreadsheet. We had quite a few user related tasks, but just one liner in the spread sheet was more than sufficient. We used this spreadsheet as a sudo-backlog. (by no means we had the rigor to try and build a proper backlog).
• Short Releases: We (were) only working on production environment. Every change made by a developer was immediately live. Only recently we created a development environment (replica of production), on which we do all our development. (I asked John from Freeset, if this change helped him, he had mixed feelings. Recently he did a large website restructuring (added some new section and moved some pages around), and he found the development environment useful for that. But for other things, when he wants to make some small changes, he finds it an over kill to make changes to dev and then sync it up with production. There are also things like news, which makes sense to do on the production server. Now he has to do in both places). So I’m thinking may be, we move back to just production environment and then create a prod on demand if we are plan to make big changes.
• Testing: Original we had plans of at least recording or scripting some Selenium tests to make sure the site is behaving the way we expected it to. This kind of took a back seat and never really became an issue. Recently we had a slight set back when we moved a whole bunch of pages around and their link from other parts of the site were broken. Other than that, so far, its just been fine.
• Evolutionary Design: Always believed in and continue to believe in “Do the Simplest, Dumbest, thing that could Possibly work“. Since we started, the project had taken interesting turns, we used quite a lot of different JavaScript libraries, hacked a bit of PHP code here and there. All of this is evolving and is working fine.
• Usability: We still have lots of usability and optimization issues on our site. Since we don’t have an expert with us and we can’t afford one, we are doing the best we can with what we have on hand. We are hoping we’ll find a volunteer some day soon to help us on this front.
• Versioning: We explored various options for versioning, but as of today we don’t have any repository under which we version our site (content and code). This is a drawback of using an online CMS. Having said that so far (been over a year), we did not really find the need for versioning. As of now we have 4 people working on this site and it just seems to work fine. Reminds me of YAGNI. (May be in future when we have more collaborators, we might need this).
• Continuous Integration: With out Versioning and Testing, CI is out of question.
• Automated Deployment: Until recently we only had one server (production) so there was no need for deployment. Since now we have a dev and a prod environment, Devdas and I quickly hacked a simple shell scrip (with mysqldump & rsync) that does automated deployment. It can’t get simpler than this.
• Hosting: We talked about hosting the site on its own slice v/s using an existing shared host account. We could always move the site to another location when our existing, cheap hosting option will not suit our needs. So as of today, I’m hosting the site under one of my shared host account.
• Rich Media Content: We questioned serving & hosting rich media content like videos from our site or using YouTube to host them. We went with YouTube for the following reasons
  • We wanted to redirect any possible traffic to other sites which are more tuned to catering high bandwidth content
  • We wanted to use YouTube’s existing customer base to attract traffic to our site
  • Since we knew we’ll be moving to another hosting service, we did not want to keep all those videos on the server which then will have to be moved to the new server
• Customer Feedback: So far we have received great feedback from users of this site. We’ve also seen a huge growth in traffic to our site. Currently hovering around 1500 hits per day. Other than getting feedback from users. We also look at Google Analytics to see how users are responding to changes we’ve made and so on.
• We don’t really have/need a System Metaphor and we are not paying as much attention to refactoring. We have some light conventions but we don’t really have any coding standards. Nor do we have the luxury to pair program.
• Distributed/Virtual Team: Since all of us are distributed and traveling, we don’t really have the concept of site. Forget on-site customer or product owner.
• Since all of this is voluntary work, Sustainable pace takes a very different meaning. Sometimes what we do is not sustainable, but that’s the need of the hour. However all of us really like and want to work on this project. We have a sense of ownership. (collective ownership)
• We’ve never really sat down and done a retrospective. May be once in a while we ask a couple of questions regarding how something were going.

Overall, I’ve been extremely happy with the choices we’ve made. I’m not suggesting every project should be run this way. I’m trying to highlight an example of what being agile really means.

For the last 9 months I’ve been working with an organization called FreeSet. David Hussman introduced me to them and I’m really thankful to David for putting me in touch with this wonderful organization. 

FreeSet is very interesting in a lot of ways:

• They don’t like to be referred as a NGO. NGOs has a sympathy connotation. Freeset is a Fare Trade Business and they are in Business for Freedom. Freedom for the women.
• They believe in changing the society by being part of the society. It would be easy to take the women out of the society & free them. Living in the same locality & transforming the society is a much more sustainable model. It will changes things fundamentally and provide Freedom from within
• While ever organization I’ve worked for or consulted for, I’ve begged them to stay small and stay focused. FreeSet is the only exception where I really want them to grow their head count. There are 1000’s of women who need help. Today FreeSet produces Jute Bags and T-Shirts. They surely have the potentially to grow into other business.

Why does every business have to be about Profit? Why can’t it be about the people?