Sunday, 25 September 2016

KABI - The new Agile Methodology for BI Projects - Implement BI projects quicker happily

KABI is a new agile software development methodology useful for achieving quicker implementation of Business Intelligence (BI) solutions. The methods defined in KABI can be used in full or in part for any other non BI projects that share similar characteristics as BI development projects. The word KABI is created by combining "KA" from KAnban and "BI" from Business Intelligence.

KABI is a lightweight, iterative, continuous feedback based agile software development methodology that enables every team member to work to their full potential even when there are several unknowns, resource constraints, dependencies and unpredictability of work load. Thereby KABI ensures optimal team productivity throughout the project duration.

At the heart of KABI is "Peer Inspiration". Every team member plays an important role and every team member works in an exemplary way. They do their part of the job so well that it inspires the whole team. "Mutual Trust" between team members is another one of the core values.

Benefits of KABI


  • KABI ensures that the team spends less time on non-productive tasks and more time on developing the product/deliverable. 
  • KABI ensures clear visibility of tasks completed on daily basis.
  • KABI ensures that the team is highly flexible and can accommodate unplanned work at any time with no or minimal overhead.
  • KABI ensures that none of the team members are over loaded.
  • KABI ensures that the team remains highly motivated throughout the project duration.
  • KABI enables the team to find defects/gaps/misalignments very much earlier and thereby reduces rework.
  • All of the above points ensures that the team progresses faster and hence achieve quicker implementation while continuing to deliver higher priority unplanned deliverables throughout the phase of implementation.

KABI Workflow

In KABI, development is iterative but it is not time boxed like in Scrum. Planning and releases can happen at any time on need basis. Feedback and improvements are continuous and a daily routine. Product stakeholders are notified as and when there is a new product increment or deliverable by the product owner. Stakeholder demos are once in every two months or on need basis. User stories are written such that it results into smallest concrete deliverables independent of the size or complexity of the deliverables. As there is no time boxing there is no need for breaking a user story based on time required to implement a user story. Real complexity and effort estimate of a first time task is only determined after work on that task has started. 

KABI primarily has 5 activities as listed below, these are explained in detail in the following sections.
  1. User stories creation and prioritization
  2. Implementation
  3. DDAS ( Daily Demo and Sync-up) 
  4. Weekly Status Reporting
  5. High level Planning

KABI roles
  1. Product Owner (PO)
  2. Development Team ( Developer, Architect, Data Modeler and Quality Engineer)

KABI requirements
  1. Co-location of development team and online collaboration
  2. Common goal
  3. Virtual Agile Kanban Board
  4. Mostly experienced team members (not necessarily in the same tools)
  5. Management trust and backing

User Stories Creation and Prioritization


In KABI requirements are created as user stories, prioritized and ordered (highest at the top) by the product owner (PO).  There are only two statuses (To-do and In progress) that are tracked on the KABI board. KABI board is accessible to every team member but only product owner can change priorities of user stories. Product owner rearranges the to-do list as and when the priority changes and keeps it always ordered according to the current priorities.

Implementation

Development team picks up user stories from top of the to-do list and implements the user stories based on the available capacity. Releases happen as and when a product increment / deliverable is ready or consolidated and released when consolidation makes more sense in terms of time taken for  a release. 


The implementation activity is quite simple. Quality engineer (QE) adds test scenarios to the top of the list user stories. If the user stories are not clear for QE then QE discusses it with PO. Depending on the scenarios the user stories are either modified or split or consolidated. This step already refines the user story to a good level.


Now user stories are ready for a developer to pick up.  Developer first analyses the user story and breaks it down to sub-tasks. Best practice is to break it down to a sub-task of not more than 1 day long. This way every day there is a progress seen for the user story. This is the first time we get a good idea of when the user story can be actually completed. If the user story is not clear for developers they discuss it with QE and PO and once there is clarity developer continues his work. 


While developer is developing, QE is already creating the test cases based on test scenarios and also setting up the test data. Developer does his part of the testing and verifies if it satisfies all the test scenarios and if possible uses the test data setup by QE to test his deliverable. Once testing by the developer is completed it is then passed on to the QE for quality assurance. Then the rest is usual process of defect logging, fixing, reworking, optional peer reviews, ensuring zero defects before releasing to higher environments.

It is very important that product owner ensures that his dev team is not distracted by stakeholders directly providing requirements to the dev team. Dev team should be given continuous uninterrupted times for development activities. 


Most often many stakeholders claim that their requirement has the highest priority. However, product owner has the full view of all the requirements and can combine, break, remove redundant requirements and prioritize requirements accordingly.  This is where the experience of the dev team comes in handy. If the dev team is directly approached by users with new requirements they politely redirect stakeholders to contact the product owner and do not give into pressure and get into prioritization of user stories unless there is a technical need. When in doubt we refer to the KABI's guiding principles aka manifesto placed on our team wall

KABI Manifesto aka Guiding Principles
  1. On-Time healthy breakfast over delicious heavy breakfast ready at evening.
  2. Menu card approach over mama's kitchen.
  3. Common sense and facts over theories, books, models and everything else. 

What is the point in preparing a delicious heavy breakfast if it is ready only in the evening? It is too late for today and stale for tomorrow.  Same applies for deliverables, providing a healthy (working) deliverable on-time is preferred over engineering too much and delivering something late when it is no longer relevant. Yes, if you can deliver a delicious breakfast on-time, this is perfect.

What does a menu card in a restaurant do? Informs customers about available choices and helps them in deciding what to choose. What if there was no menu card? Every customer would enquire about available food, prices, available combinations and so on, such a waste of time. If it was your mama's kitchen then you would ask for your own specific combination (customization). We prefer creating standard deliverables that works for 95% of the stakeholders and create a list (menu) that showcases our deliverables for stakeholders to choose from over working on customizations for 5% of our stakeholders and delaying standard deliverables. 

Third point is a straight forward one compared to the first two. We rely more on common sense and Facts than any other sources. 

DDAS

DDAS (Daily Demo and Sync-up) is one of the core and very effective practice introduced in KABI. It may initially look like a big overhead but eventually one will realize the benefits of practicing this especially if one is managing or heading a BI project.

Daily Demo & Sync-up of ideally 30 minutes during start of the day is for a intra-team demo and sync-up. Team members will use this time to show/demo/present the work carried out on the previous day and talk about the planned work for the current day.

The best practice is to have a big wall monitor in the team room such that all team members can see it comfortably sitting at their desks and avoid wasting time moving and settling down in a conference room. If that's not feasible then the other option is that team members physically move to each team member's desk to see the demo as each one presents.

A demo does not mean that always a report has to be run or a job has to be run and shown. It could also be as simple as showing the test scenarios written by QE or user stories written by the PO or table designed or even a one line code that caused an issue previous day.

DDAS ensures that the progress is very transparent to all team members and there is continuous knowledge sharing happening within the team.

DDAS is mainly for the team members. Guests are welcome but they are mostly silent listeners. It is not expected to do any special data setup or additional preparation for the DDAS. It is only to showcase the as-is situation and get feedback from all team members. It's the time for PO to quickly check if the development is happening as per requirements, it's the time for the architect to quickly check if the coding standards and framework is followed,  it's the time for QE to see if there are any gaps, in general it's the time for all team members to align on the progress of a deliverable.

Team members are encouraged to ask questions related to the demo topic and get doubts clarified on the spot. Every team member actively participates in DDAS and provides immediate feedback and suggests/requests for any changes and raises any concerns. Team agrees about which of the identified changes will be done/not done. 

At the end of DDAS each team member is also aware on what every other member of the team is planning to work on and avoid any duplicate/conflicting work for the day.


DDAS Rules

Some of the rules to be followed to ensure DDAS works as expected is listed below

  • Team members should make themselves available for DDAS. 
  • All other meetings are considered lower priority than DDAS.
  • As each team member is expected to demo and talk, it is possible that the allocated 30 minutes is not enough. Team members should keep the relevant user stories/sub tasks, jobs, reports, data model, tables, objects open upfront before the DDAS starts to save team's time. They should mentally prepare upfront on what they are going to talk during DDAS.
  • If a team member requires more details or design related  detailed discussions are required or a training is required on particular topic, this has to be noted during DDAS and then arranged outside of DDAS. Note that DDAS is strictly not meant to be a training session or design discussion session. Results of a previous discussion can be shared during DDAS.
  • DDAS should not be used to share long personal updates, a quick one-liner is allowed.
  • DDAS should be used to demo previous day's work only and not repeat the topics already covered in the previous day's DDAS.
  • During DDAS team member is supposed to only demo or talk about his contribution or part of the work and not on behalf of other team members unless the other team member is unavailable for DDAS and a combined work has been done.
  • During the day if there are any changes to planned work then the team member should shout out or notify other team members.
  • There can't be a day when a team member has nothing to demo or nothing to tell if the team member has really worked previous day. Each team member has the right to question and ensure that the team member shares what he did previous day. 
  • If previous day was a non working day then updates about the last working day should be shared, it's that simple.

In spite of  making all of the above arrangements and following all the rules if DDAS extends beyond 30 minutes, there is nothing to worry, It's not a waste of time, It's a good sign of team working harder the previous day so they have more to show or it's a good sign of them planning to do more today so they have more to say or It is a good knowledge sharing session between team members and there are many benefits of continuous knowledge sharing. 
We used Atlassian JIRA time tracking sheet as pointers to showcase what we did previous day.  This also ensured that we tracked our time strictly to the tasks we have been doing and hence track how much time was required for a particular task for future purposes.

DDAS Notes
Taking notes (just the key points) during DDAS is optional but this can also be very useful. We used confluence to track daily notes. Even if this is not done always, it could be started whenever a new member joins the team till the time he gets fully on-boarded and is familiarized with the project. It usually didn't take more than 5 minutes to add the notes to confluence. There are 3 benefits of doing this.

  • New joiners would actively involve in understanding each of the topics demoed and discussed during DDAS.
  • When new joiners write notes back to confluence the entire team gets to see if new joiner has understood the topic or not.
  • When a team member comes back from leave/vacation he can refer to the notes page to see if he missed anything important.

Weekly Status Report

A short one page report created quickly at the end of the week (Friday) that provides a high level description of what was done in current week, what is planned for next week at a high level and other sections as shown in the image below.

The product owner creates the report based on inputs from the team. High level plans for the next week are agreed-to by all team members.

This is the only progress report shared with the higher management. The KABI board is always anyway available for the details.

Short term deliverables refers to those that will be done approximately in a months' time.

It is a good practice to print the High Level Plan for the week and stick it somewhere where team cannot miss seeing it every day. This is useful during DDAS and constantly reminds team of current week's plan. We placed a hard copy on the back of the team room's door even though we maintained a copy in Atlassian confluence. 



Most of you must be wondering how and when do we come up with the timelines. This is where the High Level Planning comes into play.

High Level Planning

As a first step product owner does the analysis and comes up with a high level rough estimate and timelines. It is assumed that product owner also has sufficient experience in implementing BI solutions and can make an educated guess by looking at the requirements and understand if a new star schema is required or is it an addition of fact or a dimension or is it just an enhancement of an existing fact or dimension or a change to existing reports. In the second step the entire team sits together and goes over the prepared high level estimates and timelines and then comes up with an agreed timelines for the next few months. There is no recurring high level planning meetings. Meetings happen only on need basis. 

Comparison between Scrum and KABI


Scrum
KABI
Daily standups. In most of the teams this has become just a formality or is used to ensure people come to office on time. No useful info is shared.
Replaced by DDAS (Daily Demo and Sync-up). Every team member really demos or presents what was done yesterday and tells what he plans to do today.
Assigning user story points, task breakdowns, sprint planning and capacity planning.
Not required. We deliver what we can, we plan what we think we can deliver in that week during weekly status report creation.
User stories have to be broken down to less than sprint duration at the minimum. Waste of time in breaking user story to minimum size.
No time boxing. User stories are totally deliverable based, so no need to trim down the user story. No time wasted.
Sprint retrospective. Wait for sprint to end
Continuous feedback during DDAS including feedback about the processes itself. Team to agree or disagree instantly and not wait.
Roles - Scrum master, development team, and product owner.  Scrum master has to ensure processes are followed.

Co-location preferred.
Roles - Product owner and development Team.
Every member ensures processes are followed.

Co-location preferred.
Sprint commitments sent every two to four weeks.
Replaced by weekly status report. We cannot commit to something we don't know. We develop and then notify.
Sprint demo - End of sprint Customer demos - Sprint demo or need basis
Sprint demo replaced by DDAS. Helps in finding gaps on daily basis.
Customer demos once in two months  or on need basis.
Unplanned higher priority activities would break Sprints, Lot of rearrangement required
Unplanned higher priority activities can be dealt smoothly.
No rearrangement required, just stop current work and pick it up later.
Releases at end of sprint.


Product backlog

Sprint backlog

Impediments handled by scrum master. Time wasted in developer - scrum master coordination. 
Release as frequently as required.

To-do list

To-do list

Every team member deals with the impediment himself or escalates to management in weekly status report.

I apologize to all scrum followers in advance. To KABI, scrum looks the same way as how waterfall looked to scrum some years ago. 


Wind-up

I hope KABI is useful for all those who are struggling with the existing methodologies while implementing new BI Solution or building any other solutions that have similar challenges or for those who are just looking out for alternate agile methodology for implementing BI or other projects. 

A typical BI solution has primarily 3 layers; Data collection layer, data integration & storage layer and information presentation  & access layer. KABI is extremely useful even when you have none of these layers and you are starting absolutely from scratch.

KABI methodology was created by me and successfully used for implementing a Business Intelligence solution by me and my team members in one of the projects which had several challenges, unknowns, dependencies and unplanned activities throughout the implementation phase. Thanks to higher management support to let us experiment KABI even though the whole organization was more or less working with scrum methodology. Thanks a lot to my team members who readily accepted the change from scrum to KABI and became main part of the methodology change experiment. I am sure without their full and continuous support nothing would work. We tried to follow all of the methods defined above to the extent possible and made good progress but there is always scope for improvement. 

We now have a working BI solution backed by a solid dimensional data warehouse. Users are on-boarded and are able to create their own reports and we have a set of standard reports that can be enabled for a new customer without any development efforts. We achieved all this in spite of having to continue with the additional responsibilities throughout the implementation phase. To give a rough idea about the additional responsibilities; We have delivered over 100 ad hoc reports in a year, carried out multiple production data analysis exercises, supported sales in every RFP/RFI's that required our inputs and supported operations team in multiple reporting and non reporting incidents.


If you are interested in reading what drove me to create KABI please check my next article - Why KABI was created?


Updated - 03-Oct-2016 - Thanks to friends, ex-colleagues, colleagues and blog readers for your public and private feedback and comments. I have posted frequently asked questions in this new post KABI-FAQ.

6 comments:

  1. Perfect approach Anoop... Congratulations for successful implementation of new Agile Methodology. We will try to implement "KABI" in upcoming BI projects. DDAS is playing a key role in KABI.

    -Jasdeep Malik

    ReplyDelete
    Replies
    1. Thanks Jasdeep, Yes, DDAS makes a big difference. I hope KABI helps you in your upcoming BI projects. I will be interested to know.

      Delete
  2. Congratulations Anoop and thanks for sharing the knowledge!! I wish you all the good luck for new challenges and learing. This article is really helpful to understand the methodology and the advantages and scenarios!!

    Best Wishes,
    Anita

    ReplyDelete
    Replies
    1. Thanks Anita, Motivates me to write more articles.

      Delete
  3. I think this is a good set of practices that has apparently worked quite well in your context. I completely agree that Kanban can be a great method to handle the BI world. However, I don't think KABI stand on its own as a methodology- I feel it's essentially Kanban with some supplemental practices discovered as you went along. If another BI team implemented Kanban in a different organization, they would probably wind up with something different, but could quite well be just as successful.

    Also, I don't agree that Scrum should be discarded by all because the attempt didn't succeed in this context. I have seen Scrum flourish in the EDW/BI world (as well as Kanban).

    ReplyDelete
    Replies
    1. Thanks for taking time and reading the post and also providing valuable comment. I completely agree. I would like to clarify that, I believe, given similar conditions/context, a BI Team would be able to progress faster with KABI than with Scrum.

      Lets assume that a Head of Marketing/Finance wants to setup a BI Solution for the department with a self-organized team of 4-7 members and the team is also expected to deliver reports based on source databases on adhoc basis, perform production data analysis, support on incidents etc while continuing to develop the BI Solution. This team could take KABI as the development methodology without any modifications and I think that the team will be able to implement BI solution faster than it would do it with Scrum. As KABI is still new it will take some more time and more successful projects to become an established methodology.

      And yes, I am sure many Teams would have had success with Scrum in BI or other projects. However we need to see if they had similar context. Just thinking, could they have done even better with KABI? I don't know the answer.

      Delete