The project we’ll be discussing involves developing an AI-based software solution to analyze Contact Management data across a massive collection (in the order of hundreds of millions) of CRM records and across a variety of popular CRM systems.
The project was to create a scalable solution that could predict at certain probability levels which records are up-to-date (and which are obsolete), which records refer to the same person but at different roles/jobs, or conversely, whether 2 Jane Smith’s are duplicates or are truly 2 different Jane Smith’s. It also had to gracefully handle large variations in formatting, spelling, and data quality, including outright data errors.
A few notes for context:
- This project happened a few years before the emergence of modern LLMs.
- The project was designed to be used with all kinds of CRMs, but …
- One of the CRM companies offered us a substantial advance payment in exchange for an earlier delivery timeline, which resulted in the hard deadline.
Challenges
The first key challenge is naturally the inherent uncertainty of developing AI algorithms. Aside from that, there were many external dependencies that could easily throw our timeline off. Some of the important ones were:
- API agreement with the CRM dev team,
- Signed agreements by the CRM company’s customers to allow access to their data,
- How to access the massive dataset but still keep it secure
- It was unknown how clean the large data sets were, especially coming from various CRM systems
Given the many unknowns, including the AI algo itself, the project had a large number of inherent risks beyond typical software development risks. Within this context, our team had to adapt Agile principles to still deliver the agreed upon scope within the contractual timeframe.
After some brainstorming, we decided to use three project management techniques, with some adaptations: Risk Management, Critical Path Management, and Swarm methodology.
Adapting Agile: Our Three-Pronged Approach
Risk Management Technique
Risk Management is a systematic approach to identifying, assessing, and mitigating potential risks that could impact the project.
Projects often go late due to last minute surprises. Given the project’s complex AI algorithm, massive dataset, and many 3rd party dependencies, there were many many potential surprises. The hard deadline made it crucial to identify and address potential risks early on, and thereby avoid costly surprises late in the timeline.
We employed Risk Management Techniques early in the project. The thinking was if a risk was identified early, we probably had time to address it. But if it came up as a late surprise, the whole project would likely be thrown off.
Some of the key risks we identified early included:
- Technical Risks: Integration challenges with CRM systems, potential scalability problems with the algorithm.
- Data Consistency Risks: This turned out to be a big one, as there were myriad of ways CRM data was entered, resulting in variety of data issues such as:
- Name spelling errors, persons or companies
- Same data but different formats, e.g., phone number formats, address formats, use of abbreviations, etc.
- First name conventions, e.g., are “Ali” and “Alison” the same name for the same person, how about Bill, William, Billy, Will?
- External Risks, incl. dependencies on the CRM company’s dev teams, such as API agreement to data upload for initial development and then the feed of large data set for testing
We prioritized these risks based on their potential severity and likelihood of occurrence, akin to the technique we discussed in deeper details in the Book.
As part of identifying the highest severity risks, we employed Critical Path Management technique, which is the discussion in the next section.
Critical Path Management
Critical Path Management (CPM) is a project management technique that involves identifying the longest sequence of dependent tasks – the Critical Path – that determines the minimum duration required to complete the project.
Our goal with CPM was to shorten the critical path as much as possible, thereby reducing the overall project minimum timeline, and in this context, to create some buffer before the deadline.
Critical Path Management goes hand-in-hand with Risk Management. By thinking Critical Path, it helped us identify the risks that were directly on the Critical Path, and therefore needed to be prioritized. Naturally, Critical Path risks would automatically push out our timeline if not handled. Hence the principle here was if a risk item is directly on a Critical Path, we would bump up its severity factor to High.
The Key Critical Path in this project was as following (high level):
Multiple Testable Algorithms → Test and compare algos with small data set → Test and compare algos with larger data set → Tune performance of algos if needed → Test and compare algos with Very Large data set → Tune performance of algos if needed → repeat as needed → final integration → final testing → release candidate
However, multiple dependencies feed into the Critical Path items.
One example was to obtain signed releases from the CRM customers to allow their data to be used for the many stages of algo testing (and final testing).
Since these signed releases were considered high risk items, we assigned a dedicated liaison to work closely and EARLY with the CRM companies (our customers) to work with their customers to obtain the releases in time for our needs.
This item alone turned out to have many sub-risks and complications, as some customers were not willing to sign the release, some others had lengthy/cumbersome chain-of-approval, some even renegotiated for discounts/deals to release the data.
All in all, if we had not identified and assigned a liaison to handle this early, it would have easily pushed out our timeline beyond the deadline.
Swarm Methodology
The Swarm methodology is a collaborative problem-solving approach where multiple team members with relevant skills come together in real-time to focus on and quickly resolve a single, high-priority issue.
As defined above, typically Swarm is employed during crunch time, when teams/team members come together to address some critical issues to meet deadlines. However, in our case, because we identified risks early in the project, we were able to proactively apply Swarm methodology selectively early to tackle risks identified as high-probability and high-severity, thereby avoiding the situation where the risks became real issues late in the timeline.
We used this method alongside risk management. Specifically, one of the risks we identified early was the usability of the incoming data sets from various sources. Even though we had a contractual deadline with one customer, we also had other customers that did not impose a tight deadline, but our system still needed to be trained with their data sets to ensure the resulting algorithm accounted for all likely data sets.
Because the data drop timing was quite unpredictable, it didn’t make sense to set up a team to handle incoming data on a permanent basis. Also, some data drops needed exponentially more work than others, so the effort level was quite unpredictable as well.
After some discussions, we decided to try using an adaptation of the Swarm Method. Specifically, every time we received a data drop, an engineer would give it a quick look, and if needed, he would call for a Swarm. Then the designated Swarm team would come together quickly to handle the data drop.
Sure enough, some of the incoming data did turn out to be more problematic than others. In those cases, the Swarm method worked out quite well, mitigating data risks early and quickly and reducing the project duration overall. Obviously, these also became distractions on the engineers’ other tasks. We consciously discussed it in our retrospection sessions. All in all, the team agreed that the trade-offs were worthwhile due to the Critical-Path nature of data digestion, while the other tasks often had some slacks. (The team overall was also empowered to self-manage, specifically to organize around the Swarm if possible, or skip a Swarm when their other tasks are in jeopardy, but that’s beyond the scope of this blog).
Conclusion: Agile’s Enduring Adaptability
This real-life project demonstrates that Agile’s core principles of fast iteration and adaptability are not incompatible with fixed timelines and firm scopes. By strategically adopting and integrating robust Risk Management, precise Critical Path analysis, and adaptive methodologies like Swarm, teams can navigate complex projects with high degrees of uncertainty. The key lies in EARLY identification of risks, prioritizing those that impact the critical path, and employing flexible, collaborative approaches to tackle them proactively. This allows for continuous adjustment and problem-solving, transforming potential last-minute surprises into manageable challenges.
The combination of these three adapted techniques proved instrumental in the project’s successful delivery. By proactively identifying the AI algorithm’s performance against massive datasets and the unpredictable nature of data acquisition as high-severity, critical path risks, we were able to allocate resources and attention precisely where they were most needed. The dedicated liaison for customer data releases, for instance, addressed a significant external dependency early on, preventing what could have been a major bottleneck. Similarly, the selective and proactive application of the Swarm method for incoming data sets allowed us to quickly resolve data consistency issues as they arose, preventing them from escalating into project-derailing problems later in the timeline.
This overall approach meant that instead of reacting to crises, we were consistently mitigating potential issues BEFORE they impacted the critical path. This proactive management of risks, coupled with a clear focus on the project’s most time-sensitive elements, allowed us to maintain momentum and deliver the complex AI solution within the firm contractual deadline, and with surprisingly stress-free completion. In fact, during our retrospect, the team agreed that if we had missed any of these items, very likely we would have missed the deadline. All-in-all, this experience underscores Agile’s enduring power as a project management philosophy, proving that with thoughtful adaptation, it can deliver success even under the most stringent constraints.
