How AI can help with requirements analysis tools

The root cause of failure

Franks discovered how important requirement quality while managing software development for NCR — an omni-channel provider — from 1978 to 2000, where he never had a failed project. When Franks started he was an expert at software development but hadn’t yet mastered software project management. After his predecessor was fired, he was promoted to engineering manager.

“Knowing that I was anything but qualified, I worked hard to learn how not to get fired,” Franks said. He discovered that most projects that grew beyond $2 million had a greater chance of failure than success. One of his key discoveries was how important good estimation was up front. In the larger projects, rework ended up about 60% of the total cost.

Franks’ first big project was with a large bank in Cincinnati and he told the CEO they needed to redo the requirements. The CEO was upset, but after Frank’s team took the time to redo the requirements, they delivered the project on time and on budget. His team only discovered three defects in the first six months in over 2 million lines of code. “I knew that getting requirements right and understanding the size was critically important,” he said.

Augment software analysis

When Franks first heard about using AI for requirements analysis, he experimented with ScopeMaster to size a project and identify problems. He quickly found 150 defects and was able to generate a clean set of requirements in a couple days.

“I was impressed, it would have taken me weeks to do that manually,” Franks said. “Not only did I get a function point count, but it helped me to figure out how many programmers, business analysts and testers should work on the project.”

One area where this kind of AI-driven analysis excels is missing requirement identification. Franks has found that projects with about 1,000 function points should cost about a $1 million and take a year to execute and require about 50,000 lines of code in C# or Java.

However, these projects will also have about 1000 requirements defects.

“That seems impossible, but since I have been managing software engineering for multimillion projects for about 40 years, that ends up being about right,” he said. A great deal of these defects are missing requirements to implement the complete set of Create, Read, Update, Delete operations in a program.

Franks found that he can use ScopeMaster to find and fix about 500 defects in a week. “If you can fix half your requirements in a week, you can suddenly be on time and on budget with 100% of your requirements done,” he said.

Different defects for Agile

Agile development teams tend to be more efficient than their waterfall development counterparts and are generally able to develop higher quality software. In Franks’ experience Agile teams have about .6 defects per function point. The good thing is feedback comes a lot faster, but doing Agile projects also can result in a lot of rework as a result of requirement defects. “Most of the rework has to do with ambiguous requirements rather than missing requirements,” he said.

These problems are compounded for scaled Agile teams where coordination between multiple teams is hard. Teams can centralize requirements to help coordinate them since Agile and Scrum teams don’t tend to generate as much shared documentation.

Franks recently worked on a slot machine application development project that was spread across multiple teams. He discovered they had implemented three different fields to keep information about the jackpots implemented by three different teams. Once he identified this, he had them use the same name and helped cut work down by a third. “Once you use terms consistently, you don’t have duplicate work. With agile sometimes you do duplicate work because you don’t know what terms may be inconsistent,” he said.

Types of defects in user stories

The NLP engines in these requirement analysis tools evaluate the text of user stories and then cross references the results with all the other user stories in a set. It detects the functional intent from each user story.

Colin Hammond, CEO of ScopeMaster said they see 10 types of defects in functional user stories. Today AI can detect defects related to ambiguity, inconsistency, complexity, duplication, omissions, testability and sizeability. However, AI struggles to find defects such as whether a requirement is valid, design laden or current.

The frequency and impact of requirements defects by type varies. Hammond said they typically see seven times more omissions than duplicates. Omissions are what cause projects to go over the original estimate, budget and timeline. Ambiguities are particularly harmful, and often mask other defects. Hammond recommends that teams focus first on disambiguating requirements. Then it’s easier to remove any subsequently exposed defects.

Still in its infancy

Using AI to analyze and process requirements and user stories is still in its infancy. Franks said none of the tools have domain knowledge. For example, when he works on an ATM project for banks, these tools aren’t able to catch things that are specific to how ATMs work.

Also, they’re not able to translate user stories into code. While user stories are the foundation for requirements, developers must also understand nonfunctional requirements and UI issues. Today, AI requirements analysis tools excel in guiding developers to set up packages, classes and methods from the user stories. However, other kinds of platform-specific code generators could help fill in the gaps for things like UI element generation on the front end.