Asking some tough questions for new software selections in a generative AI era

Brian Sommer Profile picture for user brianssommer January 26, 2024
Don’t buy your next ERP, CRM, HRMS, etc. solution quite so fast. You might want to seriously investigate what each vendor is doing (or not) vis-à-vis its new AI-fueled applications/solutions. If you’re used to using some old boilerplate as the guts of a software RFP, here are some important additions to that workhorse documentation.

Business man considers question mark with arrows © Jirsak - shutterstock
(© Jirsak - shutterstock)

AI has been a part of the software scene for many years although most software buyers haven’t really incorporated AI-specific questions in their RFIs (requests for information), RFPs (requests for proposals) and other software purchase documentation. This lack of interest in documenting AI may have been due to:

  • Many mainstream software vendors didn’t fully embrace early AI or ML (Machine Learning) technologies into their products. Case in point: RPA (Robotic Process Automation) has been around for many years, but rarely have we seen it included as part of a standard ERP product line. 
  • Many of those same vendors were content to let ML, RPA, etc. solutions to co-exist as solution partners to their own products. Customers would need to work with specialized integrators, consultants and process architects to configure and integrate these tools. 
  • Process automation work often triggers process re-imagination, process animation trials, workflow statistic gathering, etc. These activities involve skillsets that are not common within a traditional application software vendor’s internal services group. These activities also run counter to the quick implementations that vendors tout (These implementations map old systems data to the new system, enter configuration table data, and launch the new system even though the new processes will act much like the old ones.). 

These prior AI tools were not within the vendor’s product development and services deployment core competencies and buyers rarely saw these capabilities. If they did, they were in the margins and only in recent years. 

In 2023, vendors came to a new realization re: AI, specifically generative AI. Those tools could write code, act as smart chatbots, write job descriptions, dynamically generate multiple, personal career paths for employees in real-time, and, much more. What vendor leadership saw in 2023 were tools that could possibly: 

  • replace or substantially enhance huge chunks of existing applications
  • make all of us rethink what we need human employees to do in most any business process
  • change the economics and labor requirements of businesses everywhere (including those in tech companies, too) 

Last year also saw vendors scrambling as generative AI caught most firms completely by surprise. Its multitude of capabilities was daunting, its risks not fully understood, its potential competitive advantage was staggering, and, tech firms rushed to embrace the tools warts and all. 

For customers of these applications/solutions, there are a number of new risks. Privacy, security and litigation concerns are just three important categories of risks. 

Smart software buyers need to add additional protections to their software purchases. The first step is to ask questions of these vendors and get comfortable with the different decisions, responses and approaches each firm has taken in their quest to sprinkle/saturate AI within their product suite. 

The questions every software RFP needs

Indemnification - There have already been some extraordinary/awful AI-powered impacts in businesses today. A chatbot at a Chevy dealership was recommending buyers get Fords or other brands of vehicles. Other chatbots hallucinate. Others curse. And the list goes on. 

Software buyers want to know who is responsible for the output of these tools. Software vendors that are using third-party LLMs (Large Language Models) are unlikely to stand behind these tools especially if they didn’t train them. So, before you incorporate any of this tech into your firm, ask the vendors:

  • Is this tool/application wholly your own creation and do you warrant it to be free of hallucinations, profanity, obscenities, etc.?
  • If this tool/application contains third-party technology, do you warrant it to be incapable of producing objectionable, inflammatory, sexist, racist or other objectionable output?
  • Will you indemnify a customer against any damages, litigation, bad press, etc.  if your embedded LLMs are discovered to be trained on unlicensed materials? 
  • Will you indemnify a customer if it is found that your AI tools produce copy that infringes on a third-party’s copyright or trademark?
  • Likewise, describe the testing you did to ensure your tool does not plagiarize the works of others
  • Please show us the signed agreements you have with all third-parties whose content you used to train your tools.
  • Can the vendor provide proof of ownership or licensing for ALL of the materials that the AI tools use especially those that trained the models?
  • Will the vendor warrant that the data used in its tools was acquired from sources that had explicit, express, written permission to share this data?
  • Will the vendor provide a multi-million-dollar surety bond to cover potential damages caused by its AI-powered tools?
  • If the AI tool provides forecast information (based on a mix of customer and third-party data it was trained upon), does the customer have any recourse if the forecast is wildly wrong and causes substantial economic harm to the customer?
  • If the software buyer uses these tools in an end-customer facing role (e.g., a shopping assistant chatbot), will the vendor completely indemnify the buyer should the tool make bad recommendations, use offensive language, trigger a permanent loss of this customer’s business, etc.?  
  • What kinds of risks must the software buyer assume that the vendor will NOT indemnify? 
  • When does responsibility for the actions/recommendations/content of these tools shift from the vendor to the software buyer? 
  • Can/Will the vendor defend the AI-models, algorithms, etc. in a court of law? Who knows how this black box works and who can explain its workings to a jury?

Effect repairs to AI -  Over time, users of these tools will learn that the tools need recalibrating, new or different training datasets, etc. There’s also the possibility that a specific LLM, algorithm or other technology needs to back out certain training data, planning/forecasting elements, etc. These AI tools are not static and readers should not expect that a solution purchase is a one and done deal.  

Specifically, software buyers should make inquiries like:

  • Can the vendor easily remove a dataset that was used to train models and recalibrate the results? Can this be as simple as selecting/deselecting an item from a list? How is the learning model re-calibrated? Is this something that the customer can do? What will this cost?
  • Can a customer select which data sets to include/exclude? For example, can a customer opt for datasets based on public domain data (e.g., data older than 35 years and not subject to copyright protection)?

Error correction procedures should be present with all AI tools -

  • Can the vendor diagnose and correct a tool and/or dataset if a customer finds a hallucination, plagiarized content, offensive content, etc.? 
  • Does the diagnosis (or repair) have to be completed on-site or can it be done virtually?
  • Can the vendor replicate bad, incorrect, hinky, etc. results previously returned by the AI tool?
  • Does the tool maintain a log file so that developers can replicate the parameters that triggered an adverse result?
  • Does the tool have version controls like traditional applications?
  • What is the error correction process and what escalation procedures are present?
  • If necessary, can a vendor ‘recall’ an AI tool and what recourse will the customer have while this product is offline? 

AI and ESG - Cloud data centers, where most AI tools operate out of, are big consumers of energy (to power servers and to crunch data) and water (to keep this equipment cool). Software buyers might think these environmental concerns are not theirs to worry with as these data centers are run by a third-party. That assumption would be incorrect as ESG regulatory reporting requires firms to track the emissions and other data that suppliers incur on behalf of your firm. This is referred to as Scope 3 ESG reporting. 

And, in AI terms, this ESG impact could be substantial.  If a potential software vendor cannot provide this information to you (or do so in a reasonable timeframe or with some level of accuracy), then your firm may need to seek a different solution vendor. 

Specifically, prospective software buyers should ask vendors:

  • Provide the last two years of Scope 3 emissions data from your AI-cloud data centers (if it even exists)
  • Tell us where these data centers are 
  • Are these data centers using no-carbon emissions fuel sources (e.g., solar or geothermal energy)? Or, is this vendor using carbon offset credits in lieu of using a low/no-carbon emissions energy source?
  • Is the cooling water being repurposed as heating (e.g., for a nearby school), recycled or simply evaporated?
  • Is the data captured aggregated across all of the data center’s users or is it being measured separately for each customer?
  • Is the data highly averaged, highly aggregated and not customer specific?
  • Can the emissions data be more granular?
  • Will the AI data center trigger any data sovereignty concerns for the customer?
  • Can a customer get real-time carbon emissions data?
  • Once AI tools are trained, can the training data be moved to offline storage that requires no electricity or cooling?

ESG reporting also has a strong social component, too. Firms are supposed to report about their suppliers’ impact in local communities, their workforces, etc. To that end, companies must know if their suppliers (no matter if the supplier is a direct supplier or is n-tiers deep within the customer’s supply chain):

  • Use forced labor
  • Fail to pay a livable wage
  • Are restricted entities
  • Etc.

Software buyers should:

  • Examine the CSR (corporate social responsibility) reports of all potential vendors with an eye towards the kinds of data the vendor reports on its suppliers. Is it at an appropriate level of detail? Is it balanced and complete or does it simply report the data elements that favor the vendor?
  • Request the vendor’s Scope 3 emissions reports for the last 2 years
  • Request an interview with the vendor’s chief sustainability officer to elaborate on current and future plans in this rapidly evolving area.
  • Assess the quality and detail of Scope 3 environmental and social data the vendor currently collects.

AI and Ethics - “Just because you can doesn’t mean you should” is something more vendors should consider. Not every potential application of AI should become a market offering. Some tools could inadvertently trigger outsized PR and legal issues for a firm – issues whose potential costs could outweigh any planned benefit. For example, what good is an AI-powered job candidate selection tool if it exposes the firm to allegations of discrimination? The damage to the software customer’s brand, employment reputation, etc. could be huge and dwarf the cost of the application. 

Software buyers need to understand how vendors will be making product and ethical decisions regarding these all-new technologies. For example:

  • Does the vendor have a risk assessment process and does it reject certain proposed uses of AI in the product line? Can the vendor provide examples of AI-tools it will not produce?
  • Does the vendor have an Ethics/Compliance Officer that is influencing product scope of AI applications?
  • How does the vendor decide what third-party data to use with its AI tools? What sort of due diligence is performed on this data to ensure:
    • The data was acquired legally and its use would not surprise those whose data is in the tools (e.g., if a LLM was trained on copyrighted books, did the authors of those works give their explicit permission for this specific use?)
    • The AI-tools will NOT generate outputs that substantially resemble (or outright replicate) data within the tools.
  • Does the vendor distinguish between AI-tools that can incorporate a customer’s data and those that only use third-party data? How does it prevent a customer’s private data from entering into its public tools/apps?

Pilot program - These new AI tools are just that: new. New technology can be buggy, wonky, difficult to implement, hard to use, etc. In fact, it might not ever deliver the promised value. 

Buyers of these new tools are being asked to accept these products almost as a leap of faith. But a smart, rational firm won’t necessarily accept the claims of great benefits, ease of use, etc. on new solutions without proof. 

Buyers might ask:

  • Can you provide us references for each of these AI tools? Don’t be surprised if these are rare and may lack any long-term experience with the tool.
  • Can we get a limited use or pilot subscription to fully test out the product over a 3-6 month timeframe (Note: vendors hate these requests and may demand substantial upfront monies)?
  • If we license/subscribe to several of these AI tools but a couple of them don’t end up meeting our expectations, how do we get a refund on those? What other product alternatives might we be able to use instead? 

Adult supervision - Every vendor has a CYA (cover your a**) caveat with regard to new AI capabilities: Firms should always have a ‘co-pilot’ reviewing the outputs of any generative AI tool. But knowing human nature (and businesses), that isn’t likely to happen. No human is going to review every comment and interaction a chatbot has with your customers, suppliers, website visitors, jobseekers, etc.  It’s more likely that co-piloting will occur with some frequency during the early days of the AI’s deployment and then quickly diminish thereafter. 

What customers need are strong vendor tools that monitor AI outputs. These tools are independent of the technology and are trained to look for certain bits of language, behaviors, etc. Think of this as AI monitoring AI. 

Customers should not buy or use AI tools that lack meaningful, immediate controls and guardrails. What would happen if a chatbot started hallucinating at your firm in the middle of the night when the ‘co-pilot’ was home asleep? Prospective buyers must insist on seeing, testing and stressing whatever controls a vendor offers for their AI tool. But a human co-pilot is only a (small) part of the equation. 

My take

I do a lot of selections for/with clients. This is my first blush listing of new AI-propelled requirements. I am quite aware that there are many other areas customers should probe, too. For instance, shouldn’t customers get staffing estimates and education/training requirements from the vendor for the different AI tools that a customer might subscribe to or license? This alone might surprise a buyer when they realize that they don’t have anyone internally that’s qualified to evaluate, let alone maintain, these new technologies.  

Please send me your additions, suggestions, critiques, etc. and hopefully we’ll all help software buyers get better business outcomes.  

Oh, and no,  an AI tool didn’t help me write one bit of this. 

A grey colored placeholder image