What an independent AI audit can tell a church buying a tool

A policy demand becomes a purchasing question

Tim Higgins’s October 11 Wall Street Journal report brought fresh attention to religious leaders’ engagement with AI, including a National Association of Evangelicals policy statement. The policy itself predates that report. Source: wsj.com

The NAE board adopted its resolution on October 1 and announced it on October 5. It calls for independent human auditors of AI systems, accountability traceable to people and protection for whistleblowers. These are policy demands in a foundational statement, not evidence that the association has established an auditing service or certified products. Source: nae.org Source: nae.org

For a church choosing a tool, our question is narrower: what would a report from an independent assessor actually establish about the proposed use? The word ‘audited’ cannot answer that by itself. Leaders need to connect a particular claim, the evidence offered for it and the decision they are considering. This analysis compares official guidance and develops a fictional purchasing exercise. We have not audited a product or observed a church conducting this exercise.

Ask which part of the system was examined

The UK Department for Science, Innovation and Technology’s February 2024 Introduction to AI assurance distinguishes assessments of data, models, complete systems and organizational governance. Its toolkit includes risk and impact assessments, bias audits, compliance audits and other methods. These have different purposes. The guidance recommends combining techniques according to the context and level of risk, rather than expecting one method to settle every question. Source: gov.uk

Our purchasing implication is to read the subject of the report before reading its verdict. A church might receive evidence about a supplier’s internal procedures while needing an answer about a public-facing service. Both questions matter, but one report may cover only one of them. Ask the supplier to identify the connection explicitly. If the assessed component sits inside a larger product, ask which added features and integrations were included. An impressive report can still leave the church’s actual question open.

Independence and relevance need separate checks

The Core of NIST’s 2023 AI Risk Management Framework 1.0 asks for documented tests and performance demonstrated under conditions resembling deployment. Measure 1.3 involves internal experts outside the frontline development team and/or independent assessors, with relevant domain experts, users and affected communities consulted as appropriate. That is different wording from the NAE’s call for independent auditors. NIST also asks organizations to document risks they cannot measure and to monitor systems in use. NIST says the framework is being revised; the provisions discussed here come from version 1.0. Source: airc.nist.gov Source: nist.gov

For a buyer, we suggest two separate questions. First, what gives the assessor sufficient distance from the work being judged? Second, what makes the examination relevant to this congregation’s proposed activity? Independence alone cannot supply missing knowledge of a language, teaching tradition or local workflow. Equally, familiarity with church life does not establish technical competence. A useful report should let leaders see the contributions and limits of each kind of expertise rather than forcing them to infer everything from an organization’s name.

A strong demonstration can leave important gaps

NIST’s July 2024 Generative AI Profile advises against extrapolating capabilities from narrow, unsystematic or anecdotal assessments. It recommends checking generated sources and citations, and evaluating capability claims empirically. Its discussion of pre-deployment testing explains how laboratory conditions and benchmarks can diverge from real-world use; prompt sensitivity and varied contexts can widen that gap. Source: nvlpubs.nist.gov

Applied to a church’s teaching-resource search, a demonstration that finds several passages correctly would leave further questions. What happens when a visitor asks about a subject the archive does not cover? Does the answer identify a source that actually supports its claim? Who decides whether a summary has removed an important qualification? These are our proposed questions, not findings about an existing service.

A committee should also resist reversing the error: an assessment with disclosed weaknesses may be more informative than a polished demonstration that exposes no testing method. The decision concerns whether the evidence supports the intended use, not whether the presentation feels reassuring.

Two fictional offers for the same church task

Imagine a congregation considering a service that answers visitors’ questions using its published teaching archive. It wants people to locate relevant materials, understand when an answer is uncertain and reach a person when the archive is insufficient. The following offers and discussion are entirely fictional.

Supplier A presents an external review of its general-purpose model and a demonstration using three familiar questions. The committee can see that someone outside the development team has examined something. It cannot yet tell whether the review covered the church-facing product, its document collection or its process for directing visitors to staff. The immediate response would be a request for those connections, rather than either automatic approval or dismissal.

Supplier B supplies a report covering a restricted document-search service and identifies the versions examined. Its example material looks closer to the proposed use. But imagine that the report covers only English and assumes a trained reviewer approves each answer before anyone sees it. If the church wants unattended multilingual replies, that proposal also leaves a substantial gap. A narrower assessment can be more relevant without covering the use actually planned.

The committee could respond by changing the task. It might begin with staff using the search tool to locate materials, while visitors continue to receive replies from people. Or it could decide that its existing index and contact form already meet the need. Neither choice is a measured outcome of this hypothetical example; both illustrate that procurement includes deciding what service to offer, not simply choosing a vendor.

Before any trial, the team could prepare invented public questions, agree what an acceptable answer would contain and identify who can judge it. Include a question the archive cannot answer and one on which the church’s own materials give qualified or differing explanations. The point would be to expose a decision the team must make, not to produce a reassuring score. No confidential pastoral material is needed for this exercise.

Request an evidence packet people can read

Our proposed request to a supplier has five parts. It is a discussion aid, not a technical standard or a substitute for specialist assessment.

First, identify the exact product, version, settings and features covered, with the assessment date and exclusions. Ask whether the report concerns the model, the application the church would use, or the organization operating it.

Second, explain the assessor’s role: who commissioned the work, what access was available, who selected the material examined and how disagreements were recorded. The church needs enough information to understand the review’s independence without assuming that payment alone settles the question.

Third, show how important conclusions were reached. If a supplier reports an accuracy percentage, ask what counted as correct, what counted as a failure and what was left out. A percentage without a defined task and method would be difficult for the committee to use.

Fourth, identify unresolved findings and the proposed response. Ask which matters require a change by the supplier and which would depend on church staff. A promised human check needs a real person with the time and competence to perform it.

Finally, explain what would trigger another review: a changed model, a new language, a new integration or a different intended audience. The committee should know whom it can contact when the evidence no longer matches the service in front of it.

The resource objection deserves a practical answer

A small congregation might reasonably object that this asks too much. It may have no procurement specialist, little technical expertise and no budget to commission its own laboratory assessment. A long questionnaire could consume volunteer time without making anyone wiser.

Our response is to reduce the decision to what the church can responsibly understand and operate. Start with the report a supplier already offers, identify the few claims on which the proposed use depends and seek qualified help for gaps the team cannot evaluate. If those gaps remain central, narrow the task or postpone it. Sharing expertise through a denomination could be useful, but an assessment of one configuration should not silently become permission for every congregation’s different use.

The goal is a decision with intelligible reasons. Accumulating documents that nobody can interpret would not achieve it. Neither would accepting a vague assurance because the alternative feels burdensome.

The responsibility a report cannot carry

Christian interpretation adds a question about whose experience counts. In 1 Corinthians 12:12–27, Paul describes the church as an interdependent body, rejecting the idea that one member can dismiss another as unnecessary. The passage concerns belonging and mutual care, not a procurement procedure. Our application is that people affected by a tool deserve attention even when they did not choose it. 1 Corinthians 12:12-27 (NIV)

An assessment can give leaders reasons for confidence within a stated scope. It cannot decide the church’s teaching commitments, supply missing pastoral relationships or take responsibility for the service the church ultimately provides. The useful next step is to name the decision, request the evidence that bears on it and say openly what remains unresolved.