Introducing Domain Review Agent: AI-assisted supplier reviews, built on security expertise

Reviewing a supplier assessment means reading every answer to every control to work out what needs a closer look. And the issues that matter are often buried in the detail. A supplier can answer "Yes", be marked compliant, and still admit in their own notes that the control is only partly in place, or not in place at all. Across dozens of suppliers, reviewers sometimes miss these gaps or spend hours finding them.

Today we're launching Domain Review Agent, the latest part of Smart Assessment Reviews. With one click, it runs a first-pass review across all a supplier’s domain's controls, flags the ones that need a closer look and explains why. For each flag, it suggests a next step (start a discussion, raise a remediation, or open a risk), ready for you to confirm or edit. Nothing is created or sent until you decide.

We built this with our users in mind, for teams that need to review suppliers thoroughly without turning every assessment into a manual deep dive - from solo reviewers, lean IT and security teams, to organisations managing a fast-growing supplier base.

In customer interviews, we heard the same tension again and again: teams want to move faster, but not at the expense of judgement. Speed matters when review volumes are growing, but so does the expertise needed to spot issues that could have serious consequences.

Our early access programme showed that the Domain Review Agent could support teams who need both speed and depth of expertise. 90.5% of actions reviewers took matched the AI’s recommendation, meaning less time parsing responses and more time acting on what matters. An early access user added that this would help them standardise responses to suppliers and catch issues they might otherwise miss in the detail.⁠

AI in supplier reviews is only useful if you can trust it, and asking a general chatbot "is this supplier secure?" won't give you an answer you can stand behind. So we gathered the expertise and judgement of experienced security professionals, codified it, and built it into a review assistant that does the groundwork for you.

‍

Codified security knowledge

Reviewing a supplier's security posture takes context and judgement. We built both into the Domain Review Agent from the start.

First, we worked with security professionals from around 20 of our most security-conscious clients, bringing more than 100 years of combined industry experience. Together, we codified what really matters when reviewing a supplier's assessment. We codified the different lenses a security professional uses to verify a supplier's responses, so our Domain Review Agent can pick up on nuances such as governance versus technical controls, gaps in policy, and controls based on criticality.

We then documented our clients' workflows for the different types of issues that can appear in a supplier's security posture. This means that for some issues, the Domain Review Agent recommends a risk note for internal tracking. For others, it recommends contacting the supplier to adjust their security stance by changing or implementing controls.

The result is a review that reads supplier responses critically and finds gaps, even where a supplier would be marked compliant against existing policies.

For example, one supplier answered "Yes" when asked whether sensitive data is kept out of the AI models behind its services, and was marked compliant. Its own notes, however, said data masking wasn't applied across all services and couldn't be relied on as a comprehensive control. The Domain Review Agent flagged the contradiction and, because the supplier was rated critical and held personal data, recommended requesting remediation.

Equally, not every non-compliant answer is a risk to log. Another supplier said it didn't hold Cyber Essentials and pointed to its SOC 2 Type 2 certification instead, but didn't attach the report as evidence. Rather than raising a remediation, the Domain Review Agent recommended starting a discussion: asking whether the supplier plans to certify, or can provide evidence of equivalent controls.

‍

Testing with synthetic data

To test that knowledge, we needed realistic supplier answers in volume. So we built a proprietary tool that generates synthetic assessment responses, written as if submitted by real suppliers.

Our security team helped shape this dataset so it reflects the scenarios that come up in real supplier assessments. As with the rest of our product, we've iterated on the dataset to improve its coverage and completeness, giving us a reliable benchmark to measure the Domain Review Agent against.

‍

Continuous evaluation

We continuously test, monitor and improve our AI features, using two types of evaluation during development and monitoring.

At Risk Ledger, our LLM-as-a-judge evaluators give objective measures of accuracy, hallucination, correctness and tone of voice. Our accuracy evaluator checks that outputs faithfully reflect the inputs, which is particularly important when supplier answers are quoted verbatim in the reasoning. Our hallucination evaluator checks whether outputs are grounded in the information provided, rather than introducing unsupported content.

Alongside these, our proprietary code evaluators check the structure of AI responses, helping us test and refine how those responses are formed and presented.

‍

Choosing the right models

Different AI models excel in different use cases. Throughout the development of our AI solutions, we have tested numerous models from different providers and families, to ensure that the experience in the product is both quick and accurate. As with any architectural decision, there is a trade-off between speed, accuracy, and cost. At Risk Ledger, we have picked our mix of models based on a combination of accuracy and speed.

As new models are released, we continuously test them to determine if there are any significant changes in the outcomes from our evaluations. If we determine new models provide a significant upgrade for our use cases, we can easily switch our products and features to use them, while also updating our evaluations to ensure that monitoring stays consistent.

‍

Safety and control

Our clients expect the highest standards of security from us, and we've designed our AI features with that in mind. As businesses become increasingly dependent on AI, it is becoming extremely important to understand what safety mechanisms are in place for data security, retention and reliability.

For this first release, we decided that every finding from the Domain Review Agent should be presented to a user, who then decides what happens next; as we felt this was the safest way to introduce AI functionality into the product. As users get used to AI workflows and we continue to monitor how the solution performs, we may introduce automations for low-risk actions, so your team can focus on the parts of the review that matter most.

When it comes to data retention, our LLM providers store requests for up to 90 days, strictly for abuse monitoring. We store and manage the requests and responses within our existing cloud infrastructure, and access to that information is controlled by the same mechanisms that we use to handle all of our client and supplier data. Neither Risk Ledger nor our LLM providers use your data to train or fine-tune any AI models.

You can learn more about our AI governance policy here.

‍

‍

Early access results

We gave nearly 50 users early access to the Domain Review Agent. Over two months, they ran 170+ reviews, surfacing findings and suggested next steps. Of the recommended actions reviewers took, over 90.5% matched the Domain Review Agent's primary recommendation.

For us, that's the clearest sign that AI can make a real difference to supplier reviews when it's built on security expertise. Reviewers spend less time reading every answer and more time on the issues that matter, with every decision still in their hands. We'll keep testing, monitoring and improving the Domain Review Agent as more teams start using it.

‍

Get started with Domain Review Agent

Our Domain Review Agent does the first pass on supplier assessments, so your team can spend its time on the risks that need attention. It flags, explains and suggests. Your team decides what happens next.

Existing customers: Our Domain Review Agent is now available to users on on standard plans. Read out help docs on it here. To opt in or opt out of AI features for your organisation, contact Support or your Customer Success Manager.

New to Risk Ledger? Book a demo to see how the Domain Review Agent can help your team review suppliers faster.

‍