Managing AI scoring for grant applications

AI scoring gives reviewers a consistent, evidence-based starting point for evaluating narrative responses. It evaluates text answers against rubrics your organization defines, so reviewers spend less time on first-pass triage and more time on judgment and follow-up. AI scoring is distinct from using AI summaries to review grant applications, which generates narrative overviews rather than numeric rubric scores.

Permissions

Anyone with the AI Scoring Admin role can create, edit, and publish scoring criteria and frameworks, and access organization AI scoring settings.

Anyone with the AI Scoring Viewer role can view published criteria and frameworks and see scores on grant requests. Users without this role do not see the AI evaluation tab on grant requests.

Enabling AI Scoring for your organization

Use the AI features page in Settings to turn AI scoring on or off for your organization.

1. Open Settings and go to the AI features page.
2. Locate the AI scoring card.
3. Use the toggle on the card to turn AI scoring on or off for your organization.

When AI scoring is on, a confirmation message appears and the AI scoring section appears in the settings sidebar, with links to Scoring frameworks and Criteria library.

Creating and publishing scoring criteria

Scoring criteria are the individual rubric items the AI uses to evaluate applications. Build your criteria in the Criteria library, then attach them to frameworks to put them to work.

1. Open Settings.
2. Select Criteria library under AI scoring in the sidebar.
3. Select Create criteria.
4. Enter a Criteria name and Description so reviewers know what is being measured.
5. Complete the Context field with guidance on what a strong or weak answer looks like for your programs.
6. Add the example and explanation fields for a high-quality response and a low-quality response (the form uses paired Response text with Why is this strong? and Why is this weak? explanations).
7. Select Save draft if you are not ready to go live, or Publish when the criterion should be available to frameworks. Publishing runs validation on required fields.

You can save a draft with just the name. Publishing requires all fields to be complete.

Building and publishing scoring frameworks

A scoring framework groups criteria and assigns them to programs. The product enforces a maximum of 10 criteria per framework.

1. Open Settings.
2. Under AI scoring, select Scoring frameworks.
3. Select Create scoring framework (this action requires framework creation access provided through the AI Scoring Admin role).
4. Enter a Framework name (up to 100 characters).
5. Under Assigned programs, select the submission templates the framework should apply to. Each program can only belong to one published framework at a time.
6. Add published criteria from the library, then select Publish when the framework is ready. Only published frameworks are applied to new submissions on the assigned programs.

Reviewing AI evaluation on grant requests

Once a framework is published, scoring runs automatically when a grant request is submitted on an assigned program. Reviewers can then open the request and view results on the AI evaluation tab.

Expanding a criterion shows the AI evaluation, cited evidence from the application, and the criteria description used to generate the score.

Monitoring scoring status, failures, and updates

Scoring runs asynchronously after submission. The AI evaluation tab shows an in-progress state while scoring is underway.

• Retries and failures: If scoring cannot complete, the platform automatically retries up to three times in the background. If all attempts fail, the tab shows an error state. There is currently no way to manually reprocess a failed score.

• Rubric versions: When a criterion used by your framework has been updated, an Update available badge appears on that criterion. Select Review Update to apply the new version to the framework, or keep the current version. Scores on submissions already scored are not affected. They remain tied to the version that was active at the time. Note that criteria can be shared across multiple frameworks; editing a shared criterion updates it for all frameworks using it. To make changes for one framework only, create a copy of the criterion first.

  

Understanding current limitations

AI scoring has a number of constraints in its initial release that affect how and when it can be applied.

1. Language: AI scoring accepts and analyzes non-English content, but the evaluation results (including scores, justifications, and cited evidence) are always displayed in English.
2. Content type: Scoring reads text responses in the application. File attachments such as PDFs or images are not scored as part of this flow.
3. Scoring model: Each criterion is scored out of 10, and the combined result is presented as an overall score out of 100. All criteria are equally weighted. Frameworks can include up to 10 criteria.
4. Submission size: Submissions over 500,000 characters may fail to score.