Checking AI Output
How to check an AI tool against the source: the step of the visit each NPPR section serves, and BCCNM, health authority and CMPA guidance on reviewing AI-drafted notes and answers.
Reviewed by Joe Grafe Due
Review overdue
This page was due for review on April 30, 2027. Check the linked sources for newer versions before relying on it.
NPPR is built so an NP can check what an AI tool says against the source document, as the human in the loop. This page maps each step of the visit to the NPPR section that holds the sources for it, then lists the published guidance on reviewing AI output before it is used or charted.
Key Sources¶
- BCCNM - Using Artificial Intelligence in Practice (2026) — BCCNM's learning resource for NPs: the standards that apply to AI and the questions to ask about privacy, bias, consent, and safety and transparency (evaluating AI output, accountability, over-reliance)
- Island Health - AI ToolKit: AI Scribe Limitations and Mitigations — the known errors of AI scribes and how Island Health expects medical staff, including NPs, to catch them; also copy-and-paste guidance
- AAFP - The Promise and Pitfalls of AI in Primary Care (2024) — International source: US review (FPM 2024;31(2):27-31) of what large language models get wrong, such as fabricated content and bias, and when output is easy to verify
Where to Check Each Step of the Visit¶
| Step of the visit | What the AI produced | Check it against |
|---|---|---|
| Assessment | History questions, exam, scores, tests to order | Assessments |
| Diagnosis | A differential or a working diagnosis | Approaches and Conditions |
| Treatment and prescribing | A management plan, a drug and dose | The condition page's Key Sources and Prescribing section, and Prescribing |
| Referral | Where to refer, or whether to | The condition page's Advice and Referral section, and Populations |
| Patient education | A handout or instructions | The condition page's Patient Resources |
| Charting | The visit note | Documentation, Privacy and Records and the sources below |
Reviewing AI-Drafted Notes¶
- Northern Health - AI Ambient Scribes (2026) · FAQs (2025) — what Northern Health requires of medical staff, including NPs, before an AI-drafted note enters the record, and which scribe features are approved and not approved (decision support, summaries, recommendations)
- Doctors of BC - Practical Considerations for Using AI Scribes — Physician-focused: Doctors Technology Office guide to choosing and implementing an AI scribe, with a section on its limitations and how to mitigate them
- CMPA - AI Scribes: Answers to Frequently Asked Questions (2025) — Physician-focused: CMPA's answers on consent, accuracy of chart entries, privacy and bias; revised December 2025, with links to each provincial college's AI guidance
Checking AI Answers¶
- Evidence Packs — NPPR's list of the current provincial and Canadian documents for each condition, to load into an AI tool and to check its citations against (in development)
- AAFP - How to Write Effective Generative AI Prompts for Family Medicine (2025) — International source: US article (FPM 2025;32(5):12-16) with sample prompts for general and clinical AI tools such as OpenEvidence, and responsible prompting
Also known as: checking ai, verify ai, ai errors, hallucination, human in the loop, ai review, ai output, chatgpt, openevidence