Skip to content

Warn before saving an agent rule that RuleBot can't carry out - #104807

Draft
yuwenmemon wants to merge 1 commit into
mainfrom
yuwen-683438-agentRuleSaveCheck
Draft

yuwenmemon wants to merge 1 commit into
mainfrom
yuwen-683438-agentRuleSaveCheck

Conversation

@yuwenmemon

Copy link
Copy Markdown
Contributor

Explanation of Change

Admins can type an agent rule straight into the agent rule page, and the AI rules builder never sees it. A rule that RuleBot can't carry out still saves, adds RuleBot as an admin and never runs. This checks a typed or edited rule with GenerateRule when the admin taps Create rule or Save. An impossible answer shows the describe box's refusal, and the button becomes Create anyway or Save anyway, because this only warns. Any other answer, a failed check, or no answer within 20 seconds saves as before. A suggestion or a describe flow draft that the admin leaves unchanged skips the check, because Concierge wrote it.

It's behind the describeRule beta. GenerateRule can't answer impossible until Web-Expensify#56886 merges, so until then the check only adds a wait for beta accounts.

Fixed Issues

$ https://github.com/Expensify/Expensify/issues/683438
PROPOSAL: N/A

Tests

The backend answers impossible only after Web-Expensify#56886, so the snippet below swaps the backend's answer for answer in the browser and keeps its generationID.

Precondition: Enable the describeRule beta on the test account.

  1. As a workspace admin on a Control workspace, open Rules > Agents > Add agent rule (or Add AI rule), and paste the snippet into the dev console.
  2. Run answer = {state: 'impossible', reason: 'noCapability', explanation: "Agents can't send text messages.", suggestedPrompt: 'When a report is submitted, send the approver a direct message.'}.
  3. Open the Edit tab, type "Text the approver's phone whenever a report is submitted", and tap Create rule.
  4. Verify the button loads and "Checking your rule..." replaces the disclaimer.
Checking a typed agent rule 5. Verify the rule does not save, the refusal shows under the text box, and the button says Create anyway. Refusal with Create anyway 6. Run `answer = {state: 'agentRule'}`, tap Try this instead, and verify the suggestion replaces the text, is checked, and saves. 7. Open the rule you just saved, change its text, tap Save, and verify the refusal shows with Save anyway after `answer` is impossible again. Edit page refusal with Save anyway 8. Add a character to the text and verify the refusal hides and the button says Save again. 9. Tap Save anyway on a refused rule and verify it saves. 10. Pick a suggestion on the Suggested tab and tap Create rule without changing it. Verify it saves at once with no check.
Console snippet that swaps the GenerateRule answer locally
window.answer = null;
const swap = (value) => (value?.generationID && value.state && window.answer ? {generationID: value.generationID, ...window.answer} : value);
const [set, merge, update, multiSet] = [Onyx.set, Onyx.merge, Onyx.update, Onyx.multiSet].map((method) => method.bind(Onyx));
Onyx.set = (key, value) => set(key, key === 'generatedRule' ? swap(value) : value);
Onyx.merge = (key, value) => merge(key, key === 'generatedRule' ? swap(value) : value);
Onyx.update = (updates) => update(updates.map((u) => (u.key === 'generatedRule' ? {...u, value: swap(u.value)} : u)));
Onyx.multiSet = (data) => multiSet(Object.fromEntries(Object.entries(data).map(([key, value]) => [key, key === 'generatedRule' ? swap(value) : value])));
  • Verify that no errors appear in the JS console

Offline tests

  1. Go offline, type an agent rule on the Edit tab, and tap Create rule.
  2. Verify it saves at once with no check, as it does today.

QA Steps

Precondition: Enable the describeRule beta on the test account.

Staging answers impossible only after Web-Expensify#56886. Until then, check that saving still works:

  1. Open a Control workspace > Rules > Agents > Add agent rule, type a rule on the Edit tab, and tap Create rule.
  2. Verify "Checking your rule..." shows, and the rule saves after a few seconds.
  3. Pick a suggestion, tap Create rule without changing it, and verify it saves at once.
  4. Open a saved agent rule, change its text, tap Save, and verify it saves after the check.
  • Verify that no errors appear in the JS console

PR Author Checklist

  • I linked the correct issue in the ### Fixed Issues section above
  • I wrote clear testing steps that cover the changes made in this PR
    • I added steps for local testing in the Tests section
    • I added steps for the expected offline behavior in the Offline steps section
    • I added steps for Staging and/or Production testing in the QA steps section
    • I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
    • I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
    • I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).
  • I included screenshots or videos for tests on all platforms
  • I ran the tests on all platforms & verified they passed on:
    • Android: Native
    • Android: mWeb Chrome
    • iOS: Native
    • iOS: mWeb Safari
    • MacOS: Chrome / Safari
  • I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
  • I verified any copy / text that was added to the app is either coming verbatim from figma or has been approved by marketing (in order to get marketing approval, ask the Bug Zero team member to add the Waiting for copy label to the issue)
  • If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
  • I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
  • If new assets were added or existing ones were modified, I verified that:
    • The assets are optimized and compressed (for SVG files, run npm run compress-svg)
    • The assets load correctly across all supported platforms.
  • If the PR modifies code that runs when editing or sending messages, I tested and verified there is no unexpected behavior for all supported markdown - URLs, single line code, code blocks, quotes, headings, bold, strikethrough, and italic.
  • If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
  • If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
  • If the PR modifies a component or page that can be accessed by a direct deeplink, I verified that the code functions as expected when the deeplink is used - from a logged in and logged out account.
  • If the PR modifies the UI (e.g. new buttons, new UI components, changing the padding/spacing/sizing, moving components, etc) or modifies the form input styles:
    • I verified that all the inputs inside a form are aligned with each other.
    • I added Design label and/or tagged @Expensify/design so the design team can review the changes.
  • If the PR adds or modifies the UI:
    • I asked an AI agent to review the changes for accessibility issues and addressed its findings.
    • I tested with a screen reader (VoiceOver on macOS) and verified all new/changed elements are reachable with a logical focus order.
    • I verified all new/changed elements have meaningful accessible names and roles.
    • I verified state changes are announced (e.g. checked/unchecked, expanded/collapsed, selected).
  • I added unit tests for any new feature or bug fix in this PR to help automatically prevent regressions in this user flow.
  • If the main branch was merged into this PR after a review, I tested again and verified the outcome was still expected according to the Test steps.

Screenshots/Videos

Android: Native
Android: mWeb Chrome
iOS: Native
iOS: mWeb Safari
MacOS: Chrome / Safari Refusal with Create anyway

@OSBotify

Copy link
Copy Markdown
Contributor

🦜 Polyglot Parrot! 🦜

Squawk! Looks like you added some shiny new English strings. Allow me to parrot them back to you in other tongues:

View the translation diff
diff --git a/src/languages/de.ts b/src/languages/de.ts
index c68d5bff..0b2355d4 100644
--- a/src/languages/de.ts
+++ b/src/languages/de.ts
@@ -8823,7 +8823,7 @@ Fügen Sie weitere Ausgabelimits hinzu, um den Cashflow Ihres Unternehmens zu sc
                 createRule: 'Regel erstellen',
                 createAnyway: 'Trotzdem erstellen',
                 saveAnyway: 'Trotzdem speichern',
-                checkingRule: 'Ihre Regel wird geprüft...',
+                checkingRule: 'Ihre Regel wird überprüft ...',
                 fallbackBanner:
                     'Dies ist keine Standardregel, daher haben wir sie als Agentenregel erstellt. Agentenregeln greifen, nachdem sich eine Ausgabe oder ein Bericht geändert hat, zum Beispiel indem sie die Ausgabe zurückhalten oder den Bericht ablehnen.',
                 fallbackBannerWithExplanation: (explanation: string) => `Dies ist keine Standardregel, daher haben wir sie als Agentenregel formuliert. ${explanation}`,
diff --git a/src/languages/fr.ts b/src/languages/fr.ts
index ca9efc6e..a107fdf5 100644
--- a/src/languages/fr.ts
+++ b/src/languages/fr.ts
@@ -8846,7 +8846,7 @@ Ajoutez davantage de règles de dépenses pour protéger la trésorerie de l’e
                 createRule: 'Créer une règle',
                 createAnyway: 'Créer quand même',
                 saveAnyway: 'Enregistrer quand même',
-                checkingRule: 'Vérification de votre règle...',
+                checkingRule: 'Vérification de votre règle…',
                 fallbackBanner:
                     'Ce n’est pas une règle standard, nous l’avons donc rédigée comme une règle d’agent. Les règles d’agent agissent après la modification d’une dépense ou d’une note de frais, par exemple en bloquant la dépense ou en rejetant la note de frais.',
                 fallbackBannerWithExplanation: (explanation: string) => `Ce n’est pas une règle standard, nous l’avons donc rédigée comme une règle d’agent. ${explanation}`,
diff --git a/src/languages/ja.ts b/src/languages/ja.ts
index 87b560c1..56b1861f 100644
--- a/src/languages/ja.ts
+++ b/src/languages/ja.ts
@@ -8661,9 +8661,9 @@ ${reportName}`,
                 describeRuleForConcierge: 'ルールの内容を入力すると、Concierge が自動作成します',
                 gotIt: '了解しました',
                 createRule: 'ルールを作成',
-                createAnyway: 'このまま作成',
+                createAnyway: 'それでも作成',
                 saveAnyway: 'このまま保存',
-                checkingRule: 'ルールを確認しています...',
+                checkingRule: 'ルールを確認しています…',
                 fallbackBanner:
                     'これは標準ルールではないため、エージェントルールとして作成しました。エージェントルールは、経費やレポートに変更があった後に動作し、例えば経費を保留にしたり、レポートを却下したりします。',
                 fallbackBannerWithExplanation: (explanation: string) => `これは標準ルールではないため、エージェントルールとして作成しました。${explanation}`,
diff --git a/src/languages/pl.ts b/src/languages/pl.ts
index 1e9357e7..a5d6be34 100644
--- a/src/languages/pl.ts
+++ b/src/languages/pl.ts
@@ -8778,7 +8778,7 @@ Dodaj więcej zasad wydatków, żeby chronić płynność finansową firmy.`,
                 createRule: 'Utwórz regułę',
                 createAnyway: 'Utwórz mimo to',
                 saveAnyway: 'Zapisz mimo to',
-                checkingRule: 'Sprawdzanie reguły...',
+                checkingRule: 'Sprawdzanie twojej reguły...',
                 fallbackBanner:
                     'To nie jest standardowa reguła, więc przygotowaliśmy ją jako regułę agenta. Reguły agenta działają po zmianie wydatku lub raportu, na przykład wstrzymując wydatek albo odrzucając raport.',
                 fallbackBannerWithExplanation: (explanation: string) => `To nie jest standardowa reguła, więc przygotowaliśmy ją jako regułę agenta. ${explanation}`,
diff --git a/src/languages/zh-hans.ts b/src/languages/zh-hans.ts
index 50b347c7..2ad2a549 100644
--- a/src/languages/zh-hans.ts
+++ b/src/languages/zh-hans.ts
@@ -8452,7 +8452,7 @@ ${reportName}`,
                 createRule: '创建规则',
                 createAnyway: '仍然创建',
                 saveAnyway: '仍然保存',
-                checkingRule: '正在检查你的规则...',
+                checkingRule: '正在检查您的规则…',
                 fallbackBanner: '这不是一条标准规则,所以我们将其制定为代理规则。代理规则会在报销或报表发生变更后生效,例如暂扣该报销或驳回该报表。',
                 fallbackBannerWithExplanation: (explanation: string) => `这不是一条标准规则,因此我们将其起草为代理规则。${explanation}`,
             },

Note

You can apply these changes to your branch by copying the patch to your clipboard, then running pbpaste | git apply 😉

View workflow run

@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Looks like you've decreased code coverage for some files. Please write tests to increase, or at least maintain, the existing level of code coverage. See our documentation here for how to interpret this table.

Files with missing lines Coverage Δ
src/CONST/index.ts 95.90% <ø> (ø)
src/libs/actions/Policy/Rules.ts 79.57% <100.00%> (+1.38%) ⬆️
...es/workspace/rules/AgentRules/AddAgentRulePage.tsx 90.78% <100.00%> (+0.24%) ⬆️
...orkspace/rules/AgentRules/AddAgentRuleWriteTab.tsx 82.35% <100.00%> (+82.35%) ⬆️
...s/workspace/rules/AgentRules/EditAgentRulePage.tsx 82.14% <100.00%> (+82.14%) ⬆️
src/pages/workspace/rules/DescribeRuleForm.tsx 93.61% <ø> (-0.39%) ⬇️
src/pages/workspace/rules/ImpossibleRuleNotice.tsx 100.00% <100.00%> (ø)
...kspace/rules/AgentRules/useAgentRulePromptCheck.ts 97.22% <97.22%> (ø)
... and 17 files with indirect coverage changes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants