# Record CI test results automatically GitHub Actions test results can be recorded automatically against PM on Rails acceptance scenarios. **You do not need to add a PM on Rails token or PAT to GitHub Secrets.** **You do not need a library, a CLI or a converter script either.** There are only three steps: 1. Connect the PM on Rails project to the GitHub repository with the GitHub App 2. Put the acceptance scenario's Scenario ID in the test name as `[pmr:]` 3. Upload your test runner's **JUnit XML** as an artifact whose name starts with `pmr-scenario-report` After that, PM on Rails imports the results automatically whenever the GitHub Actions workflow finishes. ```text GitHub Actions (JUnit XML) ↓ pmr-scenario-report* artifact ↓ PM on Rails reads it through the GitHub App ↓ Pass / fail is recorded on the scenario named by the test's Scenario ID ``` > **GitHub Actions does not call the PM on Rails API directly.** Normal GitHub Actions integration does not require `PM_ON_RAILS_TOKEN`. ## 1. Connect the GitHub repository Open **Settings → GitHub integration** in PM on Rails and connect the repository. The GitHub App needs **Actions: Read** permission. PM on Rails uses it to read artifacts left by GitHub Actions. See [GitHub integration](https://pmonrails.com/en/docs/github.md) for the connection steps. ## 2. Put the Scenario ID in the test name Each PM on Rails acceptance scenario has a Scenario ID. You can copy it from the scenario screen or read it with MCP using `list_scenarios` / `read_scenario`. Include that ID in the test's name as `[pmr:]`. **That is the only thing your repository has to hold.** You do not copy the scenario name, Gherkin or version into the repository — the source of truth lives in PM on Rails and is looked up by ID. | Language / runner | How | | --- | --- | | pytest (Python) | Prefer `[pmr:]` in the test name or a parametrize `ids` entry. `record_property("pmr_scenario", "")` also works, but the default `junit_family=xunit2` emits a `PytestWarning` (CI with `filterwarnings = error` will fail). Set `junit_family=xunit1` if you use `record_property` | | Pest (PHP) | `it('logs in with valid credentials [pmr:]', ...)` | | PHPUnit (PHP) | PHPUnit 11+: a named data set `#[TestWith([], '[pmr:]')]` (the JUnit name becomes `testLogin with data set "[pmr:...]"`). Older PHPUnit: put `[pmr:]` in the data-provider string key | | Jest / Vitest (TypeScript) | `it("logs in with valid credentials [pmr:]", ...)` | | go test (Go) | A subtest name: `t.Run("[pmr:]", ...)` (with gotestsum `--junitfile`) | | Anything else | Any runner works as long as `[pmr:]` ends up in the JUnit `` | If your runner cannot put it in the name, emitting `` under the `` works the same way. One test may carry several Scenario IDs. When several tests map to one Scenario, any failure makes the Scenario fail; otherwise at least one pass makes it pass; a set containing only skipped tests is recorded as skipped. ```xml ``` ### Why the Scenario ID alone is enough Every version of a PM on Rails scenario has **its own Scenario ID**. Editing the acceptance criteria creates a new version with a new ID and turns the old one into a superseded version. Agreed versions are never rewritten in place. So the ID a test reports already pins down which version was verified. There is no need to keep scenario names or Gherkin hashes on the test side for matching. ## 3. Upload the JUnit XML from GitHub Actions The artifact name **must start with `pmr-scenario-report`**. Its content is the JUnit XML exactly as your runner wrote it. ```yaml - name: Run tests # Your runner's JUnit output. pytest: --junitxml / PHPUnit: --log-junit / # Jest, Vitest: jest-junit / go: gotestsum --junitfile run: vendor/bin/pest --log-junit /tmp/junit.xml - name: Upload PM on Rails scenario report if: always() uses: actions/upload-artifact@v4 with: name: pmr-scenario-report path: /tmp/junit.xml if-no-files-found: ignore ``` For separate unit / integration jobs, suffixes are fine: ```text pmr-scenario-report-unit pmr-scenario-report-integration ``` An artifact may contain several XML files; every **JUnit** file (`testsuites` / `testsuite`) is read. Non-JUnit XML such as `coverage.xml` / `clover.xml` is skipped and the filename is left in the sync history. A single malformed XML file stops the whole run. **Each artifact may be up to 32 MB** (200 MB of reports in total once unpacked; test logs and captured output inside the JUnit are fine). **If even one artifact of a run is over the limit, nothing from that run is recorded** — recording only the part that could be read would hide the failures in the part that could not, and make the scenario look verified. The sync history names the oversized artifact; shrink or split it and run (or re-import) again. ### Why `if: always()` matters The artifact must still be uploaded when tests fail. Without it, PM on Rails cannot receive the fact that the scenario failed because the report disappears together with the failed job. ## 4. Check the imported results When the workflow finishes, PM on Rails reads the artifact and looks up each acceptance scenario by the Scenario ID in the test name. Recorded results appear under **Testing and feedback → Scenario tasks** with: - pass / fail / skipped - source “CI automated test” - workflow name - a link to the GitHub Actions run - the verified version and Gherkin (filled in from the scenario in PM on Rails) Import details are available under **Settings → GitHub integration → Sync history**. ## Scenarios that could not be recorded are commented on the pull request The import runs after the workflow has finished, so **CI itself stays green**. When a result could not be recorded — for example a test still carries an old version's ID — PM on Rails comments on the run's pull request with a table of test, scenario and reason, headed with the workflow name (the comment is written in Japanese). - There is **one comment per workflow**, rewritten on every new run (no new comments pile up). Re-importing the same run leaves the comment unchanged - Once everything is recorded, the same comment is rewritten to say that all scenarios were recorded - Runs without a pull request (a push to main, for example) and runs without an artifact are not commented on - No new permission is needed (the GitHub App's Issues: Read & write is used). If the comment fails, the results are still recorded and the sync history says the comment on PR #N failed ## Only agreed versions are recorded A TestResult is written only when the version behind that Scenario ID is **approved / active**. Anything else is skipped, and the reason is kept in the sync history. | Version state | Behaviour | | --- | --- | | Approved / active | Recorded | | Draft / in review (not agreed yet) | Not recorded. Agree on the scenario, then re-run | | Superseded (an edit created a newer version) | Not recorded. The history and the pull request comment show **the current version's Scenario ID** — review the test against the current Then, then move it to that ID. If the new version was set back to draft, the reason says it is waiting for agreement | | Rejected / deprecated / archived | Not recorded | Editing an agreed scenario marks the old version superseded and creates **a new version with a new ID** (the new version stays active). While a test still carries the old ID, its CI results are not recorded. For example, if editing the acceptance criteria produced version 4 but the test still reports the version 3 ID, that test is for an old specification and is not counted as current verification. Check that the test verifies version 4's Then, fix it if needed, switch the ID, and run CI again. **Do not just replace the ID mechanically** — a test that checked the old Then would be recorded as passing the new specification. ## Pull requests from forks are not recorded Workflow runs triggered by a pull request from a forked repository are never recorded, even when their results are correct — otherwise anyone outside the organisation could mark arbitrary scenarios as passing. Open pull requests from a branch in the same repository. A run whose `head_repository` is missing is also not recorded. Re-import via `POST /api/github/workflow-runs/{run_id}/scenario-report` (editor or above) does not apply this fork gate; not re-importing a fork run is the caller's responsibility. ## Common problems | Symptom | Check | | --- | --- | | Artifact not found | Does its name start with `pmr-scenario-report`? | | Nothing is recorded at all | Does the test name (JUnit `name`) contain `[pmr:]`? Is the file JUnit XML? | | Artifact cannot be read | Does the GitHub App have Actions: Read? | | "not recorded: scenario not agreed yet" | Has the scenario been approved? | | "superseded Scenario ID" | Did you review the test against the current Then and switch it to the current ID shown in the history / pull request comment? | | "run from a fork is not recorded" | Is the pull request coming from a branch in the same repository rather than a fork? | | "over the import limit (32 MB)" | Did you shrink or split that artifact? One oversized artifact stops the whole run | The reason is kept under **Settings → GitHub integration → Sync history**. ## Turn a CI pass into completion evidence A pass/fail-only result ("1 passed") records **which Scenario version an automated test passed against**. It is not completion evidence: the completion gate asks what was actually observed to confirm the Scenario's Then. When the test emits that observation, **every passing CI run supplies evidence for the current version automatically**. Nobody has to rewrite a TestResult afterwards. After a Scenario is revised, update the test for the new Then, replace the ID, and let CI pass. Emit it as `` under the JUnit ``. | Language / runner | How | | --- | --- | | pytest (Python) | `request.node.user_properties.append(("pmr_observed", f"the list shows {count} row for booking {number}"))` (`record_property` works too; the `junit_family` note above applies) | | Playwright (TypeScript) | `test.info().annotations.push({ type: "pmr_observed", description: `...` })` with `embedAnnotationsAsProperties: true` on the JUnit reporter | | Anything else | Any runner works as long as `` ends up directly under the `` | Three rules: - **Emit it after the assertions pass, with the values observed in that run.** Put the on-screen wording, counts or state in it, not a fixed "it passed" - **Write it in the words of the Then.** The completion gate judges this sentence by the same rules as an observation written by a person. A sentence unrelated to the Then, or one that only says the test passed, does not count - **Up to 1000 characters each.** It is not a place for logs In these cases only pass/fail is recorded, as before, and it is not completion evidence: - the test did not emit `pmr_observed` - any test mapped to the same Scenario failed (observations from the tests that passed are not used either) When the observation cannot come from a test, such as a screenshot or a check in an external service, a person or agent still records an additional TestResult. See [Testing](https://pmonrails.com/en/docs/test-results.md) for the detailed model. ## CI results cannot be rewritten A result imported from CI (origin "CI automated test") cannot have its origin changed from the Testing screen or the API, and a manually recorded result cannot be turned into a CI result later (the executor prefix `github-ci:` is reserved for the CI import). Notes and other supplementary fields can still be edited. ## CI and local development use different authentication | Use | Authentication | | --- | --- | | Record results from GitHub Actions | GitHub App; no PM on Rails token required | | Use MCP from Claude Code / local CLI | `pm-on-rails login` | **CI does not imply PAT.** There is no need to copy a human login token into GitHub Secrets for normal GitHub Actions integration. --- https://pmonrails.com/en/docs/ci-test-results