nimmo/forgejo-auto-pr-review
Watch
1
0
Fork
You've already forked forgejo-auto-pr-review
 0
Code Issues 5 Pull requests Projects Releases Packages 1 Activity Actions

Implement a reusable AI-assisted pull request review workflow for Forgejo #1

New issue
Open
opened 2026-07-20 13:37:59 +01:00 by nimmo · 0 comments
nimmo commented 2026-07-20 13:37:59 +01:00
Owner
Copy link

Implement a reusable AI-assisted pull request review workflow for Forgejo

Summary

Create an automated code-review system for Forgejo repositories.

When a pull request is opened or updated, the system should:

  1. Collect the pull request description, changed code and relevant repository guidance.
  2. Run appropriate deterministic checks and static-analysis tools.
  3. Ask a configurable large language model to review the proposed changes.
  4. Optionally perform a second validation pass over the initial review.
  5. Publish a structured review on the Forgejo pull request.
  6. Leave final approval and merge decisions to a human reviewer.

The implementation should be designed as a reusable tool rather than being tightly coupled to one repository.

Motivation

The objective is to provide an additional automated review layer that can identify:

  • Incomplete implementations.
  • Behaviour that does not match the stated objective.
  • Likely bugs and unintended side effects.
  • Security concerns.
  • Missing tests or documentation.
  • Maintainability and compatibility problems.
  • Changes that contradict repository-specific requirements.

This system is intended to assist human reviewers, not replace them.

Primary goals

The implementation should:

  • Work with Forgejo pull requests.
  • Run when a pull request is opened, reopened or updated with new commits.
  • Support manual reruns.
  • Work with self-hosted and hosted language models.
  • Support Ollama as an initial model provider.
  • Keep the model-provider implementation replaceable.
  • Load review guidance from the repository being reviewed.
  • Combine deterministic analysis with LLM-based reasoning.
  • Produce concise, evidence-based findings.
  • Avoid repeatedly posting the same findings.
  • Be reusable across multiple Forgejo repositories.
  • Operate without granting an LLM direct access to execute code or modify the repository.

Non-goals

The first version does not need to:

  • Automatically approve or merge pull requests.
  • Automatically modify source code.
  • Replace normal continuous-integration testing.
  • Replace human security review.
  • Support every programming language and static-analysis tool.
  • Create perfect line-level Forgejo review comments.
  • Act as an autonomous coding agent.

Posting a single structured pull-request comment is acceptable for the initial version.

Proposed workflow

  1. Trigger the review

The review should run for Forgejo events corresponding to:

  • Pull request opened.
  • Pull request reopened.
  • New commits pushed to an existing pull request.
  • Draft pull request marked ready for review.
  • Manual workflow execution.

The implementation should confirm the exact event names supported by the deployed Forgejo version.

Reviews of draft pull requests should be configurable. By default, drafts may either be skipped or given a reduced preliminary review.

  1. Collect pull-request context

The context collector should gather:

  • Repository name.
  • Base and head branches.
  • Pull request number.
  • Pull request title.
  • Pull request description.
  • Commit messages included in the pull request.
  • List of changed files.
  • Unified diff or individual file patches.
  • Relevant test and static-analysis results.
  • Existing automated-review comment, when one exists.
  • Optional linked issue details when they can be obtained safely and reliably.

The pull request description should be treated as the primary statement of the intended objective.

Where the description does not clearly explain the objective, the review should explicitly state that the intended behaviour could not be fully verified.

  1. Load repository guidance

The reviewer should discover repository-specific instructions.

Suggested lookup order:

  1. ".forgejo/ai-review.md"
  2. ".forgejo/ai-review.yml"
  3. "AGENTS.md"
  4. "CONTRIBUTING.md"
  5. "SECURITY.md"
  6. Relevant files under "docs/"
  7. "README.md"

The repository should be able to define:

  • Architectural principles.
  • Required tests.
  • Supported platforms and versions.
  • Security expectations.
  • Coding conventions.
  • Files or directories that require special attention.
  • Generated or vendored files that should not be reviewed.
  • Acceptable review categories.
  • Severity definitions.
  • Language-specific checks.
  • Maximum context and diff sizes.
  • Whether a second review pass is required.

Instructions nearest to a changed file may optionally take precedence over repository-level instructions, similar to scoped "AGENTS.md" files.

  1. Run deterministic checks

The system should run configured checks before calling the model.

Examples include:

  • Existing repository test commands.
  • Formatting and linting.
  • Type checking.
  • Secret scanning.
  • Dependency vulnerability scanning.
  • Semgrep or another static-analysis tool.
  • Language-specific security tools.
  • Nix flake checks, where applicable.
  • Detection of unexpectedly large binaries or generated files.

The framework should not hard-code all tools. Repositories should be able to supply commands or select supported integrations.

Results should be normalised into a common structure containing:

  • Tool name.
  • Success, warning or failure state.
  • File and line information where available.
  • Human-readable message.
  • Machine-readable severity.
  • Relevant output excerpt.

The LLM should receive summarised results rather than unlimited raw command output.

Deterministic tool failures must not be silently converted into LLM opinions.

  1. Build the review request

The prompt builder should combine:

  • Fixed system-level review instructions.
  • Repository guidance.
  • Pull request objective.
  • Changed-file summary.
  • Relevant diff content.
  • Deterministic analysis results.
  • Required output format.

The review should ask the model to evaluate:

Objective completion

  • Does the implementation appear to fulfil the pull request description?
  • Are any stated requirements missing?
  • Are there placeholder implementations, unfinished branches or TODO items?
  • Are likely edge cases omitted?

Correctness

  • Are there logic errors?
  • Could the change cause regressions?
  • Are failure cases handled?
  • Are concurrency, state-management or lifecycle issues present?
  • Do tests meaningfully validate the new behaviour?

Security

  • Does the change weaken authentication, authorisation or validation?
  • Could untrusted input reach dangerous operations?
  • Are secrets or sensitive values exposed?
  • Are dependency or supply-chain concerns introduced?
  • Could error messages or logs expose sensitive information?

Maintainability

  • Is the implementation unnecessarily complex?
  • Does it follow existing repository patterns?
  • Is duplicated behaviour being introduced?
  • Are public interfaces and configuration changes documented?
  • Are comments and names accurate?

Compatibility and operations

  • Could supported platforms or versions be broken?
  • Are migrations, deployment changes or rollback considerations missing?
  • Could the change cause unacceptable resource usage?
  • Are configuration defaults safe?
  1. Validate the initial review

Support an optional second review pass.

The second pass should critique the initial findings and remove or downgrade comments that are:

  • Unsupported by the supplied diff.
  • Purely speculative.
  • Duplicates of deterministic-tool output.
  • Style preferences not required by repository guidance.
  • Based on files or context the model has not seen.
  • Unrelated to the pull request objective.
  • Too vague to be actionable.

The second pass may use:

  • The same model with a separate validation prompt.
  • A different model.
  • A deterministic filtering stage combined with an LLM critique.

The validated result should contain only the final findings.

  1. Publish the review

The system should publish or update a clearly identifiable Forgejo pull-request comment.

The comment should include:

  • Review status.
  • Scope of files reviewed.
  • Checks that were run.
  • Summary of the implementation.
  • Findings grouped by severity.
  • Questions or missing context.
  • Test and documentation observations.
  • Limitations of the review.
  • Model and reviewer version metadata in a collapsible section.

Suggested output structure:

Automated code review

Overall assessment

One or two paragraphs describing whether the implementation appears to meet
the stated objective.

Blocking findings

Findings that are likely to cause incorrect behaviour, security problems,
data loss or failure to meet an explicit requirement.

Important findings

Significant issues that should normally be addressed before merging.

Suggestions

Non-blocking maintainability, testing or documentation improvements.

Tests and validation

Summary of automated checks, missing tests and areas that could not be
verified.

Review limitations

Context that was unavailable, files that were omitted and any analysis that
could not be completed.

Each finding should contain:

  • Severity.
  • Short title.
  • File path.
  • Line or diff location where available.
  • Explanation.
  • Evidence from the change.
  • Likely impact.
  • A practical remediation suggestion.
  • Confidence level.

The system should avoid claiming certainty when the available context is insufficient.

  1. Update rather than spam

The tool should maintain one primary automated-review comment per pull request.

When new commits are pushed, it should:

  • Run the review again.
  • Update the existing comment where possible.
  • Identify the reviewed commit SHA.
  • Remove findings that have been fixed.
  • Preserve a limited review history or link findings to the relevant revision.
  • Avoid reposting identical findings.

A stable hidden marker may be placed in the comment so the tool can identify its previous output.

Finding severity

Use a small, well-defined severity scale.

Blocking

A high-confidence issue that could cause:

  • A serious security vulnerability.
  • Data loss or corruption.
  • Incorrect core behaviour.
  • Failure to meet an explicit requirement.
  • A production-breaking regression.

Important

A credible issue that materially affects:

  • Reliability.
  • Error handling.
  • Compatibility.
  • Test coverage.
  • Maintainability.
  • Operational safety.

Suggestion

A non-blocking improvement that would make the implementation clearer, safer or easier to maintain.

Stylistic preferences should not be reported unless they contradict documented repository standards.

Confidence

Each finding should include a confidence rating:

  • High: directly demonstrated by the supplied code or tool output.
  • Medium: strongly suggested but dependent on context not fully available.
  • Low: plausible but insufficiently supported.

Low-confidence findings should normally be placed in a questions or limitations section rather than presented as defects.

Model-provider abstraction

Model access should use a provider-independent interface.

The initial implementation should support Ollama, with configuration for:

  • Base URL.
  • Model name.
  • Context size.
  • Temperature.
  • Request timeout.
  • Maximum output tokens.

The design should make it straightforward to add providers such as:

  • OpenAI-compatible APIs.
  • Anthropic.
  • Google.
  • Other local inference servers.

Provider-specific code should not be mixed into:

  • Forgejo event handling.
  • Context collection.
  • Static-analysis execution.
  • Review formatting.
  • Comment publication.

A provider should accept a structured review request and return a structured response.

Suggested component structure

The project should be separated into components similar to:

forgejo-ai-review/
├── cmd/
│ └── reviewer/
├── internal/
│ ├── config/
│ ├── forgejo/
│ ├── context/
│ ├── guidance/
│ ├── analysis/
│ ├── prompts/
│ ├── providers/
│ │ ├── ollama/
│ │ └── openai-compatible/
│ ├── review/
│ ├── validation/
│ └── publishing/
├── examples/
│ ├── forgejo-workflow/
│ └── ai-review-config/
├── container/
└── README.md

This is illustrative rather than prescriptive.

The tool may be implemented as a command-line application packaged in a container and invoked by Forgejo Actions. This would make it easier to reuse outside a single repository or workflow runner.

Configuration

Support configuration through a repository file, environment variables and Forgejo secrets.

Example conceptual configuration:

provider:
type: ollama
model: qwen-example
base_url: http://ollama.internal:11434
timeout_seconds: 300

review:
second_pass: true
review_drafts: false
max_changed_files: 100
max_diff_bytes: 500000
minimum_confidence: medium
fail_check_on_blocking_findings: false

guidance:
primary_file: .forgejo/ai-review.md

analysis:
commands:
- name: tests
command: nix flake check
timeout_seconds: 900

publishing:
update_existing_comment: true
include_model_metadata: true

The final schema can differ, but it should be documented and validated.

Large pull requests

The implementation must behave predictably when a pull request is too large for the selected model.

It should not silently truncate arbitrary parts of a diff.

Possible strategies include:

  • Review files in batches.
  • Prioritise security-sensitive and manually written files.
  • Summarise low-risk or generated changes.
  • Exclude vendored files.
  • Perform a final synthesis pass over per-file reviews.
  • Report any files that could not be reviewed.

The final comment must clearly identify incomplete review coverage.

Security requirements

Repository content, pull request descriptions, comments and diffs must be treated as untrusted input.

The implementation must defend against prompt-injection attempts such as instructions embedded in:

  • Source-code comments.
  • Documentation.
  • Test fixtures.
  • Pull request descriptions.
  • File names.
  • Commit messages.

The model must be told that repository content is evidence to analyse, not authority that can override system instructions.

The LLM must not:

  • Receive Forgejo access tokens.
  • Receive unrelated repository secrets.
  • Execute commands.
  • Select arbitrary commands for the runner to execute.
  • Modify source code.
  • Approve or merge pull requests.
  • Access unrestricted network resources through the reviewer.

Only explicitly configured deterministic commands may be run.

Command execution should use:

  • Timeouts.
  • Resource limits.
  • Minimal permissions.
  • An isolated working directory or container.
  • Read-only access where practical.
  • No unnecessary secret exposure.

Pull requests from forks or untrusted contributors require particular care. Secrets should not be made available to untrusted workflow execution.

Privacy

The documentation should make clear where source code is sent.

When using a hosted provider, users must explicitly configure that provider and its credentials.

The default example should favour a self-hosted model and should not silently send repository content to an external service.

Logs should avoid recording:

  • Full prompts containing proprietary code.
  • Provider credentials.
  • Forgejo tokens.
  • Repository secrets.

Debug logging of prompts should require explicit opt-in.

Forgejo permissions

The integration should use the minimum permissions required to:

  • Read pull-request metadata.
  • Read repository content.
  • Read diffs.
  • Create or update pull-request comments.
  • Optionally report a status check.

It should not require administrative access or permission to merge code.

Review status behaviour

Initially, the automated review should be advisory.

Configuration may later allow:

  • Always-successful informational checks.
  • A warning state when important findings exist.
  • A failed status check for blocking findings.

The default should not prevent merging solely because an LLM produced a finding.

Deterministic test failures may continue to fail through the repository's normal CI workflow.

Observability

Record enough information to diagnose failures without exposing sensitive code.

Useful metadata includes:

  • Pull request number.
  • Reviewed commit SHA.
  • Reviewer version.
  • Provider type.
  • Model name.
  • Number of changed files.
  • Number of files reviewed.
  • Static-analysis status.
  • Number of findings by severity.
  • Prompt and response token counts where available.
  • Execution duration.
  • Whether context was truncated or batched.

Testing requirements

The project should include tests for:

  • Forgejo event parsing.
  • Pull-request context collection.
  • Guidance-file precedence.
  • Configuration validation.
  • Diff-size handling.
  • Provider abstraction.
  • Ollama response parsing.
  • Invalid or incomplete model responses.
  • Finding deduplication.
  • Existing-comment discovery and update.
  • Prompt-injection resistance.
  • Redaction of secrets from logs.
  • Second-pass finding validation.

Use recorded fixtures or a mock Forgejo server for integration tests where possible.

Model calls should be mockable so the main test suite does not require a running LLM.

Documentation requirements

Provide documentation covering:

  • Project purpose and limitations.
  • Installation.
  • Forgejo Action setup.
  • Forgejo token permissions.
  • Ollama configuration.
  • Repository guidance files.
  • Configuration reference.
  • Supported static-analysis integrations.
  • Security considerations.
  • Fork pull-request behaviour.
  • Troubleshooting.
  • Example review output.
  • How to add another model provider.
  • How to run the reviewer locally against a pull-request diff.

Deliverables

The initial implementation should include:

  • A reusable review application or container.
  • Forgejo API integration.
  • Forgejo Actions workflow example.
  • Ollama model provider.
  • Repository-guidance discovery.
  • Configurable deterministic checks.
  • Structured initial review.
  • Optional second-pass validation.
  • Pull-request comment creation and updating.
  • Tests.
  • Documentation.
  • Example configuration files.

Acceptance criteria

The issue is complete when:

  • Opening a test pull request starts the review workflow.
  • Pushing another commit reruns the review.
  • The workflow reads repository-specific review guidance.
  • Configured deterministic checks are executed.
  • The selected model receives the pull-request objective, relevant diff and check results.
  • The reviewer returns a structured, validated result.
  • A Forgejo comment is created or updated.
  • Findings reference evidence from the changed code where possible.
  • Fixed findings disappear after a later review.
  • Oversized changes are handled without silent truncation.
  • Secrets are not included in prompts or logs.
  • The LLM cannot cause arbitrary commands to be executed.
  • Model-provider code can be replaced without rewriting the Forgejo integration.
  • The tool can be configured for use in a second repository without source-code changes.
  • Documentation explains that the review is advisory and requires human judgement.

Suggested implementation phases

Phase 1: Minimum viable reviewer

  • Forgejo pull-request event handling.
  • Diff and metadata collection.
  • Repository guidance loading.
  • Ollama integration.
  • One structured pull-request comment.
  • Basic size limits and error reporting.

Phase 2: Analysis and validation

  • Configurable static-analysis commands.
  • Normalised analysis results.
  • Second-pass review validation.
  • Finding deduplication.
  • Existing-comment updates.

Phase 3: Reusability and hardening

  • Container packaging.
  • Provider plugin interface.
  • Additional provider implementation.
  • Improved batching for large pull requests.
  • Security hardening.
  • Status checks and metrics.
  • Expanded documentation and examples.

Open design questions

The implementation should document decisions for the following:

  • Which implementation language best suits the Forgejo integration and distribution model?
  • Should the tool primarily operate as a Forgejo Action, webhook service or standalone CLI?
  • Which Qwen or other local model provides acceptable code-review quality on the available hardware?
  • Should reviews run on draft pull requests?
  • Should blocking findings ever fail a status check?
  • How should linked issue objectives be obtained?
  • Should line-level comments be added after the initial version?
  • How should old review history be retained?
  • Which static-analysis tools should be supported directly?
  • Should repository owners be able to define different rules for different paths?
  • What limits should apply to model cost, tokens and execution time?

Definition of success

A successful first version should reliably provide a useful additional review signal without creating excessive noise.

It should make clear:

  • What was reviewed.
  • What checks were performed.
  • What appears incomplete or unsafe.
  • What evidence supports each finding.
  • What could not be verified.
  • That a human remains responsible for the final review and merge decision.
Implement a reusable AI-assisted pull request review workflow for Forgejo Summary Create an automated code-review system for Forgejo repositories. When a pull request is opened or updated, the system should: 1. Collect the pull request description, changed code and relevant repository guidance. 2. Run appropriate deterministic checks and static-analysis tools. 3. Ask a configurable large language model to review the proposed changes. 4. Optionally perform a second validation pass over the initial review. 5. Publish a structured review on the Forgejo pull request. 6. Leave final approval and merge decisions to a human reviewer. The implementation should be designed as a reusable tool rather than being tightly coupled to one repository. Motivation The objective is to provide an additional automated review layer that can identify: - Incomplete implementations. - Behaviour that does not match the stated objective. - Likely bugs and unintended side effects. - Security concerns. - Missing tests or documentation. - Maintainability and compatibility problems. - Changes that contradict repository-specific requirements. This system is intended to assist human reviewers, not replace them. Primary goals The implementation should: - Work with Forgejo pull requests. - Run when a pull request is opened, reopened or updated with new commits. - Support manual reruns. - Work with self-hosted and hosted language models. - Support Ollama as an initial model provider. - Keep the model-provider implementation replaceable. - Load review guidance from the repository being reviewed. - Combine deterministic analysis with LLM-based reasoning. - Produce concise, evidence-based findings. - Avoid repeatedly posting the same findings. - Be reusable across multiple Forgejo repositories. - Operate without granting an LLM direct access to execute code or modify the repository. Non-goals The first version does not need to: - Automatically approve or merge pull requests. - Automatically modify source code. - Replace normal continuous-integration testing. - Replace human security review. - Support every programming language and static-analysis tool. - Create perfect line-level Forgejo review comments. - Act as an autonomous coding agent. Posting a single structured pull-request comment is acceptable for the initial version. Proposed workflow 1. Trigger the review The review should run for Forgejo events corresponding to: - Pull request opened. - Pull request reopened. - New commits pushed to an existing pull request. - Draft pull request marked ready for review. - Manual workflow execution. The implementation should confirm the exact event names supported by the deployed Forgejo version. Reviews of draft pull requests should be configurable. By default, drafts may either be skipped or given a reduced preliminary review. 2. Collect pull-request context The context collector should gather: - Repository name. - Base and head branches. - Pull request number. - Pull request title. - Pull request description. - Commit messages included in the pull request. - List of changed files. - Unified diff or individual file patches. - Relevant test and static-analysis results. - Existing automated-review comment, when one exists. - Optional linked issue details when they can be obtained safely and reliably. The pull request description should be treated as the primary statement of the intended objective. Where the description does not clearly explain the objective, the review should explicitly state that the intended behaviour could not be fully verified. 3. Load repository guidance The reviewer should discover repository-specific instructions. Suggested lookup order: 1. ".forgejo/ai-review.md" 2. ".forgejo/ai-review.yml" 3. "AGENTS.md" 4. "CONTRIBUTING.md" 5. "SECURITY.md" 6. Relevant files under "docs/" 7. "README.md" The repository should be able to define: - Architectural principles. - Required tests. - Supported platforms and versions. - Security expectations. - Coding conventions. - Files or directories that require special attention. - Generated or vendored files that should not be reviewed. - Acceptable review categories. - Severity definitions. - Language-specific checks. - Maximum context and diff sizes. - Whether a second review pass is required. Instructions nearest to a changed file may optionally take precedence over repository-level instructions, similar to scoped "AGENTS.md" files. 4. Run deterministic checks The system should run configured checks before calling the model. Examples include: - Existing repository test commands. - Formatting and linting. - Type checking. - Secret scanning. - Dependency vulnerability scanning. - Semgrep or another static-analysis tool. - Language-specific security tools. - Nix flake checks, where applicable. - Detection of unexpectedly large binaries or generated files. The framework should not hard-code all tools. Repositories should be able to supply commands or select supported integrations. Results should be normalised into a common structure containing: - Tool name. - Success, warning or failure state. - File and line information where available. - Human-readable message. - Machine-readable severity. - Relevant output excerpt. The LLM should receive summarised results rather than unlimited raw command output. Deterministic tool failures must not be silently converted into LLM opinions. 5. Build the review request The prompt builder should combine: - Fixed system-level review instructions. - Repository guidance. - Pull request objective. - Changed-file summary. - Relevant diff content. - Deterministic analysis results. - Required output format. The review should ask the model to evaluate: Objective completion - Does the implementation appear to fulfil the pull request description? - Are any stated requirements missing? - Are there placeholder implementations, unfinished branches or TODO items? - Are likely edge cases omitted? Correctness - Are there logic errors? - Could the change cause regressions? - Are failure cases handled? - Are concurrency, state-management or lifecycle issues present? - Do tests meaningfully validate the new behaviour? Security - Does the change weaken authentication, authorisation or validation? - Could untrusted input reach dangerous operations? - Are secrets or sensitive values exposed? - Are dependency or supply-chain concerns introduced? - Could error messages or logs expose sensitive information? Maintainability - Is the implementation unnecessarily complex? - Does it follow existing repository patterns? - Is duplicated behaviour being introduced? - Are public interfaces and configuration changes documented? - Are comments and names accurate? Compatibility and operations - Could supported platforms or versions be broken? - Are migrations, deployment changes or rollback considerations missing? - Could the change cause unacceptable resource usage? - Are configuration defaults safe? 6. Validate the initial review Support an optional second review pass. The second pass should critique the initial findings and remove or downgrade comments that are: - Unsupported by the supplied diff. - Purely speculative. - Duplicates of deterministic-tool output. - Style preferences not required by repository guidance. - Based on files or context the model has not seen. - Unrelated to the pull request objective. - Too vague to be actionable. The second pass may use: - The same model with a separate validation prompt. - A different model. - A deterministic filtering stage combined with an LLM critique. The validated result should contain only the final findings. 7. Publish the review The system should publish or update a clearly identifiable Forgejo pull-request comment. The comment should include: - Review status. - Scope of files reviewed. - Checks that were run. - Summary of the implementation. - Findings grouped by severity. - Questions or missing context. - Test and documentation observations. - Limitations of the review. - Model and reviewer version metadata in a collapsible section. Suggested output structure: ## Automated code review ### Overall assessment One or two paragraphs describing whether the implementation appears to meet the stated objective. ### Blocking findings Findings that are likely to cause incorrect behaviour, security problems, data loss or failure to meet an explicit requirement. ### Important findings Significant issues that should normally be addressed before merging. ### Suggestions Non-blocking maintainability, testing or documentation improvements. ### Tests and validation Summary of automated checks, missing tests and areas that could not be verified. ### Review limitations Context that was unavailable, files that were omitted and any analysis that could not be completed. Each finding should contain: - Severity. - Short title. - File path. - Line or diff location where available. - Explanation. - Evidence from the change. - Likely impact. - A practical remediation suggestion. - Confidence level. The system should avoid claiming certainty when the available context is insufficient. 8. Update rather than spam The tool should maintain one primary automated-review comment per pull request. When new commits are pushed, it should: - Run the review again. - Update the existing comment where possible. - Identify the reviewed commit SHA. - Remove findings that have been fixed. - Preserve a limited review history or link findings to the relevant revision. - Avoid reposting identical findings. A stable hidden marker may be placed in the comment so the tool can identify its previous output. Finding severity Use a small, well-defined severity scale. Blocking A high-confidence issue that could cause: - A serious security vulnerability. - Data loss or corruption. - Incorrect core behaviour. - Failure to meet an explicit requirement. - A production-breaking regression. Important A credible issue that materially affects: - Reliability. - Error handling. - Compatibility. - Test coverage. - Maintainability. - Operational safety. Suggestion A non-blocking improvement that would make the implementation clearer, safer or easier to maintain. Stylistic preferences should not be reported unless they contradict documented repository standards. Confidence Each finding should include a confidence rating: - High: directly demonstrated by the supplied code or tool output. - Medium: strongly suggested but dependent on context not fully available. - Low: plausible but insufficiently supported. Low-confidence findings should normally be placed in a questions or limitations section rather than presented as defects. Model-provider abstraction Model access should use a provider-independent interface. The initial implementation should support Ollama, with configuration for: - Base URL. - Model name. - Context size. - Temperature. - Request timeout. - Maximum output tokens. The design should make it straightforward to add providers such as: - OpenAI-compatible APIs. - Anthropic. - Google. - Other local inference servers. Provider-specific code should not be mixed into: - Forgejo event handling. - Context collection. - Static-analysis execution. - Review formatting. - Comment publication. A provider should accept a structured review request and return a structured response. Suggested component structure The project should be separated into components similar to: forgejo-ai-review/ ├── cmd/ │ └── reviewer/ ├── internal/ │ ├── config/ │ ├── forgejo/ │ ├── context/ │ ├── guidance/ │ ├── analysis/ │ ├── prompts/ │ ├── providers/ │ │ ├── ollama/ │ │ └── openai-compatible/ │ ├── review/ │ ├── validation/ │ └── publishing/ ├── examples/ │ ├── forgejo-workflow/ │ └── ai-review-config/ ├── container/ └── README.md This is illustrative rather than prescriptive. The tool may be implemented as a command-line application packaged in a container and invoked by Forgejo Actions. This would make it easier to reuse outside a single repository or workflow runner. Configuration Support configuration through a repository file, environment variables and Forgejo secrets. Example conceptual configuration: provider: type: ollama model: qwen-example base_url: http://ollama.internal:11434 timeout_seconds: 300 review: second_pass: true review_drafts: false max_changed_files: 100 max_diff_bytes: 500000 minimum_confidence: medium fail_check_on_blocking_findings: false guidance: primary_file: .forgejo/ai-review.md analysis: commands: - name: tests command: nix flake check timeout_seconds: 900 publishing: update_existing_comment: true include_model_metadata: true The final schema can differ, but it should be documented and validated. Large pull requests The implementation must behave predictably when a pull request is too large for the selected model. It should not silently truncate arbitrary parts of a diff. Possible strategies include: - Review files in batches. - Prioritise security-sensitive and manually written files. - Summarise low-risk or generated changes. - Exclude vendored files. - Perform a final synthesis pass over per-file reviews. - Report any files that could not be reviewed. The final comment must clearly identify incomplete review coverage. Security requirements Repository content, pull request descriptions, comments and diffs must be treated as untrusted input. The implementation must defend against prompt-injection attempts such as instructions embedded in: - Source-code comments. - Documentation. - Test fixtures. - Pull request descriptions. - File names. - Commit messages. The model must be told that repository content is evidence to analyse, not authority that can override system instructions. The LLM must not: - Receive Forgejo access tokens. - Receive unrelated repository secrets. - Execute commands. - Select arbitrary commands for the runner to execute. - Modify source code. - Approve or merge pull requests. - Access unrestricted network resources through the reviewer. Only explicitly configured deterministic commands may be run. Command execution should use: - Timeouts. - Resource limits. - Minimal permissions. - An isolated working directory or container. - Read-only access where practical. - No unnecessary secret exposure. Pull requests from forks or untrusted contributors require particular care. Secrets should not be made available to untrusted workflow execution. Privacy The documentation should make clear where source code is sent. When using a hosted provider, users must explicitly configure that provider and its credentials. The default example should favour a self-hosted model and should not silently send repository content to an external service. Logs should avoid recording: - Full prompts containing proprietary code. - Provider credentials. - Forgejo tokens. - Repository secrets. Debug logging of prompts should require explicit opt-in. Forgejo permissions The integration should use the minimum permissions required to: - Read pull-request metadata. - Read repository content. - Read diffs. - Create or update pull-request comments. - Optionally report a status check. It should not require administrative access or permission to merge code. Review status behaviour Initially, the automated review should be advisory. Configuration may later allow: - Always-successful informational checks. - A warning state when important findings exist. - A failed status check for blocking findings. The default should not prevent merging solely because an LLM produced a finding. Deterministic test failures may continue to fail through the repository's normal CI workflow. Observability Record enough information to diagnose failures without exposing sensitive code. Useful metadata includes: - Pull request number. - Reviewed commit SHA. - Reviewer version. - Provider type. - Model name. - Number of changed files. - Number of files reviewed. - Static-analysis status. - Number of findings by severity. - Prompt and response token counts where available. - Execution duration. - Whether context was truncated or batched. Testing requirements The project should include tests for: - Forgejo event parsing. - Pull-request context collection. - Guidance-file precedence. - Configuration validation. - Diff-size handling. - Provider abstraction. - Ollama response parsing. - Invalid or incomplete model responses. - Finding deduplication. - Existing-comment discovery and update. - Prompt-injection resistance. - Redaction of secrets from logs. - Second-pass finding validation. Use recorded fixtures or a mock Forgejo server for integration tests where possible. Model calls should be mockable so the main test suite does not require a running LLM. Documentation requirements Provide documentation covering: - Project purpose and limitations. - Installation. - Forgejo Action setup. - Forgejo token permissions. - Ollama configuration. - Repository guidance files. - Configuration reference. - Supported static-analysis integrations. - Security considerations. - Fork pull-request behaviour. - Troubleshooting. - Example review output. - How to add another model provider. - How to run the reviewer locally against a pull-request diff. Deliverables The initial implementation should include: - A reusable review application or container. - Forgejo API integration. - Forgejo Actions workflow example. - Ollama model provider. - Repository-guidance discovery. - Configurable deterministic checks. - Structured initial review. - Optional second-pass validation. - Pull-request comment creation and updating. - Tests. - Documentation. - Example configuration files. Acceptance criteria The issue is complete when: - Opening a test pull request starts the review workflow. - Pushing another commit reruns the review. - The workflow reads repository-specific review guidance. - Configured deterministic checks are executed. - The selected model receives the pull-request objective, relevant diff and check results. - The reviewer returns a structured, validated result. - A Forgejo comment is created or updated. - Findings reference evidence from the changed code where possible. - Fixed findings disappear after a later review. - Oversized changes are handled without silent truncation. - Secrets are not included in prompts or logs. - The LLM cannot cause arbitrary commands to be executed. - Model-provider code can be replaced without rewriting the Forgejo integration. - The tool can be configured for use in a second repository without source-code changes. - Documentation explains that the review is advisory and requires human judgement. Suggested implementation phases Phase 1: Minimum viable reviewer - Forgejo pull-request event handling. - Diff and metadata collection. - Repository guidance loading. - Ollama integration. - One structured pull-request comment. - Basic size limits and error reporting. Phase 2: Analysis and validation - Configurable static-analysis commands. - Normalised analysis results. - Second-pass review validation. - Finding deduplication. - Existing-comment updates. Phase 3: Reusability and hardening - Container packaging. - Provider plugin interface. - Additional provider implementation. - Improved batching for large pull requests. - Security hardening. - Status checks and metrics. - Expanded documentation and examples. Open design questions The implementation should document decisions for the following: - Which implementation language best suits the Forgejo integration and distribution model? - Should the tool primarily operate as a Forgejo Action, webhook service or standalone CLI? - Which Qwen or other local model provides acceptable code-review quality on the available hardware? - Should reviews run on draft pull requests? - Should blocking findings ever fail a status check? - How should linked issue objectives be obtained? - Should line-level comments be added after the initial version? - How should old review history be retained? - Which static-analysis tools should be supported directly? - Should repository owners be able to define different rules for different paths? - What limits should apply to model cost, tokens and execution time? Definition of success A successful first version should reliably provide a useful additional review signal without creating excessive noise. It should make clear: - What was reviewed. - What checks were performed. - What appears incomplete or unsafe. - What evidence supports each finding. - What could not be verified. - That a human remains responsible for the final review and merge decision.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
fix/action-yaml-parser
issue-4-normalize-analysis
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
nimmo/forgejo-auto-pr-review#1
No description provided.