Azure DevOps Integration
Connect Otto Detective to Azure DevOps and Otto reviews the pull requests in the repositories you choose. Otto uses a service principal (application identity) to access your Azure DevOps organization. It reviews and comments, and never commits, pushes, or merges.
Prerequisites
Before connecting, you need:
- An Azure DevOps organization
- Admin access to your Azure DevOps organization settings (to add the service principal)
- Your Azure AD Tenant ID (found in Azure Portal → Microsoft Entra ID → Overview)
Connecting Azure DevOps
Step 1: Add Otto's Service Principal to Azure DevOps
- In Azure DevOps, go to Organization Settings → Users → Add users
- Search for Otto's service principal by app name or client ID
- Set access level to Basic
- Add the service principal to relevant projects as Contributor
Step 2: Connect in Otto
- Go to Settings → Integrations in your Otto dashboard
- Click Connect on the Azure Repos card
- Enter your Azure DevOps organization name (from
https://dev.azure.com/your-org-name) - Enter your Azure AD Tenant ID
- Click Connect
Otto validates access to your organization and automatically creates webhook subscriptions for PR events.
What Otto posts on a pull request
- Inline findings on the lines that changed, ranked P0–P3, with suggested changes and a prompt you can paste into your AI coding tool.
- A summary thread that states the verdict (Approved, Changes requested, or Commented), the findings, and Merge confidence: X/5. Each review posts a fresh summary thread.
- The
otto/reviewstatus on the iteration under review.
Comment /otto review to run a fresh review at any time, or /otto explain to ask Otto what the code does and why. Reply to any of Otto's threads to ask a follow-up question.
The change-flow diagram, one-click suggestions, and the single updated summary are GitHub features today.
The otto/review Status
Whenever Otto starts a fresh review of a pull request, it posts an otto/review status on the PR iteration it is reviewing. A fresh review happens when a PR is created, when new commits are pushed (if your reviewer trigger listens to PR updates), or when someone comments /otto review. Replies to comments don't touch the status.
The status says whether Otto is still reviewing. It never reflects Otto's verdict, which lives in the review summary:
| State | Meaning |
|---|---|
| Pending | Otto is reviewing this iteration. |
| Succeeded | Otto finished reviewing, whatever the verdict. |
| Not applicable | Otto couldn't finish: it errored, timed out (after 60 minutes), the PR was too large, or the org is out of credits. |
A required status policy treats "not applicable" as not applying to the PR, so a finished or broken Otto review never blocks completion. Comment /otto review to retry.
Waiting for Otto before completing a PR
To stop PRs from being completed while Otto is reviewing, add a status check policy to the target branch (Branch policies > Status checks > +):
| Setting | Value |
|---|---|
| Status to check | otto/review |
| Policy requirement | Required |
| Authorized identity | Otto's service principal, so only Otto can satisfy the policy |
| Reset conditions | Optional; see the caveat below |
| Policy applicability | Apply by default |
Caveat: a required status that is never posted blocks completion. Otto only posts it on PRs it reviews, so require it only on branches where Otto reviews every PR. With Reset status whenever there are new changes turned on, every push resets the policy until Otto reviews the new iteration. Only turn it on when your PR reviewer trigger also runs on PR updates; otherwise each push blocks the PR until someone comments
/otto review.
Webhook Events
Otto automatically subscribes to these Azure DevOps events:
| Event | Description |
|---|---|
| PR created | New pull request opened |
| PR updated | Pull request modified |
| PR comment | Comment added to pull request |
Required Permissions
The Otto service principal needs these permissions in your Azure DevOps organization:
| Permission | Purpose |
|---|---|
| Code (read/write) | Read the code under review. Write comes with the Contributor role; Otto Detective never commits or pushes |
| Pull requests (read/write) | Post review threads and the otto/review status |
| Work items (read/write) | Used by Otto Builder's ticket workflows |
| Identity (read) | Access user and team information |
Best Practices
Start with a Test Project
Initially, give Otto access to a test repository to validate your configuration before expanding to production repos.
Require Otto Before Completing PRs
If you want PRs to wait for Otto, add otto/review as a required status policy (see above). Without that policy, Otto's review never blocks completion.
Tune Reviews Per Repository
Commit an otto.yml file to set review depth and instructions per path, ignore files, or skip automatic reviews.
Troubleshooting
"Failed to validate access"
The service principal has not been added to your Azure DevOps organization, or it lacks sufficient permissions.
Fix: Go to Azure DevOps → Organization Settings → Users → Add users, search for Otto's service principal, and grant Basic access level.
"Tenant ID not found" or token acquisition error
The Tenant ID may be incorrect.
Fix: In the Azure Portal, navigate to Microsoft Entra ID → Overview and copy the Tenant ID value.
Webhook subscriptions not created
Otto creates webhook subscriptions automatically during connection. If they fail, Otto will still connect but won't receive PR events.
Fix: Disconnect and reconnect the integration, or contact support.