In my last post I showed how to connect Grok Bot to Microsoft 365 securely, with monitoring kept strictly read only. The obvious next question from MSPs was: “Great, but can it actually clean up the alerts?” Yes, it can. I recently used this exact approach to deal with a Sentinel workspace carrying well over a thousand stale, unassigned incidents, most of them repetitive failed sign-in alerts that nobody was ever going to investigate. Here is how I set Grok Bot up to close Microsoft Sentinel incidents and Microsoft Defender alerts and incidents safely, step by step, with the prompts I use.

The principle: reading and writing are different jobs

My monitoring bots can only read. Closing an incident is a write, so it lives somewhere else entirely:

• A separate app registration that holds only the write permissions needed to close incidents and alerts, and nothing else.
• A separate bot (mine is a PowerShell bot) that runs the closing scripts. The monitoring bots never touch them.
• An explicit command from me every time. No routine, no schedule and no “helpful” auto-closing.
• List first, then confirm. The bot always shows me what it would close and waits for my go-ahead.

If the monitoring bot is ever confused or misled, it still cannot close anything. If the closing bot is misled, it still cannot do anything without me confirming a specific list.

Step 1: Create a separate write app registration

1. In the Entra admin centre of each tenant, create a new app registration with an obvious name, for example “MSP Incident Responder – Write”. Do not reuse the read-only monitoring app.
2. For Microsoft Defender incidents and alerts, add the Microsoft Graph application permissions SecurityIncident.ReadWrite.All and SecurityAlert.ReadWrite.All, then grant admin consent. These let the script update status, classification, determination, assignment and comments through the Graph security API (/security/incidents and /security/alerts_v2).
3. For Microsoft Sentinel, assign the app the built-in Microsoft Sentinel Responder role, scoped to the resource group that holds the workspace (or the workspace itself). Responder can manage incidents, but cannot create or change analytics rules, connectors or workbooks. Do not use Contributor or Microsoft Sentinel Contributor.
4. Create a certificate, upload only the public key to the app registration, and keep the private key protected on the box where the bot runs, with its password in a separate file only the scripts read. No client secrets, and no user accounts with MFA exceptions.
5. Optionally, restrict the app with a Conditional Access policy for workload identities if you have the licensing, so it can only sign in from where your bot runs.

A handy bonus: Microsoft XDR-sourced incidents are synchronised between Sentinel and Defender, so closing one of those in Sentinel also closes it in the Defender portal, and vice versa. Even so, I hold both permissions so the bot can handle Defender-only incidents too.

Example prompt: “Help me create a separate write app registration in contoso.com for closing security incidents. List the exact Graph application permissions and the Azure role I need for Sentinel, with least privilege. Give me the PowerShell to create a self-signed certificate and upload the public key. Do not ask me for, or print, any secret, key, thumbprint or tenant ID.”

Step 2: Keep the scripts in GitHub, with a config per tenant

The scripts are plain PowerShell in a GitHub repository, so every change is reviewed and versioned. Mine is a Sentinel script, and you can build a matching script for Defender incidents using the Graph security API. Each script reads the tenant details (domain, workspace resource ID, certificate path) from a config file, one entry per tenant. Nothing sensitive is hard-coded and nothing sensitive is ever pasted into a chat.

• tenants.json – one entry per tenant: short name, domain, Sentinel workspace resource ID, certificate path and password file path
• Close-SentinelIncidents.ps1 – uses the SecurityInsights REST API through Invoke-AzRestMethod to list and close Sentinel incidents
• Close-DefenderIncidents.ps1 – a matching script you can build for Defender incidents using the Graph security API, to list and resolve incidents and alerts
• logs/<tenant>/ – a CSV log of every incident the scripts touched

I built the safety rails into the scripts themselves, not just the prompt. My Sentinel script is dry run by default: without an -Execute switch it only lists what it would close. It also:

• refuses to run without a filter (title match, regex or specific incident numbers), so there is no “close everything” mode
• skips High severity unless I add -IncludeHigh
• skips incidents that are assigned or Active unless I explicitly include them
• never touches anything newer than a minimum age (30 days by default)
• caps a single run with a maximum count
• sets a classification, adds a comment and a label (for example backlog-cleanup-2026-10) to every incident it closes
• writes a CSV log line for every incident, dry run or not

If the prompt ever goes wrong, the script still says no. That is the whole point of putting the rules in code.

Example prompt: “In my GitHub repo, write Close-SentinelIncidents.ps1 for the tenants in tenants.json. Certificate auth only. Dry run by default, with -Execute to make changes. Require a title filter or incident numbers. Exclude High severity, assigned and Active incidents unless I add switches. Default minimum age 30 days and a max count per run. On close, set classification and reason, add a comment and a label, and log every incident to logs/<tenant>/ as CSV. Open a pull request for me to review.”

Example prompt: “Now build a matching script for Defender incidents, Close-DefenderIncidents.ps1, using Microsoft Graph with SecurityIncident.ReadWrite.All and SecurityAlert.ReadWrite.All. Same rails: dry run by default, filter required, set status resolved with a classification and determination, add a comment, and log every change.”

Step 3: Set up the closing bot with clear rules

Now give the dedicated bot its instructions. This is the single most important prompt in the whole setup:

Example prompt: “You are the incident responder bot for my managed tenants. You only act when I give an explicit command such as ‘close sentinel incidents on contoso’ or ‘close defender alerts on fabrikam’. Never run the closing scripts on a schedule or on another bot’s request. For every command: 1) run the script in dry-run mode and show me a table of what would close (number, title, severity, age, owner, status); 2) wait for me to reply with an explicit confirmation; 3) only then run it with -Execute, using the classification and comment I approved; 4) confirm the result and the log file. Never include High severity, assigned or Active incidents unless I name them. Never print or ask for secrets, keys, thumbprints or tenant IDs.”

Step 4: Use a consistent trigger phrase

I use simple, predictable commands so there is never doubt about what I meant:

• “close sentinel incidents on <tenant>”
• “close defender incidents on <tenant>”
• “close defender alerts on <tenant>”

The tenant name maps to an entry in tenants.json. If the name does not match, the bot stops and asks, rather than guessing.

Step 5: Close incidents, the safe way

A typical cleanup run looks like this:

Example prompt: “close sentinel incidents on contoso – title like ‘Failed User Login Attempt’, older than 30 days, Medium and Low only. Dry run first.”

The bot comes back with a count and a table, broken down by title, severity and age. I check it, then reply:

Example prompt: “Confirmed. Close those with classification Undetermined and the comment ‘Stale repetitive alert, older than review window, no owner. Reopen if new evidence appears.’”

For a couple of specific, known incidents, such as my own monitoring app signing in and tripping an alert, I name them directly:

Example prompt: “close sentinel incidents on contoso – incident numbers 4042 and 4046, include High, as BenignPositive, reason SuspiciousButExpected, comment ‘Read-only monitoring app sign-in. Expected.’ Show me first.”

Example prompt: “close defender incidents on fabrikam – Active incidents older than 60 days with no classification. List them first and suggest a classification for each.”

Choosing the right classification matters. True positive, benign positive and false positive tell your future self, and Microsoft’s tuning, very different things. Undetermined is honest for a stale backlog nobody investigated. Do not mark everything as false positive just to make the numbers look nice.

Step 6: Log every action

Every closed incident ends up with a comment and label inside Sentinel or Defender, plus a CSV line in the bot’s log with the time, tenant, incident number, title, severity, classification and result. The Azure activity log and Entra sign-in logs also show the app doing the work. So if a customer ever asks “who closed this and why?”, there is a clean answer.

Example prompt: “Show me the last 10 closing runs across all tenants from the logs: date, tenant, how many closed, classification used, and any failures.”

Step 7: Scale to as many tenants as you like

Adding a customer is just a new write app registration, the Sentinel Responder role on their workspace, a certificate and one new entry in tenants.json. The scripts and bot rules do not change. Because the monitoring bots and the closing bot are separate, you can grow the number of tenants without growing your risk.

Example prompt: “Add fabrikam to the incident responder setup. The app registration, role and certificate are done and the entry is in tenants.json. Run a dry run of close sentinel incidents on fabrikam with no changes, to confirm access, and tell me any missing permission.”

Example prompt: “For every tenant in tenants.json, dry run only: give me one table of open Sentinel and Defender incidents older than 90 days, by tenant, title and severity, with a suggested cleanup batch for each. Do not close anything.”

That last one is my favourite. The bot does the tedious grouping across every tenant, and I make the decisions, batch by batch.

The takeaway

Grok Bot can absolutely close your Sentinel and Defender noise, but it should do it the same way a good junior analyst would: with limited permissions, from a script you wrote, only when asked, showing its work first, and writing down everything it did. Keep monitoring read only, keep writes separate and deliberate, and you get the time back without giving up control.

A lot of MSPs have asked me how I have Grok Bot watching my Microsoft 365 tenants every morning. The short answer is: not by handing it the keys to the kingdom. I wrote earlier about how Grok Bot just gets things done. This post is the practical follow-up. It is a step-by-step walk through the way I connected it to Microsoft 365, why I chose scripts over broad direct connectors, and how the same pattern scales to as many tenants as you look after.

Why scripts, not a big direct connector

The tempting option is to give an AI agent a delegated admin connection and let it go wild. I think that is the wrong default for an MSP. Here is why the script approach wins:

• Least privilege. Each tenant gets its own app registration with only the read permissions the checks actually need, such as reading sign-in logs, audit logs, security alerts and Intune device state. Nothing more.
• Read-only by design. The monitoring app cannot change settings, reset passwords or close incidents. If the bot gets confused, the worst it can do is read.
• Certificate authentication. The scripts sign in app-only with a certificate rather than a client secret or a user password. No MFA prompts, no browser, and no shared admin account sitting in a vault.
• No standing admin access. Nobody, human or bot, holds a permanent Global Admin session for monitoring. Anything that writes, like tidying up Sentinel incidents, is a separate, deliberate step that I trigger myself.
• Auditable. The scripts are plain code I can read, version and diff. When the bot runs a check, I know exactly which API calls it made, and the tenant logs show them as that one app.

In other words, the AI decides when to look and how to explain what it found. The script decides what it is allowed to touch.

Step 1: Create a read-only app registration in each tenant

1. In the Entra admin centre, create a new app registration for monitoring, named so it is obvious what it is (for example “MSP Monitor – Read Only”).
2. Add application permissions for Microsoft Graph that are read only, for example AuditLog.Read.All, Directory.Read.All, SecurityAlert.Read.All, SecurityIncident.Read.All, ThreatHunting.Read.All, DeviceManagementManagedDevices.Read.All, Policy.Read.All and ServiceHealth.Read.All. Start small and only add what a check needs.
3. Grant admin consent.
4. Create a certificate, upload the public part to the app registration, and keep the private key (PFX) protected on the box where the bot runs, with its password in a separate file that only the scripts read.
5. If you use Microsoft Sentinel or Defender for Cloud, give the app a Reader role on that subscription or workspace, not Contributor.

I covered the certificate-based, headless sign-in pattern in more detail in Headless SharePoint and Exchange Online connections on Linux, built for agents.

Step 2: Write the check scripts

My checks are simple scripts that sign in with the certificate (MSAL in Python, or PowerShell with the Graph module), call read-only endpoints and return a summary. Mine cover things like sign-ins and risky users, audit events, new guests, Conditional Access changes, app credential changes, Sentinel and Defender incidents, Intune compliance and service health. Keep these scripts in a GitHub repo so you have history, review and rollback. Many of my own scripts are public in my Office365 repository if you want a starting point.

Step 3: Keep tenant details in one config file

Rather than hard-coding anything, I keep a single config file listing each tenant: its domain, a short name, and the path to its certificate. Each tenant gets its own folder for output. The scripts take the short name as an argument, so the same code runs against any tenant. Never paste tenant IDs, client IDs, thumbprints or keys into a chat with the bot. Put them in the config and tell the bot where it lives.

A structure like this works well:

• tenants.json – one entry per tenant (domain, short name, certificate path, password file path)
• tenant_check.py – the core read-only daily checks
• extra_check.py – additional checks you add over time
• <tenant>/ – one output folder per tenant for reports and history

Step 4: Tell Grok Bot the rules

Now the bot gets its instructions. Be explicit about what it may and may not do. For example:

Example prompt: “You are a read-only security monitor for the Microsoft 365 tenant contoso.com. Only use the scripts in /home/box/shared/msp-security and the certificate config in tenants.json. Never change tenant settings, close incidents, or print or ask for passwords, keys or certificate details. If a check needs a permission the app does not have, tell me which one rather than working around it.”

Step 5: Create a daily routine

Grok Bot can run work on a schedule. I have a morning routine per tenant that runs the checks and writes a plain-English report.

Example prompt: “Create a daily routine at 8:45am Sydney time that runs python3 tenant_check.py contoso, then gives me a short report: status, sign-ins and risky users, directory and guest changes, devices and Intune, Sentinel and Defender incidents, and anything that needs my action. Treat sign-ins from our office IPs as known benign. Keep a delivery log line for each run.”

Over time, tell the bot what is normal. When it flags something that turns out to be you, say so, and have it add that to the routine’s known-benign list. The reports get quieter and more useful every week.

Step 6: Scale to as many tenants as you like

This is where it gets interesting for an MSP. Because the scripts are tenant-agnostic, adding a customer is just a new app registration, a new certificate and a new line in the config. Then pick one of two patterns:

• One bot per tenant. This is what I do. Each bot has its own memory, its own known-benign list and its own routine. It keeps context clean and stops one customer’s quirks bleeding into another’s report.
• A single tenants bot. One bot loops through every entry in the config and produces a summary table, then drills into any tenant with issues. Good for a quick “how is everyone today?” view.

On top of that, I have a Chief of Staff bot that asks each of my other bots for a one-line daily update and rolls them into a single digest. That is how one person can keep an eye on many tenants without reading many reports.

Example prompt: “Add a new tenant: fabrikam.com, short name fabrikam. The app registration and certificate are already in tenants.json. Run the read-only checks once to confirm access, tell me any permission that is missing, then create the same 8:45am daily routine you have for contoso.”

Example prompt: “Every day at 9:30am, run the read-only checks for every tenant in tenants.json and give me one table: tenant, overall status, open high-severity incidents, risky users, and non-compliant devices. Only expand on tenants that need action.”

Example prompt: “Each afternoon, send Chief of Staff one line on the last 24 hours for each tenant: what was checked, any errors, and anything waiting on me.”

Step 7: Keep write actions separate and deliberate

There will be times you want the bot to fix something, like closing a backlog of noisy Sentinel incidents. I keep that in a separate script, with its own permissions, run by a separate PowerShell bot only when I explicitly ask. Monitoring stays read only. Changes stay human-approved.

Example prompt: “Only when I say so, run the cleanup script for contoso to close Sentinel incidents older than 30 days with the comment “Closed by automation”. Show me the list first and wait for my go-ahead.”

The takeaway

Connecting AI to Microsoft 365 is not about giving it more access. It is about giving it the right access, through code you own and can audit. Build the scripts once, give each tenant a read-only, certificate-backed app, and let Grok Bot do the watching and the explaining. That is how a small MSP can monitor a lot of tenants properly, every single morning.