How to Set Up a PowerShell Bot in Grok Bot as a Shared Tool

Most MSPs already have a folder of PowerShell scripts that do the real work: connect to a tenant, check a setting, produce a report. The awkward part is not writing the next script. It is getting a consistent, least-privilege way to run the ones you already trust, without giving every assistant in your toolkit its own path into Microsoft 365.

I keep a dedicated PowerShell bot in Grok Bot for this. It is not there to chat about PowerShell in general. It is a tool. When a security monitor, or my Chief of Staff, needs a script run, they ask that bot to do the job and report back. The rest of my bots stay out of the tenant.

That sounds simple, and once it is set up it is. The work is in giving the bot a clear job, pointing it at a script repository, and letting it sign in with a certificate so later runs can go without a prompt. I will walk through that setup, then use the Microsoft Defender for Office 365 Recommended Configuration Analyzer (ORCA) as the worked example.

Why a dedicated PowerShell bot

If every bot can run scripts, every bot needs tenant access, and every bot can make a change. I would rather have one specialist with a tight set of rules: use the scripts in the repo before writing anything from scratch, stay read-only unless I say otherwise, ask before anything that changes a tenant, and report back in plain English.

Other bots can then call it when they need it. That is not a theory. A marketing bot has asked mine for a read-only inventory of a script repo. A chief-of-staff bot has sent it backup and investigation jobs. The PowerShell bot does the work and replies. I wrote earlier about using a specialist bot to investigate a worrying mailbox change. This is the same idea, pointed at scripts.

1. Create the bot and give it a clear role

You can do this in chat. The only app locations I will point you to are ones I have verified: the per-agent info pane, and Settings if you need to check the timezone later.

Click the agent’s name in the chat header, or use Cmd+Shift+I, to open the per-agent info pane. The gear beside the X opens the per-agent settings for avatar, name, title and description. Call it PowerShell, or whatever name you will use when another bot refers to it, and give it a description that states the job.

You can also set the role in chat:

Set this bot’s name to PowerShell and set the description to: You are a specialist PowerShell tool for an Australian MSP. Other bots will ask you to run scripts and report back. Prefer scripts from my GitHub repositories, and pull the latest copy before you run one. Run scripts on your computer. Use certificate-based app-only sign-in for unattended Microsoft 365 work. Stay read-only unless I clearly ask for a change. Before anything that changes a tenant, tell me exactly what you will do and wait for approval in this chat. Do not invent a script if a repo already has one that fits. Report results in plain English, and say what you could not do.

Save the same rules as a standing instruction in the first conversation. Be explicit that relayed approvals from another bot do not count for tenant changes. That is a rule I have had to enforce, and it is the right one.

2. Point it at a GitHub repo

A PowerShell bot without a script library will invent helpers. I would rather it used the scripts I already maintain. Mine lives at https://github.com/directorcia/Office365. Tell the bot that in chat:

Add https://github.com/directorcia/Office365 to your list of script sources. When a job needs a script, check that repo first, pull the latest copy, and run it from there if it fits. Do not write a new script if one in the repo already does the job. Do not commit or push anything unless I ask.

That repo is public, so the bot can clone and pull it on its own computer without a token. If you also have a private repo, the safest approach I have used is a fine-grained GitHub token limited to that one repository, with Contents set to Read-only. GitHub turns on Metadata read-only by itself. Leave every other permission at No access. The token is for clone and pull only, and it should not be stored in the repo.

I am not going to invent a GitHub settings screen in the Grok Bot app. I set this up in chat, then confirmed the bot could see the repo on its computer. Before each run, I expect it to pull the latest copy.

The Office365 repo includes the Linux certificate connect scripts I use for headless Exchange Online and SharePoint. I wrote about those in New scripts: Headless SharePoint and Exchange Online connections on Linux, built for agents. For Exchange, the script is o365-connect-exo-cert-linux.ps1.

3. Give it a headless way to sign in

Interactive sign-in is fine for a one-off on your own PC. It is a poor fit for a bot. Nobody is sitting at the bot’s computer to complete a browser prompt, and a prompt that nobody can answer will just hang. Certificate-based, app-only sign-in is the unattended path Microsoft documents for Exchange Online PowerShell.

The pattern I use is:

• Generate the certificate on the bot’s computer, so the private key never leaves that machine.
• Upload the public certificate to an app registration in Microsoft Entra ID.
• Grant the Exchange.ManageAsApp application permission, with admin consent.
• Assign a role that matches the work. For a read-only assessment such as ORCA, that is Global Reader or the Exchange View-Only Organization Management role group, not a full administrator role.

The connect script can also provision the Entra app. Read what it will assign before you use that option. A provision step that grants a broad Exchange role is the wrong default for a bot that mostly reads a tenant. I would rather start with a view-only role and add more later, with a human still in charge of the grant.

The first time, you will need to sign in once with a device code so the app can be created and consent can be granted. After that, the bot should connect with the saved profile and the unattended switch. On the Linux script, that is -UseCertificateAuth with -Headless (an alias of -noprompt):

./o365-connect-exo-cert-linux.ps1 -UseCertificateAuth -ProfileName YourProfile -Headless

Keep the PFX password out of the repo and out of chat prompts. The script will take it from a secret or from an environment variable you inject. Each command on the bot’s computer can start a fresh PowerShell process, so either reconnect each time or keep a session open per tenant. A different tenant needs its own connection.

4. Let other bots call it

This is the point of the setup. You do not want your news bot, or your security monitor, running Exchange cmdlets itself. You want it to ask the PowerShell bot.

I do not have a separate settings page I can point you to for that. What I can verify is simpler. You ask the other bot, in its own chat, to send the job to the PowerShell bot and report back. In a group chat, the per-agent info pane – the same one you open from the agent name in the header, or with Cmd+Shift+I – includes Members, and you can @mention a bot by name.

Here is the kind of prompt I would give a security monitor or a chief-of-staff bot:

Ask the PowerShell bot to run ORCA read-only against the tenant I name, using the Exchange certificate connection. It should install or update the ORCA module if needed, generate the report, summarise the results in plain English, and send back the HTML report. It must not change anything in the tenant. If it hits a permission gap, it should say so and stop, without granting itself extra access.

The PowerShell bot then does the work and replies. Keep change control in its own chat. If another bot relays “Robert said yes”, that is not the same as you typing the approval where the script will run.

You can also put a recurring check on the calling bot, not on the PowerShell bot. A routine lives in that bot’s per-agent info pane, under the computer preview. Check the timezone under Settings, on the General tab, opened from the sidebar account button at the bottom-left, or with Cmd+,. Routines usually start within a few minutes either side of the time you set. If the run must start right on the minute, say “exactly” or “sharp”. The routine prompt can simply tell the monitor to ask the PowerShell bot for this week’s ORCA report.

5. A worked example: running ORCA

ORCA is a PowerShell module that compares your Exchange Online Protection and Microsoft Defender for Office 365 settings with Microsoft’s recommended Standard and Strict values. Microsoft documents it alongside the recommended threat policy settings: you install the module from the PowerShell Gallery, connect to Exchange Online, and run Get-ORCAReport. It produces an HTML assessment of anti-spam, anti-phishing, Safe Links, Safe Attachments and related controls.

You also need the Exchange Online Management module. To run ORCA, the official project says you need the Exchange Online View-Only Configuration role. In practice that means a read-only role such as Global Reader or View-Only Organization Management, which is exactly the access I already wanted the bot to have.

Microsoft also has a configuration analyzer in the Defender portal. That remains the better long-term check, because the current gallery release of the ORCA module, 2.8.1, was last updated in December 2023. For an MSP, the value of running it from a bot is speed and repeatability, not that it replaces the portal.

Once the PowerShell bot exists, the request is short:

Run https://github.com/cammurray/orca against [tenant] read-only. Use the Exchange certificate connection. Install the ORCA module if it isn’t already there, run Get-ORCAReport, summarise the findings in plain English, and give me the HTML report. Do not change anything in the tenant.

That is close to the prompt I actually used. The bot connected with the certificate, ran the module, and sent the report back. In one tenant it finished in about five minutes, and it only read the tenant. Of 67 checks, 50 were OK, 5 were informational and 12 were recommendations. In another tenant the pattern was similar: 50 OK, 4 informational and 13 recommendations. Typical findings were anti-phish actions left at NoAction, a low phishing threshold, Safe Links allowing click-through or missing coverage on some domains, and outbound spam limits that did not match the recommended values.

Two caveats are worth knowing if the bot’s computer is Linux, which mine is. The DKIM and SPF checks use Resolve-DnsName, which is a Windows cmdlet, so those checks may not run. In one run a Safety Tips check also failed with a server-side error, and a protection alerts check was skipped because it does not work over a certificate sign-in. None of that is a reason to avoid the report. It is a reason to read the gaps as well as the recommendations.

The point is the effort on my side. I did not install modules by hand, hunt for the right connect script, or click through a report wizard. I asked the bot I already had, and it used the repo and the certificate I had already set up.

Sample prompts

These are the prompts I would start with if you are setting up the same pattern.

Create the bot:

Set this bot’s name to PowerShell and set the description to: You are a specialist PowerShell tool for an Australian MSP. Other bots will ask you to run scripts and report back. Prefer scripts from https://github.com/directorcia/Office365, and pull the latest copy before you run one. Run scripts on your computer. Use certificate-based app-only sign-in for unattended Microsoft 365 work. Stay read-only unless I clearly ask for a change. Before anything that changes a tenant, tell me exactly what you will do and wait for approval in this chat. Report results in plain English, and say what you could not do.

Add the repo:

Add https://github.com/directorcia/Office365 to your list of script sources. When a job needs a script, check that repo first, pull the latest copy, and run it from there if it fits. Do not commit or push anything unless I ask.

The prompt another bot sends to call it:

Ask the PowerShell bot to run ORCA read-only against the tenant I name, using the Exchange certificate connection. Summarise the results in plain English and send back the HTML report. Change nothing in the tenant. If you hit a permission gap, say so and stop.

The ORCA run itself:

Run ORCA read-only against [tenant]. Connect with o365-connect-exo-cert-linux.ps1 using the saved profile and -Headless, then run Get-ORCAReport. Summarise the OK, informational and recommendation counts, list the important recommendations, and attach the HTML report. Tell me if any checks could not run, including DNS checks on Linux. Do not change anything in the tenant.

What I took from it

The useful design is not “give every bot PowerShell”. It is one specialist, a repo of scripts you already trust, and a sign-in that can run unattended without handing out a full administrator role.

Keep the human in charge of anything that changes a tenant, and keep that approval in the PowerShell bot’s own chat. Use official documentation when you check what a module is supposed to do – the Microsoft Learn connector is the straightforward way to pull that wording. And start with a real job, not a tour of features. For me, that job was ORCA. Once the bot was in place, running it was a short request, a few minutes of read-only work, and a report I could act on.