If you aren’t aware, when you built or run anything new in Copilot Studio it will all be charged PAYG against Azure. To allow this you need to connect your Power Platform environment to an Azure subscription. You can find details on how to do that here:
Now with all that in place I built a new modern agent in Copilot Studio to answer M365 questions (called Sage) built using the new Github Copilot harness and using Claude Opus 5 as the LLM.
I then tested it a few times in ‘Preview’ and was happy that it was all working. I knew at this point, all that was going to cost me a few bucks because now even creation costs with the new Copilot Studio. All good so far and still in budget.
Next, I wired up a new Workflow in Copilot Studio to wait for a message to be posted into a Microsoft Teams channel, take that, post it to the newly created Sage agent, then take the reply from the agent and post it back into the same channel. Quick and easy to create. Job done, I thought.
Can you see the logic flaw yet? I certainly didn’t initially. In short, the workflow I created basically replies to every message posted into a channel. Ahem, those replies then trigger the agent to run again and post yet another message, which again triggers another message posting from the agent, and on and on. So, I had created an infinite loop.
My mistake was now running the workflow and calling the agent and posting into the Team every minute or so. I didn’t recognise my error for a few hours! Yes, hours. I estimate the loop I created with the workflow ran for about 3.5 hours in total. Ouch. When I finally realised upon checking back into the channel I immediately deleted the workflow to stop the race condition, however I knew I was going to pay for my mistake.
Fast forward a day or so when I have all the billing data available. Here’s what the results of my oversight were:
The error had cost me around AU$250. D’Oh!
All of this is always a learning experience, so now that I had understood the ‘bill shock’ amount I wanted to see what more billing information I could obtain about what had actually happened. I visited the Power Platform admin center | Licensing Copilot Studio, scrolled down to Top 5 agents and users, then selected View all agents which showed me this:
then when I drilled into my environments I can see:
and at the bottom you can see the autonomous consumption of 16,376.01 credits. If I divide that by the 3.5 hour run time I get 4,678.86 credits consumed per hour. If I then divide that by 60 to get the cost per minute I get 77.98 credits. Thus, each post to the channel in effect cost around US$0.78 which is about AU$1.20.
The detail also shows that creating the agent cost around 67.26 + 379.77 = 447.37 credit which is around US$4.50 and AU$6.95 to create.
I have now added notifications at 90% capacity like so in this admin console because they are not enabled by default:
I would expect, like the budget notifications from Azure, they are not immediate which makes avoiding costly mistakes harder when your error maybe racking up a few dollars per minute charges!
I accept full responsibility for my error and oversight and bill incurred, however I think there are some important learnings and observations here with the new PAYG billing for AI services. These in essence boil down to the fact that it very difficult to get a good understanding of exactly what your costs are in real time or prior. Typically, you need to wait a full 24 hours until all the billing data has been collected and by then you maybe up for thousands of dollars if you are not very careful.
Another observation is that if you make a logic error in your build you won’t find that until you look at your bill. I was lucky that I found mine after a few hours, imagine if it had run for more than 24 hours before the billing data alerted me? Ouch.
I believe this lack of immediacy an d visibility on costs is going to be a major barrier for adoption of PAYG agents in Microsoft 365, whether Cowork or the new Copilot Studio, especially in SMB.
Hopefully, we get to a point like we have with Github Copilot (above) where I can quickly and easily see my usage in the development environment (here Visual Studio Code). Without this type of spending certainty many business are simply not going to use what are fantastic AI tools to help their business. This risk of runaway costs is simply too great.
Another point that I want to reinforce here is that when you implement PAYG with agents you need to monitor your costs DAILY! This will be a big change for many MSPs who may occasionally go into a customers tenant to look at licensing monthly. If your customer has PAYG AI and you are responsible for managing these costs you need to keep an eye on this every single day to minimise what a single logic error could cost.
Ensure you enable all the alerting that you can when you use PAYG AI services, no matter where or whom they are from. Hopefully, doing this and my sorry tale here helps you better monitor your costs and avoid ‘AI usage bill shock’.
I keep seeing people look at AI pricing pages and assume someone has made a typo.
Input tokens cost one amount. Output tokens cost another. Cached input may cost less again. At first glance that feels odd. A token is a token, surely?
Not really.
The easiest way to think about it is this: reading is cheaper than writing. AI can read a lot of your prompt in parallel. It can process the instructions, pasted document, previous conversation and system rules in large chunks. That is the input side.
Writing the answer is different. The model generates the response one token at a time. Each next word, fragment, line of code or JSON field depends on what came before it. The longer the response, the longer expensive compute is tied up producing it.
That is why output tokens usually cost more. You are not just paying for text. You are paying for generation.
This matters more than most people think
Once you move into agents, automation, Copilot Studio, Azure OpenAI, GitHub Copilot, or anything that runs repeatedly, the economics change quickly.
A short prompt that produces a long report can cost more than a large prompt that produces a tiny answer. That surprises people. They assume the big document is the expensive part. Sometimes it is not. The real cost can be the polished, verbose output they asked the model to create.
Ask an AI system to read a SharePoint policy library and return three risks, and you are probably dealing with an input-heavy, output-light workload. That can be relatively efficient.
Ask it to create a 25-page report, an executive summary, a remediation plan, a Teams post, a client email and a formatted table every time it runs, and you have created an output-heavy workload. That is where the bill starts to move.
The MSP lesson is simple: design the output
This is where MSPs need to stop treating AI as magic and start treating it as infrastructure.
When we built servers, we cared about CPU, RAM, disk and backup windows. With AI, we need to care about prompts, context, output length, caching and repeatability.
The bad habit is asking for everything every time. “Give me the full report.” “Include all the detail.” “Make it comprehensive.” That sounds harmless until the same workflow runs fifty times across fifty tenants.
A better approach is to be deliberate. Ask for the smallest useful output first. Use summaries where summaries are enough. Generate detailed reports only when there is a reason. Reuse stable instructions and context where caching is available. Put spending limits around anything consumption-based. Review what the agent writes, not just what it reads.
Inside Microsoft 365, this means being clear about the difference between ordinary Copilot use in Word, Excel, Outlook or Teams and consumption-based AI work that may be billed differently. A user drafting an email is one thing. An agent chewing through documents and producing long artefacts all day is another.
This is not a pricing trick
I do not see the input/output price split as some mysterious vendor tax. It reflects how the technology behaves.
The mistake is pretending it does not matter.
AI costs are not just about how many people have a licence. They are about what those people, agents and workflows ask the model to produce. The output is where the hidden weight often sits.
So the practical rule is this: do not just prompt for the result. Design the cost shape of the result.
That might be the difference between AI being a useful business tool and AI becoming the next cloud bill nobody wants to open.
In December 2025, Gartner issued an urgent advisory recommending that organizations “block all AI browsers for the foreseeable future” due to critical cybersecurity risks.AI browsers like Perplexity’s Comet and OpenAI’s ChatGPT Atlas introduce threats including irreversible data loss, prompt injection vulnerabilities, and unauthorized credential access.With 27.7% of organizations already having at least one user with an AI browser installed,the time to act is now. [computerworld.com]
This comprehensive guide provides step-by-step instructions for configuring Microsoft 365 Business Premium (M365 BP), specifically Microsoft Defender for Cloud Apps, to detect, monitor, and block AI-enabled browsers like Comet from accessing your enterprise resources.
Understanding the AI Browser Threat Landscape
Why AI Browsers Are Dangerous
According to Gartner analysts, “The real issue is that the loss of sensitive data to AI services can be irreversible and untraceable. Organizations may never recover lost data.” [computerworld.com]
Key Security Concerns:
Autonomous Actions Without Oversight – AI browsers can autonomously navigate websites, fill out forms, and complete transactions while authenticated, creating accountability concerns for erroneous or malicious actions [computerworld.com]
Traditional Controls Are Inadequate – “Traditional controls are inadequate for the new risks introduced by AI browsers, and solutions are only beginning to emerge,” according to Gartner’s senior director analyst Evgeny Mirolyubov [computerworld.com]
Multi-Modal Communication Gaps – A major gap exists in inspecting multi-modal communications with browsers, including voice commands to AI browsers [computerworld.com]
Immature Security Posture – Discovered vulnerabilities highlight broader concerns about the maturity of AI browser technology, with solutions likely taking “a matter of years rather than months” to mature [computerworld.com]
Navigate to Settings → Endpoints → Configuration Management
Go to Enforcement Scope → Network Protection
Verify status is set to “Block mode” (not just Audit mode)
Apply to All devices or specific device groups
Why This Matters: Network Protection ensures that blocks work across all browsers (Chrome, Firefox, etc.), not just Microsoft Edge. [wolkenman….dpress.com]
Phase 3: Unsanction and Block Comet Browser
Objective: Mark Comet and other AI browsers as unsanctioned to trigger automatic blocking.
Step 3.1: Search for Comet in Cloud App Catalog
Go to Cloud Apps → Cloud App Catalog
Use the search function to find “Comet” or “Perplexity”
Click on the application to review its risk assessment
Note: If Comet hasn’t been discovered yet in your environment, you can still add custom URLs for blocking (see Phase 6).
Step 3.2: Unsanction the Application
Click the three dots (⋮) at the end of the application row
Critical Note: Ensure the “Microsoft Defender for Cloud Apps – Session Controls” application is NOT blocked by other Conditional Access policies, or session controls will fail. [learn.microsoft.com]
Phase 5: Create Session Policies to Block AI Browser User Agents
Objective: Create real-time session policies that identify and block AI browsers based on user-agent strings and behavioral patterns.
Step 5.1: Create Access Policy for User-Agent Blocking
This is one of the most effective methods to block specific browsers like Comet. [securityhq.com]
Device type: (Optional) Apply to specific device types
Step 5.4: Create Custom User-Agent String Filters
While Defender for Cloud Apps doesn’t expose direct user-agent string matching in the UI by default, you can leverage activity filters: [securityhq.com]
Known AI Browser User-Agent Patterns to Block:
User-Agent patterns (Create separate policies or use contains logic):
- Contains "Comet"
- Contains "Perplexity"
- Contains "axios" (common in automated tools)
- Contains "ChatGPT" (for Atlas browser)
- Contains "AI-Browser"
- Contains "agentic"
Advanced Method – Using Session Policy with Inspection:
In Inspection method, configure content inspection rules
Step 5.5: Set Actions
Actions:
Select “Block”
Enable “Notify users” with custom message:
Access Denied: AI-Enabled Browser Detected
Your organization's security policy prohibits the use of AI-enabled browsers
(such as Comet, Atlas, or similar tools) to access corporate resources due to
data security and compliance requirements.
Please use Microsoft Edge, Chrome, or Firefox to access this resource.
If you believe this is an error, contact your IT helpdesk.
Step 5.6: Enable Governance Actions
Select “Send email to user”
Select “Alert severity” as High
Enable “Create an alert for each matching event”
Step 5.7: Activate the Policy
Review all settings
Click Create
Policy becomes active immediately
Monitor via Activity Log for matches
Phase 6: Block Comet Domains via Custom Indicators
Objective: Manually add Comet-related domains to Defender for Endpoint indicators for network-level blocking.
Step 6.1: Identify Comet-Related Domains
Based on Perplexity’s infrastructure, key domains include: [computerworld.com]
This site has been blocked by your organization
Microsoft Defender SmartScreen blocked this unsafe site
This web page was blocked by Microsoft Defender Application Control
perplexity.ai has been blocked by your IT administrator
Phase 7: Create Cloud Discovery Policies for Alerting
Objective: Set up automated alerts when AI browsers are detected in your environment.
Step 7.1: Create App Discovery Policy
Navigate to Cloud Apps → Policies → Policy Management
Users see warning message but can proceed (with logging)
Collect usage data for 2-4 weeks
Transition to Block mode after user education
Option 3: Scoped Blocking by Device Groups
Target specific departments first:
In Defender for Endpoint, create device groups:
Finance Team
Executive Leadership
High-Risk Departments
Apply indicators only to these groups initially
Expand gradually after validation
Option 4: DLP Integration for Data Leaving via AI Browsers
Even with blocks, ensure data leakage prevention:
Create Microsoft Purview DLP policies
Target “All locations” including endpoints
Configure rules to detect sensitive data:
Credit card numbers
Social Security numbers
Confidential project names
Block upload/sharing of sensitive content
Identifying Comet Browser Technical Indicators
User-Agent String Analysis
While official Comet user-agent strings aren’t publicly documented by Perplexity, AI browsers typically exhibit these patterns:
Common AI Browser User-Agent Characteristics:
Mozilla/5.0 (Platform) ... Comet/[version]
Mozilla/5.0 (Platform) ... Perplexity/[version]
Chromium-based with custom identifiers
May contain "AI", "Agent", "Agentic" in UA string
Detection Strategy:
Review Activity Log in Defender for Cloud Apps
Filter for unknown/suspicious user agents
Export activity data with user-agent strings
Analyze patterns using PowerShell or Excel
Update policies based on findings
Network Traffic Patterns
Comet communicates with Perplexity cloud infrastructure: [computerworld.com]
High-frequency API calls to api.perplexity.ai
WebSocket connections for real-time AI responses
Upload of page content and browsing context
Telemetry to Perplexity servers
Monitor via Defender for Cloud Apps:
Cloud Apps → Activity Log
Filter by IP address ranges (if known)
Look for unusual upload patterns
Troubleshooting Common Issues
Issue 1: Blocks Not Working in Chrome/Firefox
Symptom: Comet/Perplexity sites accessible in non-Edge browsers
Recommended Combination: Use Cloud Discovery + Unsanctioning AND Access Policies for comprehensive coverage.
Staying Current: Monitoring New AI Browsers
AI browsers are rapidly evolving. Stay ahead of threats:
Monthly Review Checklist
✅ Cloud App Catalog Updates
Review newly discovered apps in Generative AI category
Check for new AI Model Providers
Assess risk scores of emerging tools
✅ Threat Intelligence
Monitor Gartner reports on AI browser security [gartner.com]
Follow Microsoft Security Blog
Subscribe to CISA alerts
Track CVE databases for AI browser vulnerabilities
✅ Policy Effectiveness
Review blocked connection attempts
Analyze bypass attempts
Update user-agent filters
Refine domain lists
Emerging AI Browsers to Monitor
Beyond Comet and Atlas, watch for:
Brave Leo Browser (AI-enhanced features)
Opera One (integrated AI)
Arc Browser (with AI capabilities)
SigmaOS (AI-powered browsing)
Browser Company products
Compliance and Documentation
Required Documentation
Maintain these records for audit purposes:
Policy Documentation
Policy names, purposes, and justifications
Configuration settings and filters
Approval chains and stakeholder sign-offs
Change Log
Policy modifications
Domain additions/removals
Exception approvals
Incident Reports
Blocked access attempts
Policy violations
User complaints and resolutions
Risk Assessment
Why AI browsers are blocked
Business impact analysis
Alternative solutions provided to users
Regulatory Considerations
Consider these compliance frameworks:
Framework
Relevance
GDPR
Data processing outside organization control
HIPAA
Protected health information exfiltration risk
SOX
Financial data protection requirements
PCI DSS
Cardholder data security
NIST 800-53
Access control requirements
Conclusion: Taking Action Against AI Browser Risks
The threat posed by AI browsers like Perplexity’s Comet is real, immediate, and growing. With security experts uniformly recommending that organizations “block all AI browsers for the foreseeable future,”the time for action is now—not later. [pcmag.com], [gartner.com]
Key Takeaways:
Gartner’s Warning is Clear: AI browsers introduce “irreversible and untraceable” data loss risks that traditional controls cannot adequately mitigate [computerworld.com]
Multi-Layered Defense is Essential: Combining Cloud Discovery, Session Policies, Access Policies, and Network Protection provides comprehensive coverage
Microsoft 365 Business Premium Provides the Tools: With Defender for Cloud Apps and Defender for Endpoint, you have enterprise-grade capabilities to detect and block AI browsers
User Education is Critical: Technical controls must be paired with clear communication about why AI browsers pose risks and what alternatives are approved
Continuous Vigilance Required: The AI browser landscape evolves rapidly; monthly reviews of your defenses are essential [computerworld.com]
Immediate Action Steps
This Week:
✅ Enable Cloud Discovery and filter for Generative AI apps
✅ Review current AI browser usage in your organization
✅ Enable “Enforce App Access” in Defender for Cloud Apps
✅ Verify Network Protection is enabled in Defender for Endpoint
Next Week:
✅ Create Conditional Access policy routing traffic to MDCA
✅ Unsanction Comet and other AI browsers
✅ Create custom domain indicators for Perplexity infrastructure
✅ Deploy in Report-only mode for pilot group
Within 30 Days:
✅ Create Access Policies with user-agent filtering
Conditional Access is Microsoft’s Zero Trust policy engine that evaluates signals from users, devices, and locations to make automated access decisions and enforce organizational policies. Think of it as intelligent “if-then” statements: If a user wants to access a resource, then they must complete an action (like multifactor authentication).
For SMBs using Microsoft 365 Business Premium, Conditional Access provides enterprise-grade security without requiring complex infrastructure, protecting your organization from 99.9% of identity-based attacks.
Prerequisites
License Requirements: Microsoft 365 Business Premium (includes Entra ID P1) or Microsoft 365 E3/E5
Admin Role: Conditional Access Administrator or Global Administrator privileges
Preparation: Ensure all users have registered for MFA before implementing policies
Emergency Access Account: Create at least one break-glass account excluded from all policies
Phase 1: Initial Setup and Planning (Week 1)
Step 1: Turn Off Security Defaults
Navigate to Microsoft Entra admin center (entra.microsoft.com)
Go to Entra ID → Properties
Select Manage security defaults
Toggle Security defaults to Disabled
Select My organization is using Conditional Access as the reason
Click Save
Important: Only disable security defaults after you’re ready to create Conditional Access policies immediately.
Step 2: Create Emergency Access Accounts
Create two cloud-only accounts with complex passwords
Assign Global Administrator role to both accounts
Store credentials securely (separate locations)
Document these accounts for emergency use only
Exclude these accounts from ALL Conditional Access policies
Step 3: Access the Conditional Access Portal
Sign in to entra.microsoft.com
Navigate to Entra ID → Conditional Access
Select Policies to view the main dashboard
Phase 2: Create Baseline Policies (Week 1-2)
Policy 1: Require MFA for All Users
Click New policy from templates
Select Require multifactor authentication for all users template
Name your policy: “Baseline: MFA for All Users”
Under Assignments:
Users: All users
Exclude: Select your emergency access accounts
Under Target resources:
Select All resources (formerly ‘All cloud apps’)
Under Access controls → Grant:
Select Require multifactor authentication
Set Enable policy to Report-only
Click Create
Policy 2: Block Legacy Authentication
Click New policy from templates
Select Block legacy authentication template
Name your policy: “Security: Block Legacy Authentication”
Under Assignments:
Users: All users
Exclude: Emergency access accounts
Under Conditions → Client apps:
Configure: Yes
Select Exchange ActiveSync clients and Other clients
Under Access controls → Grant:
Select Block access
Set Enable policy to Report-only
Click Create
Policy 3: Require MFA for Administrators
Click New policy from templates
Select Require multifactor authentication for admins template
Name your policy: “Security: MFA for Admin Roles”
Under Assignments:
Users: Select users and groups
Select Directory roles
Choose all administrative roles
Exclude: Emergency access accounts
Under Access controls → Grant:
Select Require multifactor authentication
Set Enable policy to Report-only
Click Create
Phase 3: Testing and Validation (Week 2)
Step 1: Use the What If Tool
Navigate to Conditional Access → Policies → What If
Enter test scenarios:
Select a test user
Choose target applications
Set device platform and location
Click What If to see which policies would apply
Review both “Policies that will apply” and “Policies that will not apply”
Document results for each test scenario
Step 2: Monitor Report-Only Mode
Leave policies in Report-only mode for at least 7 days
Navigate to Entra ID → Sign-in logs
Filter by Conditional Access = Report-only
Review impacts:
Check for “Report-only: Success” entries
Investigate any “Report-only: Failure” entries
Look for “Report-only: User action required” entries
Address any issues before enforcement
Step 3: Pilot Testing
Create a pilot group with 5-10 users
Create a duplicate policy targeting only the pilot group
Set this pilot policy to On (enforced)
Monitor for 3-5 days
Gather feedback from pilot users
Address any issues identified
Phase 4: Production Deployment (Week 3)
Step 1: Enable Policies
After successful testing, return to each policy
Change Enable policy from Report-only to On
Start with one policy at a time
Wait 2-4 hours between enabling each policy
Monitor sign-in logs after each activation
Step 2: Communicate to Users
Send announcement email before enforcement
Include:
What’s changing and when
Why it’s important for security
What users need to do (register for MFA)
Support contact information
Provide MFA registration instructions
Schedule optional training sessions
Phase 5: Advanced Policies (Week 4+)
Optional: Require Compliant Devices
Only implement after basic policies are stable
Create new policy: “Security: Require Compliant Devices”
Next week is going to be a firehose with Microsoft Ignite taking place, so be prepared for lots of links in the CIA Brief next week.
If you found this valuable, the I’d appreciate a ‘like’. This helps me know that people enjoy what I have created. If you have any feedback or suggestions around this, I’m all ears. You can also find me via email director@ciaops.com and on X (Twitter) at https://www.twitter.com/directorcia.
In this episode I cover off why adding Azure to every environment makes sense. Even though the billing model is different that doesn’t there isn’t an opportunity to add value to an environment with what Azure can provide. There are also plenty of updates from the Microsoft Cloud with many exciting new things to try. Listen along and let me know if you have any feedback.
I’m back baby! It’s been quite a while now but I’ve decided to start getting the Microsoft Cloud information and updates out there irrespectively of whether there is a guest or not. Let’s see how that goes and please let me know what you think and any suggestions you may have.
I’ll also bring you up to date with all the latest news from the Microsoft Cloud.
CIAOPS 