Comparing LLMs in Copilot services–Round 3 – Cowork (Claude)

image

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Winner – Opus

Round 2 – https://blog.ciaops.com/2026/08/08/comparing-llms-in-copilot-services-round-2-researcher/

Winner – Critique

I’ve been pitting different LLMs inside M365 Copilot against each other ina world cup style elimination to see which comes out on top. The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round the battle is between all the models available in Copilot Cowork Claude. Namely, these:

Screenshot 2026-08-11 074851

The other interesting factor here is that all these models as PAYG, so I’ll also give the costs for the same prompt.

To assess all these results together I found Gemini and SharePoint Copilot to both fail completely. Gemini keeps saying that it can’t find all the document, even though I have uploaded and also tried linking. SharePoint Copilot on the other hand starts processing but never gives me a result, no matter how long I wait. I will admit that these files are quite long (20+ pages) and quite complex, so there is a lot to digest. That said I threw them into Grok and those  results are here:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260811-Cowork-Claude-Grok-eval.md

In summary, the ranking where:

1. Opus 5

2. Fable 5 (Preview)

3. Fable 5 (Copilot) (Preview)

4. Opus 4.8

5. Sonnet 5

which all kind of makes empirical sense but still quite subjective I feel. However, for now I’ll swap to using Grok to evaluate these documents as a standard approach.

Now for the costs which don’t factor into the results:

Claude Fable 5 (Preview) = 3,387 credits
Claude Fable 5 (Copilot)(Preview) = 3,373.8 credits
Sonnet 5 = 2,509.3 credits
Opus 4.8 = 2,487 credits
Opus 5 = 2,240 credits


Which, on initial analysis, indicates that costs for all models are in the same kind of range (i.e. at least US$20 per prompt), with Fable 5 models being about 50% more expensive.

Given the analysis and costs, it seems Opus 5 with Cowork, if you are using Claude, is the most cost effective for the best result in Cowork.

So:,

Round 3 winner – Copilot Cowork Claude = Opus 5

Onto the next round.

Security Keeps the Bad Guys Out. Compliance Stops the Good Guys Doing the Wrong Thing.

image

One of the biggest mistakes I see when talking to organisations about Microsoft 365 is that they treat security and compliance as the same thing.

They’re not.

In fact, I often explain it this way: security is largely about protecting your data from people who shouldn’t have access to it, while compliance is about making sure people who do have access don’t misuse it.

Both matter. Both are essential. But they solve completely different problems.

The Security Mindset

When most people think about protecting information, their mind immediately goes to external threats.

Hackers trying to steal credentials.

Phishing emails landing in inboxes.

Ransomware attempting to encrypt files.

Business email compromise targeting finance teams.

That’s the world of security. The goal is to prevent unauthorised access and stop attackers before they can do damage.

This is where tools like Microsoft Defender, Conditional Access, Entra ID, multifactor authentication and device management come into play. They create barriers around your data and make it harder for outsiders to gain access.

The challenge is that many organisations stop there.

They invest heavily in security controls, get comfortable that their tenant is well protected, and assume the job is done.

It isn’t.

Because once someone is legitimately inside the organisation, security has largely done its job.

That’s where compliance starts.

The Compliance Challenge

Imagine an employee has access to customer records as part of their daily role.

They’re not a hacker.

They’re not bypassing security controls.

They’re simply using information they have permission to access.

Now imagine they accidentally email a spreadsheet containing sensitive customer information to the wrong person.

Or upload confidential financial data to an unauthorised location.

Or paste company information into an AI tool that hasn’t been approved by the organisation.

Security didn’t fail here.

The employee had legitimate access.

The issue is what happened after access was granted.

That’s a compliance problem.

Compliance is about governing how information is used, shared, stored and protected after someone is authorised to see it.

Why Copilot Makes This More Important

The rise of Microsoft 365 Copilot makes this distinction even more critical.

Copilot operates using the permissions that already exist within your Microsoft 365 environment. If a user has access to information in SharePoint, OneDrive, Teams or Exchange Online, Copilot can help them work with that data.

That’s why I continually tell organisations that Copilot readiness isn’t just about licensing. It’s about information governance.

I’ve seen organisations excited about deploying Copilot only to discover that sensitive documents are stored in locations where far too many people have access. The problem isn’t Copilot. The problem is that existing permissions and governance weaknesses become much more visible once AI enters the picture.

If your data is overshared, Copilot can expose that reality very quickly.

Security and Compliance Need Equal Attention

The best organisations understand that security and compliance are two sides of the same coin.

Security asks:

“Who should be allowed in?”

Compliance asks:

“What should they be allowed to do once they’re inside?”

In Microsoft 365, that means combining strong security controls with technologies such as Microsoft Purview, sensitivity labels, data loss prevention policies, retention controls and information protection.

One protects the front door.

The other governs what happens in the rooms beyond it.

Ignore either one and you’re leaving your organisation exposed.

Final Thoughts

I think many businesses are still more comfortable talking about security than compliance because security threats are easier to visualise. We can picture a hacker attacking our systems.

What’s harder to visualise is an employee accidentally sharing the wrong file, retaining information for too long, or exposing sensitive data through everyday work practices.

Yet those risks can be just as damaging.

As AI becomes more deeply woven into Microsoft 365, the organisations that succeed won’t simply have the strongest security. They’ll also have the maturity to govern their information properly.

Security protects your organisation from outsiders.

Compliance protects your organisation from itself.

And increasingly, you need both.

Is Copilot Listening to Every Meeting? The Reality Is More Complicated

image

Every few weeks someone asks me a variation of the same question.

“Is Microsoft Copilot recording all my meetings?”

The concern is understandable. AI is becoming more capable, meeting summaries appear almost instantly, action items seem to materialise out of nowhere, and people naturally wonder whether Microsoft is quietly capturing everything that happens in Teams. Sometimes the question goes a step further: “Does it do this even if I don’t have a Copilot licence?”

In my experience, the answer is far less dramatic than many people expect.

The concern isn’t really about Copilot. It’s about visibility.

The Difference Between a Meeting and a Recording

I think a lot of confusion comes from people bundling several different technologies together and calling them all “Copilot”.

A Teams meeting can exist without being recorded.

A Teams meeting can be transcribed without being recorded.

A Teams meeting can be recorded and transcribed.

And Copilot may or may not be involved at all.

That’s an important distinction.

Copilot doesn’t magically create information from nowhere. It works with the information available to it. If your organisation has enabled meeting transcription or recording, that’s where much of the content comes from. Copilot can then use that information to answer questions, create summaries, identify action items and help participants catch up.

Without that underlying data, Copilot has far less to work with.

The key point is that recording and transcription settings are administrative decisions. They aren’t automatically triggered simply because somebody owns a Copilot licence.

What About People Without Copilot?

This is where things often become misunderstood.

I’ve spoken with organisations where only a handful of staff have Microsoft 365 Copilot licences, yet meeting transcripts still exist across the business. When employees see AI-generated meeting summaries, the assumption is that Copilot must be capturing everything.

In reality, those organisations may have enabled Teams transcription or recording policies independently of Copilot.

Think about it this way. Your ability to access information and Microsoft’s ability to store information are not the same thing.

Someone without a Copilot licence may still participate in a meeting that is being recorded or transcribed. The meeting data exists because the meeting organiser or organisation allowed that functionality. Whether an individual user has access to Copilot features is a separate licensing decision.

That’s a subtle difference, but an important one.

The Governance Question Most Businesses Miss

What I find interesting is that many organisations ask whether Copilot is recording meetings, but they rarely ask who should have access to the resulting information.

That’s the more important conversation.

If meeting recordings are stored in OneDrive or SharePoint, if transcripts are available through Teams, and if Copilot can surface information that already exists, then governance matters more than fear.

I’ve seen organisations spend hours debating AI risks while leaving years of meeting recordings scattered across Microsoft 365 with inconsistent permissions.

Copilot didn’t create that problem. It simply made the problem more visible.

A useful exercise is to ask Copilot in Teams or Microsoft 365 a question about a recent meeting and then investigate why the answer was possible. The answer usually points back to existing recordings, transcripts, shared files or meeting notes that were already being retained.

AI acts as a spotlight. It doesn’t necessarily create new information.

The Question I’m Watching Closely

As Copilot adoption grows, I think we’ll see a shift in how organisations think about meetings.

For years, many businesses treated meetings as temporary conversations. Once the meeting ended, most people assumed the discussion disappeared into the ether unless someone took notes.

That assumption no longer holds true.

Whether you’re using Copilot, Teams transcription, meeting recordings or a combination of all three, conversations increasingly become searchable organisational knowledge.

The question therefore isn’t whether Copilot is secretly listening to every meeting.

The better question is whether your organisation understands what meeting data is being captured, where it is stored, how long it is retained and who can access it.

That’s where the real risk lives.

And, just as importantly, that’s where the real value of Microsoft 365 Copilot begins.

Need to Know podcast–Episode 369

In this episode of the Need to Know Podcast, I cover recent Microsoft news, including strong financial results, continued Azure growth, rising Microsoft 365 Copilot adoption, and major security updates. Key security stories include supply chain compromise, cybercrime disruption, hotel Wi-Fi credential theft, and Project Perception, which points toward a future where security is increasingly managed by AI agents rather than manual review. The episode also highlights new Microsoft 365 Copilot capabilities, web grounding domain controls, Copilot in SharePoint improvements, plus two new CIAOPS simulators for Exchange/Defender policy flow and Microsoft 365 sign-in/Conditional Access testing.

The broader discussion focuses on how AI is changing software, security, and business productivity. I argue that cheaper open-weight AI models, Microsoft’s MAI models, and tools like GitHub Copilot make it easier for businesses to build their own dashboards, simulators, and lightweight applications instead of relying only on traditional software. He also introduces the idea of a “SharePoint gardener” to keep SharePoint information organised for AI use, and explains how AI loops can continuously test, improve, and refine code or business processes with human oversight where needed.

Brought to you by www.ciaopspatron.com

you can listen directly to this episode at:

https://ciaops.podbean.com/e/episode-369-developing/

Subscribe via iTunes at:

https://itunes.apple.com/au/podcast/ciaops-need-to-know-podcasts/id406891445?mt=2

or Spotify:

https://open.spotify.com/show/7ejj00cOuw8977GnnE2lPb

Don’t forget to give the show a rating as well as send me any feedback or suggestions you may have for the show

Resources

CIAOPS Need to Know podcast – CIAOPS – Need to Know podcasts | CIAOPS

X – https://www.twitter.com/directorcia

director@ciaops.com

CIAOPS Blog

Join my Teams Shared Channel – CIAOPS

CIAOPS Merch store – CIAOPS

Become a CIAOPS Patron

CIAOPS AI Dojo

CIAOPS weekly news update – CIA Brief – CIAOPS

CIAOPS Labs – The Special Activities Division of the CIAOPS

Support CIAOPS

Get your M365 questions answered via email

Join my email list

A special thanks to the CIAOPS Patron community for making this podcast possible. You can find the benefits of a subscription to the community and become a member at https://www.ciaopspatron.com

Security & Threat Intelligence

Microsoft 365 Copilot & AI

SharePoint & Content Management

Microsoft Corporate News & Strategy

CIAOPS:

Exchange Online + Defender for Office 365 Policy Flow Simulator

M365 Sign-In and Conditional Access Flow Simulator

CIA Brief 20260809

image

Announcements & Product Updates

  • What’s New in Copilot in SharePoint: August 2026 Copilot in SharePoint’s August update can now turn SharePoint lists, Excel and CSV files into live, interactive dashboards that stay connected to the underlying data. It also adds one-click “page buttons” that launch contextual prompts, plus chat improvements for adding context. The theme is moving Copilot from simply answering questions to helping you create, analyse and act on your content.https://techcommunity.microsoft.com/blog/spblog/whats-new-in-copilot-in-sharepoint-august-2026/4535421
  • Copilot Just Changed Again — August 2026 (Updates From A Microsoft Engineer) AI – A Microsoft engineer walks through every August 2026 Microsoft 365 Copilot update across Chat, Word, PowerPoint, Excel, Teams, SharePoint and Notebooks. Highlights include new models, Word track-changes rewrites and Cowork event scheduling. A fast, practical rundown of what to switch on first this month.https://www.youtube.com/watch?v=Kwl_vyftmn8

Copilot Studio & Agents

  • Building Highly Effective Business-Process Agents — Copilot Studio Updates, August 2026 – A walkthrough of how to design robust, business-process agents in Copilot Studio. It focuses on structuring agents for reliable, repeatable multi-step automation rather than one-off tasks. Useful for anyone looking to move from simple chatbots to production-grade workflow agents.https://www.youtube.com/watch?v=6KoTjt6wIMc
  • Inside the New Agent and Workflow Harness — Copilot Studio Updates, August 2026 AI – A closer look inside Copilot Studio’s new agent and workflow harness. It shows how the updated tooling lets you build, invoke and test agents as steps within larger workflows. A good companion to the business-process agents session above.https://www.youtube.com/watch?v=v7IA56HJU2E

Guides & Resources

  • Copilot Credits Guide (August 2026) AIMicrosoft-Copilot-Credits-Guide-August-2026.pdf — explaining how Microsoft Copilot credits and metered billing work. Handy background as pricing shifts from flat-rate to consumption-based models. Note: shared as a file attachment in the channel (no external URL).File posted in AI – News channel (PDF attachment)

After hours

The AI takeover: Who controls our future? https://www.youtube.com/watch?v=jkPVjTl7o-Q

Editorial

If you found this valuable, the I’d appreciate a ‘like’ or perhaps a donation at https://ko-fi.com/ciaops. This helps me know that people enjoy what I have created and provides resources to allow me to create more content. If you have any feedback or suggestions around this, I’m all ears. You can also find me via email director@ciaops.com and on X (Twitter) at https://www.twitter.com/directorcia.

If you want to be part of a dedicated Microsoft Cloud community with information and interactions daily, then consider becoming a CIAOPS Patron – www.ciaopspatron.com.

Watch out for the next CIA Brief next week

Teams Sprawl Was Always Messy. Now It’s Costing Your Clients Answers.

teams-structure-hero

Walk into almost any SMB that’s been on Microsoft 365 for a few years and open their list of teams. I’ll wager it’s long, it’s inconsistent, and at least a third of what’s there hasn’t had a message posted since someone left the business. There’s a “Project Handover”, a “Project Handover 2”, and a “Sales – DO NOT USE”. Nobody remembers who owns half of them.

For years I treated that as a tidiness problem — annoying, but low stakes. I don’t anymore. The moment a client turns on Microsoft 365 Copilot, that untidy structure stops being cosmetic and starts shaping the quality of the answers they get.

Fewer teams than the client thinks they need

The most common structural mistake I see is a team for every project. It feels organised in the moment and it ages terribly. Six months on, the client has forty teams, most of them half-dead, and no one can find anything.

The fix I push is boring and it works: a team should map to a group of people who work together regularly, not to a task. Channels carry the topics. So instead of spinning up “Q3 Campaign”, the marketing team already exists, and Q3 Campaign is a channel inside it. When the campaign ends, you archive a channel, not a whole team. The people, the files in the SharePoint site behind it, and the history all stay in one sensible place.

I tell clients to design the teams first and resist the reflex to create. Every new team is a new SharePoint site, a new permissions boundary, another thing to govern.

Private and shared channels are the other trap. They have their place, but I see them thrown around to hide a conversation between two people, and every one of them quietly spins up its own SharePoint site behind the scenes. I treat them as the exception, not the reflex. If half your channels are private, you haven’t structured a team — you’ve built a warren.

Ownership is a structural decision, not an afterthought

The other quiet failure is ownership. A team gets created by whoever had the idea, they leave, and now it’s an orphan — no owner, no one to manage membership, guest access frozen in whatever state it was left in.

I won’t stand up a team now without at least two owners named on day one. I set expiration and archiving policies so dormant teams get flagged instead of accumulating forever, and I lean on Purview and the group settings in Entra to keep naming and guest access consistent across the client’s tenant. Structure isn’t just the shape of the teams. It’s who’s responsible when the shape needs to change.

Structure is what Copilot actually reads

Here’s the part that’s changed my thinking. When a client asks Copilot in Teams to summarise a project or pull together what was decided, Copilot reads across their channels, chats and the files sitting in those SharePoint sites. If that content is scattered over a dozen abandoned teams with three copies of the same document, the answer comes back vague or wrong — and the client blames Copilot.

It isn’t Copilot. It’s the structure underneath it. A tidy, well-owned set of teams is the difference between a genuinely useful summary and a confident piece of nonsense.

So my order of operations for clients has flipped. I do the Teams structure work before we switch Copilot on, not after they complain. Get the containers right, get ownership right, get the content living in the right place — then let Copilot loose on something worth reading.

Tidy was always nice to have. Now it’s what makes the AI worth paying for.

Comparing LLMs in Copilot services–Round 2 – Researcher

image

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Round 1 Winner – Copilot Chat = Opus

This is round 2 of my Copilot LLM battle, with the aim to determine the ‘best’ result from all the AI services and all the available LLMs used with Copilot in Microsoft 365.

The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round is all the possibilities with Researcher. As before, rathet than duplicate the reports here I have uploaded them to my Githuv repository in markdown format:

SharePoint Copilot assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher.md

Gemini assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher-Gemini-eval.md

Results (out of 10):

Critique – 9.30

Claude – 8.88

Council – 7.83

Auto – 7.63

Notes – SharePoint and Gemini produced very different rankings. I’m sticking with SharePoint’s assessment for consistency. I will also point out that gettign Gemini to do this comparison and produce a simple markdown file with the result was pretty much impossible. It does a really poor job on this simple comparison ask as you can see from the results.

So the Round 2 winner – Copilot Researcher = Critique

Onto Round 3

Copilot said: What People Say, What They Think, and What They Actually Do

image

Many years ago, David Ogilvy made an observation that still rings true today:

“The trouble with market research is that people don’t think what they feel, they don’t say what they think and they don’t do what they say.”

The older I get, the more I see this playing out in technology.

If you spend any time talking to business owners, managers, or end users about artificial intelligence and Microsoft 365 Copilot, you’ll hear all sorts of opinions. Some people tell me they’re excited. Others say they’re cautious. Many claim they’re waiting to see where the technology goes before investing further.

Then I look at what they’re actually doing.

That’s where things get interesting.

Watching Behaviour Tells You More Than Listening to Opinions

Over the last year I’ve had countless conversations with organisations evaluating Microsoft 365 Copilot. The feedback is often remarkably similar.

“We’re not sure we’ll use it enough.”

“Our staff aren’t ready.”

“We don’t have enough use cases yet.”

“We want to be careful before spending money.”

All perfectly reasonable concerns.

Yet when I sit down with those same organisations and look at how people work, I see something different. Staff are already using AI tools on their phones, in their browsers, and in personal accounts. They’re pasting email drafts into consumer AI services. They’re asking AI to summarise documents. They’re using it to help write proposals, prepare presentations, and analyse spreadsheets.

In other words, their behaviour is already telling me they’ve accepted AI. They just haven’t consciously recognised it yet.

The actions are often ahead of the stated position.

The Same Thing Happens Inside Microsoft 365

One of the most revealing exercises is simply watching how people use the tools they already have.

Someone may tell me they don’t have enough information to write a client proposal faster. Then I ask Microsoft 365 Copilot in Word to create a first draft from existing documents stored in SharePoint.

Someone insists meetings take as long as they’ve always taken. Then Copilot in Teams produces a meeting recap, extracts actions, and highlights decisions in a few seconds.

Someone believes they’re spending hours processing email because that’s the nature of modern work. Then Copilot in Outlook starts drafting replies and summarising lengthy email chains.

The surprising part isn’t that the technology works.

The surprising part is how quickly people adapt once they see it working in their own environment.

What they thought they would do is often very different from what they actually end up doing.

Technology Adoption Has Always Been Emotional

I think we sometimes treat technology decisions as purely logical exercises.

They’re not.

People rarely make decisions based solely on facts, specifications, or business cases. Emotion plays a huge role. Fear of change. Fear of looking foolish. Fear of making the wrong investment. Sometimes even fear of becoming less valuable.

That’s why surveys and focus groups often struggle to predict outcomes accurately. People answer based on how they imagine they’ll behave in the future.

Reality arrives later.

The same pattern appears with Microsoft 365 Copilot. Many organisations start cautiously, perhaps with a small pilot group. The initial conversations focus on risks, limitations, and concerns.

A few weeks later the discussion changes.

Users begin sharing prompts with each other. Teams start discovering new ways to use Copilot in meetings. Managers realise they can get through information faster. Gradually the conversation moves from whether they’ll use AI to where they should use it next.

The behaviour changes first. The mindset follows.

Pay Attention to What Happens, Not Just What Is Said

When I’m helping organisations assess readiness for AI, I place far more value on observed behaviour than survey responses.

The question isn’t whether someone says AI will help them.

The question is whether they reach for it when they’re busy.

Do they use Copilot to prepare for a meeting?

Do they ask Copilot in Outlook to help clear their inbox?

Do they use Copilot in Excel when they need answers from a spreadsheet?

Those actions reveal far more than any questionnaire.

David Ogilvy understood something fundamental about human nature. People are often poor predictors of their future behaviour.

As Microsoft 365 Copilot continues to become part of everyday work, I think we’ll see the same pattern repeat itself. The organisations that succeed won’t necessarily be the ones that talk most enthusiastically about AI.

They’ll be the ones that quietly incorporate it into their daily habits.

Because when it comes to technology adoption, what people do has always mattered more than what they say.