Comparing LLMs in Copilot services–Round 5 – Cowork

image

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Winner – Opus

Round 2 – https://blog.ciaops.com/2026/08/08/comparing-llms-in-copilot-services-round-2-researcher/

Winner – Critique

Round 3  – https://blog.ciaops.com/2026/08/11/comparing-llms-in-copilot-services-round-3-cowork-claude/

Winner – Opus 5

Round 4 – https://blog.ciaops.com/2026/08/12/comparing-llms-in-copilot-services-round-4-cowork-gpt/

Winner – GPT 5.6 Terra

I’ve been pitting different LLMs inside M365 Copilot against each other ina world cup style elimination to see which comes out on top. The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

For this round I’ve pitted the Cowork Claude winner (Opus 5) vs the Cowork GPT winner (5.6 Terra). The result is:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260811-Cowork-Grok-eval.md

1. Opus 5.0

2. GPT 5.6 Terra

and this time the gap was much larger than before. Not unexpected given that Terra is designed as a lighter weight, less powerful model. However, don’t forget the cost factor which isn’t included in these evaluations:

Claude Fable 5 (Preview) = 3,387 credits
Claude Fable 5 (Copilot)(Preview) = 3,373.8 credits

GPT 5.6 Sol = 2,800.50 credits
Sonnet 5 = 2,509.3 credits
Opus 4.8 = 2,487 credits
Opus 5 = 2,240 credits

GPT 5.5 = 1,200 credits
GPT 5.6 Terra = 260 credits

If costs were indeed taken into account it would certain swing the pendulum back toward Terra.

However, this time the winner is clear

Opus 5

moves onto the next round.

The Client Journey Tells You Where the Business Really Is

image

There is a point in every service business where growth still looks busy, but it no longer feels healthy.

The calendar is full. Ads are running. Sales conversations are happening. New clients are arriving. From the outside, it looks like momentum.

Inside the business, something feels off.

You finish the month tired and the client count has hardly moved. Your team has handled more calls, answered more questions, chased more payments, and welcomed more people, yet the result barely reflects the effort. That is usually when I stop looking at the front door and start looking at the back one.

Because more leads will not fix a leaky experience.

Growth hides the problem until it does not

Early on, client loss can be easy to ignore. A few people leave and it does not feel significant. The business is small enough that enthusiasm covers rough edges.

Then the business gets bigger.

The same onboarding gaps become more expensive. The same unclear expectations create more friction. The same missed follow-ups turn into quiet exits. Nobody has to make a dramatic complaint for the damage to be real. Sometimes they just stop engaging, stop replying, stop showing up, and eventually stop paying.

This is where many owners make the obvious move. They spend more to bring more people in.

I understand why. Marketing feels active. Sales feels measurable. Pipeline gives everyone something to discuss in the weekly meeting. But if the client experience is not holding, all you have built is a costly replacement machine.

That is not growth. That is movement.

The first weeks set the whole relationship

I have become more interested in the first part of the client journey than almost any other part of the business.

Not because onboarding is glamorous. It is checklists, expectations, reminders, handovers, notes, meetings, small promises, and boring consistency.

But boring consistency is often where profit is hiding.

The first few weeks teach a client how your business works. They learn whether you are organised. They learn whether they need to chase you. They learn whether the promise they bought is becoming something tangible.

If that early experience is vague, the client starts filling in the gaps themselves. That is dangerous. Their version of the story may not be the one you intended.

This is where Microsoft 365 can do practical work. I would map the first client journey in a shared Word document or Loop workspace, turn the repeatable steps into Planner tasks, keep client context in a Teams channel, and use Copilot to summarise meeting notes, draft follow-up emails in Outlook, and identify action items that slipped.

None of that is magic. That is the point.

The value is not in making onboarding fancy. The value is in making it visible, repeatable, and harder to forget when the business gets busy.

Retention is an operating system

A lot of businesses treat retention as a customer service problem. I think that is too late.

Retention starts when the expectation is first set. It continues when the client understands what happens next. It improves when the team can see the same information, use the same process, and spot weak signals before they become cancellation emails.

You do not need a massive transformation project to start. Pick one client segment. Write down the first six weeks. Decide what each client should receive, what your team must do, what evidence shows progress, and where the handoffs fail.

Then put that into the tools your team already opens every day.

A better onboarding process will not remove every cancellation. Nothing does. But it changes the work. Instead of constantly buying attention from strangers, you start earning confidence from the people who already said yes.

That is the growth I trust more.

Not the noisy kind.

The kind that stays.

The Real Work Is Not Another Tactic

image

Most owners I meet are not short of effort. They are short of room.

Room to think. Room to make better decisions. Room to stop reacting to every urgent customer request, vendor announcement, staff issue, cashflow wobble, and half-finished idea sitting in the inbox.

That is why so many businesses end up looking busy but feeling fragile. The owner keeps tuning the visible parts of the machine. Better campaigns. Better processes. Better meetings. Useful, but none of it fixes the real constraint if the owner is still the bottleneck.

The business usually rises to the level of the person leading it.

That is an uncomfortable sentence. It should be.

The easy work looks productive

It is tempting to stay in work that gives visible proof of progress. Rewrite the website. Change the offer. Hire another person. Buy another app. Create another spreadsheet. Push harder on social. Start another initiative.

I have done versions of this myself. Most business owners have. It feels like discipline because there is activity everywhere.

But activity is not always advancement.

The harder work is looking at the recurring patterns and asking, “Why does this keep happening around me?” Why do the same decisions come back to my desk? Why do I avoid the awkward conversation until it becomes expensive? Why do I keep saying yes to work that does not fit?

That work rarely produces a neat announcement. But it is often where the real advantage is built.

Your operating system matters more than your tactics

I think about this a lot with Copilot and Microsoft 365.

Plenty of organisations are trying to use Copilot as a faster keyboard. Draft an email in Outlook. Summarise a meeting in Teams. Turn notes into a document in Word. All good uses.

But the more interesting use is not speed. It is reflection.

After a difficult client meeting, I can ask Copilot in Teams to help me identify the decisions, risks, and unresolved questions from the transcript. I can put the next actions into Planner, save the working document in SharePoint, and use that as the basis for a better follow-up in Outlook.

That is not just automation. That is a leadership loop.

The value is not that Copilot wrote some sentences for me. The value is that I forced myself to inspect the way I work. What did I miss? What did I delay? What needs to become a rule, not another heroic rescue?

A better business is usually built from better loops.

The real edge compounds quietly

The strongest operators I know are not chasing every new trick. They are harder to knock off balance.

They recover faster from mistakes. They make decisions with less drama. They communicate sooner. They document what matters. They know which work to refuse. They build teams that do not need constant rescue because the thinking has been made visible.

That is difficult to compete with because it is not one tactic someone can copy. It is a collection of habits, standards, judgement, and self-awareness built over time.

You can copy someone’s landing page. You can copy their pricing model. You can copy their tech stack.

You cannot easily copy the way they think under pressure.

That is where I believe owners should spend more attention. Not less work on the business, but better work on the person making the business decisions.

Use the tools. Use Copilot. Use Teams, Outlook, SharePoint, and Planner to create cleaner loops.

But do not confuse the tool with the transformation.

The business changes when the owner changes the way decisions are made, captured, reviewed, and improved.

That is the work most people avoid.

It is also the work that makes you hard to catch.

Comparing LLMs in Copilot services–Round 4 – Cowork (GPT)

image

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Winner – Opus

Round 2 – https://blog.ciaops.com/2026/08/08/comparing-llms-in-copilot-services-round-2-researcher/

Winner – Critique

Round 3  – https://blog.ciaops.com/2026/08/11/comparing-llms-in-copilot-services-round-3-cowork-claude/

Winner – Opus 5

I’ve been pitting different LLMs inside M365 Copilot against each other ina world cup style elimination to see which comes out on top. The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round the battle is between all the models available in Copilot Cowork GPT. Namely, these:

Screenshot 2026-08-12 082720

The other interesting factor here is that all these models as PAYG, so I’ll also give the costs for the same prompt.

I used Grok to evaluate all the results which produced:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260811-Cowork-GPT-Grok-eval.md

In summary, the rankings are:

1. 5.6 Terra

2. 5.6 Sol

3. 5.5

Interestingly, the less powerful model (Terra) produced a better result here.

And the costs of each:


GPT 5.6 Sol = 2,800.50 credits
GPT 5.6 Terra = 260 credits
GPT 5.5 = 1,200 credits

5.6 Terra wins again big here, only costing 260 credits! This is how the ranked costs table so far looks, from most to least expensive:

Claude Fable 5 (Preview) = 3,387 credits
Claude Fable 5 (Copilot)(Preview) = 3,373.8 credits

GPT 5.6 Sol = 2,800.50 credits
Sonnet 5 = 2,509.3 credits
Opus 4.8 = 2,487 credits
Opus 5 = 2,240 credits

GPT 5.5 = 1,200 credits
GPT 5.6 Terra = 260 credits

Given that Terra is so much cheaper I’ll need to spend some more time investigating and verifying that with other requests, given that these tests are simply a one shot prompt to result.

Drum roll. The clear winner for this round is:

GPT 5.6 Terra

With all the preliminaries done we can now get onto comparing the winners of each round together.

Why AI Charges More to Write Than to Read

image

I keep seeing people look at AI pricing pages and assume someone has made a typo.

Input tokens cost one amount. Output tokens cost another. Cached input may cost less again. At first glance that feels odd. A token is a token, surely?

Not really.

The easiest way to think about it is this: reading is cheaper than writing. AI can read a lot of your prompt in parallel. It can process the instructions, pasted document, previous conversation and system rules in large chunks. That is the input side.

Writing the answer is different. The model generates the response one token at a time. Each next word, fragment, line of code or JSON field depends on what came before it. The longer the response, the longer expensive compute is tied up producing it.

That is why output tokens usually cost more. You are not just paying for text. You are paying for generation.

This matters more than most people think

Once you move into agents, automation, Copilot Studio, Azure OpenAI, GitHub Copilot, or anything that runs repeatedly, the economics change quickly.

A short prompt that produces a long report can cost more than a large prompt that produces a tiny answer. That surprises people. They assume the big document is the expensive part. Sometimes it is not. The real cost can be the polished, verbose output they asked the model to create.

Ask an AI system to read a SharePoint policy library and return three risks, and you are probably dealing with an input-heavy, output-light workload. That can be relatively efficient.

Ask it to create a 25-page report, an executive summary, a remediation plan, a Teams post, a client email and a formatted table every time it runs, and you have created an output-heavy workload. That is where the bill starts to move.

The MSP lesson is simple: design the output

This is where MSPs need to stop treating AI as magic and start treating it as infrastructure.

When we built servers, we cared about CPU, RAM, disk and backup windows. With AI, we need to care about prompts, context, output length, caching and repeatability.

The bad habit is asking for everything every time. “Give me the full report.” “Include all the detail.” “Make it comprehensive.” That sounds harmless until the same workflow runs fifty times across fifty tenants.

A better approach is to be deliberate. Ask for the smallest useful output first. Use summaries where summaries are enough. Generate detailed reports only when there is a reason. Reuse stable instructions and context where caching is available. Put spending limits around anything consumption-based. Review what the agent writes, not just what it reads.

Inside Microsoft 365, this means being clear about the difference between ordinary Copilot use in Word, Excel, Outlook or Teams and consumption-based AI work that may be billed differently. A user drafting an email is one thing. An agent chewing through documents and producing long artefacts all day is another.

This is not a pricing trick

I do not see the input/output price split as some mysterious vendor tax. It reflects how the technology behaves.

The mistake is pretending it does not matter.

AI costs are not just about how many people have a licence. They are about what those people, agents and workflows ask the model to produce. The output is where the hidden weight often sits.

So the practical rule is this: do not just prompt for the result. Design the cost shape of the result.

That might be the difference between AI being a useful business tool and AI becoming the next cloud bill nobody wants to open.

Comparing LLMs in Copilot services–Round 3 – Cowork (Claude)

image

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Winner – Opus

Round 2 – https://blog.ciaops.com/2026/08/08/comparing-llms-in-copilot-services-round-2-researcher/

Winner – Critique

I’ve been pitting different LLMs inside M365 Copilot against each other ina world cup style elimination to see which comes out on top. The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round the battle is between all the models available in Copilot Cowork Claude. Namely, these:

Screenshot 2026-08-11 074851

The other interesting factor here is that all these models as PAYG, so I’ll also give the costs for the same prompt.

To assess all these results together I found Gemini and SharePoint Copilot to both fail completely. Gemini keeps saying that it can’t find all the document, even though I have uploaded and also tried linking. SharePoint Copilot on the other hand starts processing but never gives me a result, no matter how long I wait. I will admit that these files are quite long (20+ pages) and quite complex, so there is a lot to digest. That said I threw them into Grok and those  results are here:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260811-Cowork-Claude-Grok-eval.md

In summary, the ranking where:

1. Opus 5

2. Fable 5 (Preview)

3. Fable 5 (Copilot) (Preview)

4. Opus 4.8

5. Sonnet 5

which all kind of makes empirical sense but still quite subjective I feel. However, for now I’ll swap to using Grok to evaluate these documents as a standard approach.

Now for the costs which don’t factor into the results:

Claude Fable 5 (Preview) = 3,387 credits
Claude Fable 5 (Copilot)(Preview) = 3,373.8 credits
Sonnet 5 = 2,509.3 credits
Opus 4.8 = 2,487 credits
Opus 5 = 2,240 credits


Which, on initial analysis, indicates that costs for all models are in the same kind of range (i.e. at least US$20 per prompt), with Fable 5 models being about 50% more expensive.

Given the analysis and costs, it seems Opus 5 with Cowork, if you are using Claude, is the most cost effective for the best result in Cowork.

So:,

Round 3 winner – Copilot Cowork Claude = Opus 5

Onto the next round.

Security Keeps the Bad Guys Out. Compliance Stops the Good Guys Doing the Wrong Thing.

image

One of the biggest mistakes I see when talking to organisations about Microsoft 365 is that they treat security and compliance as the same thing.

They’re not.

In fact, I often explain it this way: security is largely about protecting your data from people who shouldn’t have access to it, while compliance is about making sure people who do have access don’t misuse it.

Both matter. Both are essential. But they solve completely different problems.

The Security Mindset

When most people think about protecting information, their mind immediately goes to external threats.

Hackers trying to steal credentials.

Phishing emails landing in inboxes.

Ransomware attempting to encrypt files.

Business email compromise targeting finance teams.

That’s the world of security. The goal is to prevent unauthorised access and stop attackers before they can do damage.

This is where tools like Microsoft Defender, Conditional Access, Entra ID, multifactor authentication and device management come into play. They create barriers around your data and make it harder for outsiders to gain access.

The challenge is that many organisations stop there.

They invest heavily in security controls, get comfortable that their tenant is well protected, and assume the job is done.

It isn’t.

Because once someone is legitimately inside the organisation, security has largely done its job.

That’s where compliance starts.

The Compliance Challenge

Imagine an employee has access to customer records as part of their daily role.

They’re not a hacker.

They’re not bypassing security controls.

They’re simply using information they have permission to access.

Now imagine they accidentally email a spreadsheet containing sensitive customer information to the wrong person.

Or upload confidential financial data to an unauthorised location.

Or paste company information into an AI tool that hasn’t been approved by the organisation.

Security didn’t fail here.

The employee had legitimate access.

The issue is what happened after access was granted.

That’s a compliance problem.

Compliance is about governing how information is used, shared, stored and protected after someone is authorised to see it.

Why Copilot Makes This More Important

The rise of Microsoft 365 Copilot makes this distinction even more critical.

Copilot operates using the permissions that already exist within your Microsoft 365 environment. If a user has access to information in SharePoint, OneDrive, Teams or Exchange Online, Copilot can help them work with that data.

That’s why I continually tell organisations that Copilot readiness isn’t just about licensing. It’s about information governance.

I’ve seen organisations excited about deploying Copilot only to discover that sensitive documents are stored in locations where far too many people have access. The problem isn’t Copilot. The problem is that existing permissions and governance weaknesses become much more visible once AI enters the picture.

If your data is overshared, Copilot can expose that reality very quickly.

Security and Compliance Need Equal Attention

The best organisations understand that security and compliance are two sides of the same coin.

Security asks:

“Who should be allowed in?”

Compliance asks:

“What should they be allowed to do once they’re inside?”

In Microsoft 365, that means combining strong security controls with technologies such as Microsoft Purview, sensitivity labels, data loss prevention policies, retention controls and information protection.

One protects the front door.

The other governs what happens in the rooms beyond it.

Ignore either one and you’re leaving your organisation exposed.

Final Thoughts

I think many businesses are still more comfortable talking about security than compliance because security threats are easier to visualise. We can picture a hacker attacking our systems.

What’s harder to visualise is an employee accidentally sharing the wrong file, retaining information for too long, or exposing sensitive data through everyday work practices.

Yet those risks can be just as damaging.

As AI becomes more deeply woven into Microsoft 365, the organisations that succeed won’t simply have the strongest security. They’ll also have the maturity to govern their information properly.

Security protects your organisation from outsiders.

Compliance protects your organisation from itself.

And increasingly, you need both.

Is Copilot Listening to Every Meeting? The Reality Is More Complicated

image

Every few weeks someone asks me a variation of the same question.

“Is Microsoft Copilot recording all my meetings?”

The concern is understandable. AI is becoming more capable, meeting summaries appear almost instantly, action items seem to materialise out of nowhere, and people naturally wonder whether Microsoft is quietly capturing everything that happens in Teams. Sometimes the question goes a step further: “Does it do this even if I don’t have a Copilot licence?”

In my experience, the answer is far less dramatic than many people expect.

The concern isn’t really about Copilot. It’s about visibility.

The Difference Between a Meeting and a Recording

I think a lot of confusion comes from people bundling several different technologies together and calling them all “Copilot”.

A Teams meeting can exist without being recorded.

A Teams meeting can be transcribed without being recorded.

A Teams meeting can be recorded and transcribed.

And Copilot may or may not be involved at all.

That’s an important distinction.

Copilot doesn’t magically create information from nowhere. It works with the information available to it. If your organisation has enabled meeting transcription or recording, that’s where much of the content comes from. Copilot can then use that information to answer questions, create summaries, identify action items and help participants catch up.

Without that underlying data, Copilot has far less to work with.

The key point is that recording and transcription settings are administrative decisions. They aren’t automatically triggered simply because somebody owns a Copilot licence.

What About People Without Copilot?

This is where things often become misunderstood.

I’ve spoken with organisations where only a handful of staff have Microsoft 365 Copilot licences, yet meeting transcripts still exist across the business. When employees see AI-generated meeting summaries, the assumption is that Copilot must be capturing everything.

In reality, those organisations may have enabled Teams transcription or recording policies independently of Copilot.

Think about it this way. Your ability to access information and Microsoft’s ability to store information are not the same thing.

Someone without a Copilot licence may still participate in a meeting that is being recorded or transcribed. The meeting data exists because the meeting organiser or organisation allowed that functionality. Whether an individual user has access to Copilot features is a separate licensing decision.

That’s a subtle difference, but an important one.

The Governance Question Most Businesses Miss

What I find interesting is that many organisations ask whether Copilot is recording meetings, but they rarely ask who should have access to the resulting information.

That’s the more important conversation.

If meeting recordings are stored in OneDrive or SharePoint, if transcripts are available through Teams, and if Copilot can surface information that already exists, then governance matters more than fear.

I’ve seen organisations spend hours debating AI risks while leaving years of meeting recordings scattered across Microsoft 365 with inconsistent permissions.

Copilot didn’t create that problem. It simply made the problem more visible.

A useful exercise is to ask Copilot in Teams or Microsoft 365 a question about a recent meeting and then investigate why the answer was possible. The answer usually points back to existing recordings, transcripts, shared files or meeting notes that were already being retained.

AI acts as a spotlight. It doesn’t necessarily create new information.

The Question I’m Watching Closely

As Copilot adoption grows, I think we’ll see a shift in how organisations think about meetings.

For years, many businesses treated meetings as temporary conversations. Once the meeting ended, most people assumed the discussion disappeared into the ether unless someone took notes.

That assumption no longer holds true.

Whether you’re using Copilot, Teams transcription, meeting recordings or a combination of all three, conversations increasingly become searchable organisational knowledge.

The question therefore isn’t whether Copilot is secretly listening to every meeting.

The better question is whether your organisation understands what meeting data is being captured, where it is stored, how long it is retained and who can access it.

That’s where the real risk lives.

And, just as importantly, that’s where the real value of Microsoft 365 Copilot begins.