Comparing LLMs in Copilot services–Round 3 – Cowork (Claude)

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Winner – Opus

Round 2 – https://blog.ciaops.com/2026/08/08/comparing-llms-in-copilot-services-round-2-researcher/

Winner – Critique

I’ve been pitting different LLMs inside M365 Copilot against each other ina world cup style elimination to see which comes out on top. The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round the battle is between all the models available in Copilot Cowork Claude. Namely, these:

The other interesting factor here is that all these models as PAYG, so I’ll also give the costs for the same prompt.

To assess all these results together I found Gemini and SharePoint Copilot to both fail completely. Gemini keeps saying that it can’t find all the document, even though I have uploaded and also tried linking. SharePoint Copilot on the other hand starts processing but never gives me a result, no matter how long I wait. I will admit that these files are quite long (20+ pages) and quite complex, so there is a lot to digest. That said I threw them into Grok and those results are here:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260811-Cowork-Claude-Grok-eval.md

In summary, the ranking where:

1. Opus 5

2. Fable 5 (Preview)

3. Fable 5 (Copilot) (Preview)

4. Opus 4.8

5. Sonnet 5

which all kind of makes empirical sense but still quite subjective I feel. However, for now I’ll swap to using Grok to evaluate these documents as a standard approach.

Now for the costs which don’t factor into the results:

Claude Fable 5 (Preview) = 3,387 credits
Claude Fable 5 (Copilot)(Preview) = 3,373.8 credits
Sonnet 5 = 2,509.3 credits
Opus 4.8 = 2,487 credits
Opus 5 = 2,240 credits

Which, on initial analysis, indicates that costs for all models are in the same kind of range (i.e. at least US$20 per prompt), with Fable 5 models being about 50% more expensive.

Given the analysis and costs, it seems Opus 5 with Cowork, if you are using Claude, is the most cost effective for the best result in Cowork.

So:,

Round 3 winner – Copilot Cowork Claude = Opus 5

Onto the next round.

Is Copilot Listening to Every Meeting? The Reality Is More Complicated

Every few weeks someone asks me a variation of the same question.

“Is Microsoft Copilot recording all my meetings?”

The concern is understandable. AI is becoming more capable, meeting summaries appear almost instantly, action items seem to materialise out of nowhere, and people naturally wonder whether Microsoft is quietly capturing everything that happens in Teams. Sometimes the question goes a step further: “Does it do this even if I don’t have a Copilot licence?”

In my experience, the answer is far less dramatic than many people expect.

The concern isn’t really about Copilot. It’s about visibility.

The Difference Between a Meeting and a Recording

I think a lot of confusion comes from people bundling several different technologies together and calling them all “Copilot”.

A Teams meeting can exist without being recorded.

A Teams meeting can be transcribed without being recorded.

A Teams meeting can be recorded and transcribed.

And Copilot may or may not be involved at all.

That’s an important distinction.

Copilot doesn’t magically create information from nowhere. It works with the information available to it. If your organisation has enabled meeting transcription or recording, that’s where much of the content comes from. Copilot can then use that information to answer questions, create summaries, identify action items and help participants catch up.

Without that underlying data, Copilot has far less to work with.

The key point is that recording and transcription settings are administrative decisions. They aren’t automatically triggered simply because somebody owns a Copilot licence.

What About People Without Copilot?

This is where things often become misunderstood.

I’ve spoken with organisations where only a handful of staff have Microsoft 365 Copilot licences, yet meeting transcripts still exist across the business. When employees see AI-generated meeting summaries, the assumption is that Copilot must be capturing everything.

In reality, those organisations may have enabled Teams transcription or recording policies independently of Copilot.

Think about it this way. Your ability to access information and Microsoft’s ability to store information are not the same thing.

Someone without a Copilot licence may still participate in a meeting that is being recorded or transcribed. The meeting data exists because the meeting organiser or organisation allowed that functionality. Whether an individual user has access to Copilot features is a separate licensing decision.

That’s a subtle difference, but an important one.

The Governance Question Most Businesses Miss

What I find interesting is that many organisations ask whether Copilot is recording meetings, but they rarely ask who should have access to the resulting information.

That’s the more important conversation.

If meeting recordings are stored in OneDrive or SharePoint, if transcripts are available through Teams, and if Copilot can surface information that already exists, then governance matters more than fear.

I’ve seen organisations spend hours debating AI risks while leaving years of meeting recordings scattered across Microsoft 365 with inconsistent permissions.

Copilot didn’t create that problem. It simply made the problem more visible.

A useful exercise is to ask Copilot in Teams or Microsoft 365 a question about a recent meeting and then investigate why the answer was possible. The answer usually points back to existing recordings, transcripts, shared files or meeting notes that were already being retained.

AI acts as a spotlight. It doesn’t necessarily create new information.

The Question I’m Watching Closely

As Copilot adoption grows, I think we’ll see a shift in how organisations think about meetings.

For years, many businesses treated meetings as temporary conversations. Once the meeting ended, most people assumed the discussion disappeared into the ether unless someone took notes.

That assumption no longer holds true.

Whether you’re using Copilot, Teams transcription, meeting recordings or a combination of all three, conversations increasingly become searchable organisational knowledge.

The question therefore isn’t whether Copilot is secretly listening to every meeting.

The better question is whether your organisation understands what meeting data is being captured, where it is stored, how long it is retained and who can access it.

That’s where the real risk lives.

And, just as importantly, that’s where the real value of Microsoft 365 Copilot begins.

Comparing LLMs in Copilot services–Round 2 – Researcher

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Round 1 Winner – Copilot Chat = Opus

This is round 2 of my Copilot LLM battle, with the aim to determine the ‘best’ result from all the AI services and all the available LLMs used with Copilot in Microsoft 365.

The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round is all the possibilities with Researcher. As before, rathet than duplicate the reports here I have uploaded them to my Githuv repository in markdown format:

SharePoint Copilot assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher.md

Gemini assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher-Gemini-eval.md

Results (out of 10):

Critique – 9.30

Claude – 8.88

Council – 7.83

Auto – 7.63

Notes – SharePoint and Gemini produced very different rankings. I’m sticking with SharePoint’s assessment for consistency. I will also point out that gettign Gemini to do this comparison and produce a simple markdown file with the result was pretty much impossible. It does a really poor job on this simple comparison ask as you can see from the results.

So the Round 2 winner – Copilot Researcher = Critique

Onto Round 3

Copilot said: What People Say, What They Think, and What They Actually Do

Many years ago, David Ogilvy made an observation that still rings true today:

“The trouble with market research is that people don’t think what they feel, they don’t say what they think and they don’t do what they say.”

The older I get, the more I see this playing out in technology.

If you spend any time talking to business owners, managers, or end users about artificial intelligence and Microsoft 365 Copilot, you’ll hear all sorts of opinions. Some people tell me they’re excited. Others say they’re cautious. Many claim they’re waiting to see where the technology goes before investing further.

Then I look at what they’re actually doing.

That’s where things get interesting.

Watching Behaviour Tells You More Than Listening to Opinions

Over the last year I’ve had countless conversations with organisations evaluating Microsoft 365 Copilot. The feedback is often remarkably similar.

“We’re not sure we’ll use it enough.”

“Our staff aren’t ready.”

“We don’t have enough use cases yet.”

“We want to be careful before spending money.”

All perfectly reasonable concerns.

Yet when I sit down with those same organisations and look at how people work, I see something different. Staff are already using AI tools on their phones, in their browsers, and in personal accounts. They’re pasting email drafts into consumer AI services. They’re asking AI to summarise documents. They’re using it to help write proposals, prepare presentations, and analyse spreadsheets.

In other words, their behaviour is already telling me they’ve accepted AI. They just haven’t consciously recognised it yet.

The actions are often ahead of the stated position.

The Same Thing Happens Inside Microsoft 365

One of the most revealing exercises is simply watching how people use the tools they already have.

Someone may tell me they don’t have enough information to write a client proposal faster. Then I ask Microsoft 365 Copilot in Word to create a first draft from existing documents stored in SharePoint.

Someone insists meetings take as long as they’ve always taken. Then Copilot in Teams produces a meeting recap, extracts actions, and highlights decisions in a few seconds.

Someone believes they’re spending hours processing email because that’s the nature of modern work. Then Copilot in Outlook starts drafting replies and summarising lengthy email chains.

The surprising part isn’t that the technology works.

The surprising part is how quickly people adapt once they see it working in their own environment.

What they thought they would do is often very different from what they actually end up doing.

Technology Adoption Has Always Been Emotional

I think we sometimes treat technology decisions as purely logical exercises.

They’re not.

People rarely make decisions based solely on facts, specifications, or business cases. Emotion plays a huge role. Fear of change. Fear of looking foolish. Fear of making the wrong investment. Sometimes even fear of becoming less valuable.

That’s why surveys and focus groups often struggle to predict outcomes accurately. People answer based on how they imagine they’ll behave in the future.

Reality arrives later.

The same pattern appears with Microsoft 365 Copilot. Many organisations start cautiously, perhaps with a small pilot group. The initial conversations focus on risks, limitations, and concerns.

A few weeks later the discussion changes.

Users begin sharing prompts with each other. Teams start discovering new ways to use Copilot in meetings. Managers realise they can get through information faster. Gradually the conversation moves from whether they’ll use AI to where they should use it next.

The behaviour changes first. The mindset follows.

Pay Attention to What Happens, Not Just What Is Said

When I’m helping organisations assess readiness for AI, I place far more value on observed behaviour than survey responses.

The question isn’t whether someone says AI will help them.

The question is whether they reach for it when they’re busy.

Do they use Copilot to prepare for a meeting?

Do they ask Copilot in Outlook to help clear their inbox?

Do they use Copilot in Excel when they need answers from a spreadsheet?

Those actions reveal far more than any questionnaire.

David Ogilvy understood something fundamental about human nature. People are often poor predictors of their future behaviour.

As Microsoft 365 Copilot continues to become part of everyday work, I think we’ll see the same pattern repeat itself. The organisations that succeed won’t necessarily be the ones that talk most enthusiastically about AI.

They’ll be the ones that quietly incorporate it into their daily habits.

Because when it comes to technology adoption, what people do has always mattered more than what they say.

The Colour of Your Calendar Tells the Truth

I’ve become a big fan of colour-coded calendars over the years. Not because they look neat. Not because they make me feel organised. And certainly not because I enjoy spending time categorising appointments.

I like them because they tell the truth.

The reality is that most of us have a very different perception of how we spend our time compared to what actually happens. Ask someone where their week went and you’ll usually get an answer based on memory, emotion or whatever felt busiest. Open a colour-coded calendar and suddenly the conversation changes.

The evidence is right there in front of you.

And sometimes it’s not what you expected to see.

Busy Doesn’t Mean Progress

One of the first things a colour-coded calendar exposes is whether you’re actually moving forward or simply staying occupied.

I’ve seen weeks where the calendar looked completely full. Every day packed from morning until late afternoon. At first glance it felt productive.

Then I looked at the colours.

Almost every block was internal meetings.

Status updates. Catch-ups. Reviews. Discussions about discussions.

There wasn’t much time allocated to activities that improved the business, created something new, built a process, documented knowledge or generated future value.

That’s a confronting realisation.

Many business owners and IT professionals spend their entire week responding to what’s already happening instead of creating what happens next. A simple colour system makes that visible very quickly.

The calendar doesn’t care how important the meeting felt at the time. It simply shows where the hours went.

Seeing the Cost of Context Switching

Another thing colour coding reveals is how fragmented your day has become.

You start with a customer conversation. Then jump into Outlook to clear email. Then sit through a Teams meeting. Then back to customer work. Then another meeting. Then more email.

By lunchtime you’ve changed focus so many times that it feels like you’ve been running all morning but haven’t actually completed anything meaningful.

I see this regularly with MSPs and small business teams.

The work itself isn’t necessarily the problem. It’s the constant switching between different types of work.

When each category has its own colour, the pattern becomes obvious. Instead of solid blocks of concentrated effort, the calendar looks like confetti scattered across the day.

Once you can see it, it’s much easier to fix.

You begin grouping similar activities together. Customer calls sit together. Administration sits together. Internal meetings get pushed into dedicated windows.

The day starts working for you instead of against you.

Your Best Hours Matter Most

The biggest lesson I keep coming back to is how valuable the first few hours of the day really are.

That’s when my thinking is usually at its sharpest.

It’s when I want to write, plan, create content, build presentations or work with Microsoft 365 Copilot to develop ideas and explore new approaches. Whether I’m using Copilot in Word to draft an article or reviewing strategy notes from a Teams meeting, those activities benefit from clear thinking and uninterrupted attention.

Yet many calendars tell a different story.

The most productive hours are often handed over to routine meetings, administrative work or conversations that could easily happen later.

When I review a colour-coded calendar, that issue jumps out immediately.

The colours show whether those high-energy hours are being invested or simply consumed.

That’s an important distinction.

The Colours Never Argue

What I like most about colour coding is its honesty.

It removes assumptions.

It removes excuses.

It removes that comfortable story we tell ourselves about where our time goes.

Open your calendar and look at the next seven days.

Which colour dominates?

Customer work? Internal meetings? Administration? Personal development? Strategic planning?

Whatever colour appears most often is receiving the largest investment of your time.

And over the long term, that’s probably where your business, career and attention are heading as well.

The colours won’t tell you what to do.

But they will show you the truth.

And that’s usually the first step towards making better decisions.

Stop Waiting for Perfect Conditions

I hear versions of the same story all the time.

Someone wants to learn Microsoft 365 Copilot. Start a side project. Improve their business processes. Clean up their Microsoft 365 tenant. Build a new service offering for their clients.

But not yet.

They’re waiting for a quieter week. Waiting for a project to finish. Waiting for school holidays to end. Waiting for the inbox to settle down.

The problem is that week never arrives.

I’ve been in technology long enough to know that there is always another project. Another client request. Another issue needing attention. If you’re running an MSP or managing an SMB, the workload doesn’t magically disappear one day and leave you with hours of spare time.

That’s why I’ve come to believe that constraints aren’t the thing stopping you.

More often than not, they’re the thing that should be driving you forward.

The Reality of Limited Time

A lot of people treat constraints like roadblocks.

“I don’t have enough time.”

“I don’t have enough money.”

“I don’t have enough people.”

Those things may all be true. But they’re also true for almost everyone else.

The difference is how you respond to them.

When I work with organisations deploying Microsoft 365 Copilot, the most successful ones rarely have unlimited resources. They’re not sitting around with spare staff and empty calendars. They’re already busy.

What they do have is a willingness to work within the constraint rather than waiting for it to disappear.

Instead of trying to train every employee at once, they start with five people.

Instead of redesigning every process in the business, they pick one recurring task in Outlook or Teams and improve it.

Instead of creating the perfect governance framework, they begin with the basics and build from there.

The constraint forces focus.

And focus is often worth more than freedom.

Small Boundaries Create Better Decisions

One thing I’ve noticed over the years is that too many options can be just as dangerous as too few.

Give someone unlimited time and unlimited resources and they often spend months planning.

Give them one hour a week and suddenly they become remarkably efficient.

I’ve seen MSPs create entirely new AI service offerings because they only had a few hours available each week. The limited time meant they ignored all the distractions and concentrated on the activities that actually mattered.

The same applies to personal growth.

If you’ve only got thirty minutes available on a Friday afternoon, use it.

Open Copilot in Word and draft an article.

Use Copilot in Teams to summarise a meeting you missed.

Explore a new automation in Power Automate.

Read one Microsoft Learn article.

Thirty minutes won’t change your business overnight. But thirty minutes every week for a year absolutely will.

The point isn’t the amount of time.

It’s the consistency.

Stop Negotiating With the Future

What worries me most about the “I’ll start when things settle down” mindset is that it quietly trains us to delay action.

We convince ourselves we’re being sensible.

We’re really just postponing the uncomfortable part.

Learning something new feels hard.

Changing a process feels risky.

Starting a project creates uncertainty.

So we strike a deal with ourselves and push it into the future.

The future, of course, never objects.

It just keeps moving.

I’ve found that progress usually begins when we stop asking whether conditions are ideal and start asking what can be done with the resources available today.

That’s true whether you’re adopting Copilot, building an MSP business, improving your security posture, or simply trying to learn a new skill.

The Constraint Is the Opportunity

When I look back at the projects that made the biggest difference in my career, very few started under perfect conditions.

Most began when time was short, money was limited, or other priorities were competing for attention.

The limitation wasn’t the problem.

It was the catalyst.

So if you’re waiting for a quieter month, a bigger budget, or fewer commitments, I’d encourage you to rethink the situation.

The constraint you’re looking at today might not be the obstacle.

It might be the thing that finally forces you to focus on what matters most.

That’s usually where the real progress starts.

Writing Is Still Thinking (Even in the Age of Copilot)

A few weeks ago I caught myself doing something I suspect a lot of people are doing right now.

I had an idea. Instead of opening a blank document, I opened Microsoft 365 Copilot and started asking questions.

Within minutes I had summaries, suggestions, outlines, and options. It was impressive. It was also a little unsettling.

The reason was simple. I had an answer before I had really worked out what I thought about the question.

That’s something I’ve been reflecting on more as AI becomes a bigger part of our daily work. We often talk about AI helping us write, but I think the more important question is whether we’re still using writing to think.

The Value of a Blank Page

For most of my career, writing has been one of the ways I clarified my thinking.

Whether it was a client proposal, a blog post, meeting notes, or an internal strategy document, the process worked because it was slow enough to force decisions. You couldn’t hide gaps in your logic. When you sat staring at an empty page, every weak assumption became obvious.

That’s still true today.

When I open a Word or OneNote document and start typing, I often discover that the idea I thought was clear in my head isn’t quite as solid as I imagined. The act of putting words on a page exposes the fuzzy edges.

The danger with AI is that it can smooth over those edges too quickly. Copilot can generate something that sounds complete before you’ve done the hard work of deciding what you actually believe.

The output may be polished. Your thinking may not be.

Copilot Works Best as a Thinking Partner

This isn’t an argument against Microsoft 365 Copilot. Far from it.

What I’m seeing in organisations is that the most successful users don’t treat Copilot as a replacement for thinking. They use it as a way to challenge and refine their thinking.

A practical example might be drafting a proposal in Word. I often start with rough notes of my own. They are messy and incomplete. Only after I’ve worked through the problem do I ask Copilot to review the draft, identify gaps, suggest alternative approaches, or summarise the key points.

The sequence matters.

Thinking first. AI second.

If you reverse the order, there’s a risk that you’re evaluating someone else’s answer rather than exploring your own understanding.

That’s a subtle difference, but it’s an important one.

Faster Answers Aren’t Always Better Answers

Many of us spend our days moving between Outlook, Teams meetings, client conversations, and project work. The attraction of having instant answers everywhere is obvious.

The challenge is that genuine understanding usually takes longer than information retrieval.

Copilot can tell you what happened in a Teams meeting you missed. It can summarise a lengthy email thread in Outlook. It can pull key points from a SharePoint document in seconds.

That’s incredibly useful.

What it can’t do is decide what those things mean for your business, your clients, or your next decision.

That responsibility still sits with you.

I think that’s why many people who try AI for the first time are both impressed and disappointed. They’re impressed by the speed. They’re disappointed when speed alone doesn’t solve the problem they were really facing.

Knowledge is easy to access. Judgement is still hard work.

The Skill Worth Protecting

The organisations getting the most value from AI aren’t abandoning traditional skills. They’re strengthening them.

Clear writing. Critical thinking. Good questioning. Sound decision-making.

Those capabilities become more important as AI becomes more capable, not less.

The people who ask the best questions get better results from Copilot. The people who understand a problem deeply can spot when an AI-generated answer misses the point. The people who can write clearly can guide AI more effectively.

That’s what I’m watching most closely.

AI is changing how we work, but I don’t believe it changes the need to think carefully. If anything, it raises the value of that skill.

The blank page still matters.

Copilot may help us fill it faster, but the real value comes from understanding what deserves to be written there in the first place.

The Skill That Matters Most in the Age of Copilot: Asking Better Questions

The people getting the most value from Microsoft 365 Copilot today aren’t necessarily the most technical. They’re not the fastest typists, the best PowerShell writers or the ones with the biggest AI budget.

They’re the people who ask better questions.

That might sound obvious, but I think it’s one of the biggest mindset shifts happening right now. For years, many of us have been trained to search for information using a few keywords. Type something into Google, scan the results, click a link, repeat. The skill was finding information.

With Copilot, the skill is becoming framing the problem.

I see this regularly when working with SMBs and MSPs. Two people sit in front of the same Microsoft 365 Copilot environment. One gets a generic answer that doesn’t really help. The other gets a detailed summary, actionable recommendations and a useful first draft. The difference is rarely the technology.

It’s the question.

Most People Stop Too Early

A common pattern I see is someone opening Copilot in Teams or Outlook and typing a very short request:

“Summarise this.”

“Write an email.”

“Analyse this document.”

Copilot will do something, but not necessarily something valuable.

The better approach is to provide context. Why do you need the information? Who is the audience? What decision are you trying to make? What concerns do you already have?

The quality of the answer often improves dramatically when the question becomes more specific.

Think about a conversation with a trusted employee. If you walked into their office and said, “Help me with sales”, you’d get a confused look. If you said, “Review the last quarter of sales reports and identify the three biggest opportunities in our existing customer base”, you’d be much closer to getting a useful result.

Copilot works in a similar way.

Copilot Rewards Curiosity

One of the biggest mistakes I see is treating Copilot like a command-line interface.

People issue instructions instead of having a conversation.

The real value appears when you start exploring.

For example, after Copilot produces a meeting summary in Teams, don’t stop there. Ask what wasn’t discussed. Ask what risks were mentioned only briefly. Ask which actions have no assigned owner. Ask which topics are likely to create issues next month.

Each follow-up question improves your understanding.

What fascinates me is that the process starts to resemble working with a highly capable colleague. The first answer is rarely the destination. It’s often the starting point.

The people who get the best outcomes are usually the people who are naturally curious. They’re willing to ask one more question.

Then another.

This Changes How We Learn

I think we’re moving into a world where knowing everything becomes less important than knowing how to investigate effectively.

In the past, expertise often meant storing large amounts of information in your head. Today, much of that information can be surfaced instantly from SharePoint, OneDrive, Teams conversations and Outlook messages through Copilot.

The challenge becomes directing that capability effectively.

When I use Copilot in Word to help draft content or in Outlook to analyse a long email thread, I rarely accept the first result. I refine it. I challenge it. I ask for alternatives. I request different viewpoints.

In many cases, my role is becoming less about generating information and more about guiding the process that generates it.

That’s a different skill set altogether.

The Real Competitive Advantage

I don’t think the winners in the AI era will be the organisations with the most AI tools.

I think they’ll be the organisations that teach their people how to think clearly, define problems accurately and ask better questions.

Microsoft 365 Copilot is already showing us this reality. The technology is becoming easier to access every month. The differentiator isn’t the tool. It’s the person using it.

A poor question gives you a poor starting point.

A thoughtful question opens entirely different possibilities.

That’s why, when people ask me what skill they should develop to get more value from Copilot, my answer is increasingly simple:

Learn to ask better questions. Everything else gets easier from there.