Comparing LLMs in Copilot services–Round 2 – Researcher

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Round 1 Winner – Copilot Chat = Opus

This is round 2 of my Copilot LLM battle, with the aim to determine the ‘best’ result from all the AI services and all the available LLMs used with Copilot in Microsoft 365.

The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round is all the possibilities with Researcher. As before, rathet than duplicate the reports here I have uploaded them to my Githuv repository in markdown format:

SharePoint Copilot assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher.md

Gemini assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher-Gemini-eval.md

Results (out of 10):

Critique – 9.30

Claude – 8.88

Council – 7.83

Auto – 7.63

Notes – SharePoint and Gemini produced very different rankings. I’m sticking with SharePoint’s assessment for consistency. I will also point out that gettign Gemini to do this comparison and produce a simple markdown file with the result was pretty much impossible. It does a really poor job on this simple comparison ask as you can see from the results.

So the Round 2 winner – Copilot Researcher = Critique

Onto Round 3

Writing Is Still Thinking (Even in the Age of Copilot)

A few weeks ago I caught myself doing something I suspect a lot of people are doing right now.

I had an idea. Instead of opening a blank document, I opened Microsoft 365 Copilot and started asking questions.

Within minutes I had summaries, suggestions, outlines, and options. It was impressive. It was also a little unsettling.

The reason was simple. I had an answer before I had really worked out what I thought about the question.

That’s something I’ve been reflecting on more as AI becomes a bigger part of our daily work. We often talk about AI helping us write, but I think the more important question is whether we’re still using writing to think.

The Value of a Blank Page

For most of my career, writing has been one of the ways I clarified my thinking.

Whether it was a client proposal, a blog post, meeting notes, or an internal strategy document, the process worked because it was slow enough to force decisions. You couldn’t hide gaps in your logic. When you sat staring at an empty page, every weak assumption became obvious.

That’s still true today.

When I open a Word or OneNote document and start typing, I often discover that the idea I thought was clear in my head isn’t quite as solid as I imagined. The act of putting words on a page exposes the fuzzy edges.

The danger with AI is that it can smooth over those edges too quickly. Copilot can generate something that sounds complete before you’ve done the hard work of deciding what you actually believe.

The output may be polished. Your thinking may not be.

Copilot Works Best as a Thinking Partner

This isn’t an argument against Microsoft 365 Copilot. Far from it.

What I’m seeing in organisations is that the most successful users don’t treat Copilot as a replacement for thinking. They use it as a way to challenge and refine their thinking.

A practical example might be drafting a proposal in Word. I often start with rough notes of my own. They are messy and incomplete. Only after I’ve worked through the problem do I ask Copilot to review the draft, identify gaps, suggest alternative approaches, or summarise the key points.

The sequence matters.

Thinking first. AI second.

If you reverse the order, there’s a risk that you’re evaluating someone else’s answer rather than exploring your own understanding.

That’s a subtle difference, but it’s an important one.

Faster Answers Aren’t Always Better Answers

Many of us spend our days moving between Outlook, Teams meetings, client conversations, and project work. The attraction of having instant answers everywhere is obvious.

The challenge is that genuine understanding usually takes longer than information retrieval.

Copilot can tell you what happened in a Teams meeting you missed. It can summarise a lengthy email thread in Outlook. It can pull key points from a SharePoint document in seconds.

That’s incredibly useful.

What it can’t do is decide what those things mean for your business, your clients, or your next decision.

That responsibility still sits with you.

I think that’s why many people who try AI for the first time are both impressed and disappointed. They’re impressed by the speed. They’re disappointed when speed alone doesn’t solve the problem they were really facing.

Knowledge is easy to access. Judgement is still hard work.

The Skill Worth Protecting

The organisations getting the most value from AI aren’t abandoning traditional skills. They’re strengthening them.

Clear writing. Critical thinking. Good questioning. Sound decision-making.

Those capabilities become more important as AI becomes more capable, not less.

The people who ask the best questions get better results from Copilot. The people who understand a problem deeply can spot when an AI-generated answer misses the point. The people who can write clearly can guide AI more effectively.

That’s what I’m watching most closely.

AI is changing how we work, but I don’t believe it changes the need to think carefully. If anything, it raises the value of that skill.

The blank page still matters.

Copilot may help us fill it faster, but the real value comes from understanding what deserves to be written there in the first place.

The Skill That Matters Most in the Age of Copilot: Asking Better Questions

The people getting the most value from Microsoft 365 Copilot today aren’t necessarily the most technical. They’re not the fastest typists, the best PowerShell writers or the ones with the biggest AI budget.

They’re the people who ask better questions.

That might sound obvious, but I think it’s one of the biggest mindset shifts happening right now. For years, many of us have been trained to search for information using a few keywords. Type something into Google, scan the results, click a link, repeat. The skill was finding information.

With Copilot, the skill is becoming framing the problem.

I see this regularly when working with SMBs and MSPs. Two people sit in front of the same Microsoft 365 Copilot environment. One gets a generic answer that doesn’t really help. The other gets a detailed summary, actionable recommendations and a useful first draft. The difference is rarely the technology.

It’s the question.

Most People Stop Too Early

A common pattern I see is someone opening Copilot in Teams or Outlook and typing a very short request:

“Summarise this.”

“Write an email.”

“Analyse this document.”

Copilot will do something, but not necessarily something valuable.

The better approach is to provide context. Why do you need the information? Who is the audience? What decision are you trying to make? What concerns do you already have?

The quality of the answer often improves dramatically when the question becomes more specific.

Think about a conversation with a trusted employee. If you walked into their office and said, “Help me with sales”, you’d get a confused look. If you said, “Review the last quarter of sales reports and identify the three biggest opportunities in our existing customer base”, you’d be much closer to getting a useful result.

Copilot works in a similar way.

Copilot Rewards Curiosity

One of the biggest mistakes I see is treating Copilot like a command-line interface.

People issue instructions instead of having a conversation.

The real value appears when you start exploring.

For example, after Copilot produces a meeting summary in Teams, don’t stop there. Ask what wasn’t discussed. Ask what risks were mentioned only briefly. Ask which actions have no assigned owner. Ask which topics are likely to create issues next month.

Each follow-up question improves your understanding.

What fascinates me is that the process starts to resemble working with a highly capable colleague. The first answer is rarely the destination. It’s often the starting point.

The people who get the best outcomes are usually the people who are naturally curious. They’re willing to ask one more question.

Then another.

This Changes How We Learn

I think we’re moving into a world where knowing everything becomes less important than knowing how to investigate effectively.

In the past, expertise often meant storing large amounts of information in your head. Today, much of that information can be surfaced instantly from SharePoint, OneDrive, Teams conversations and Outlook messages through Copilot.

The challenge becomes directing that capability effectively.

When I use Copilot in Word to help draft content or in Outlook to analyse a long email thread, I rarely accept the first result. I refine it. I challenge it. I ask for alternatives. I request different viewpoints.

In many cases, my role is becoming less about generating information and more about guiding the process that generates it.

That’s a different skill set altogether.

The Real Competitive Advantage

I don’t think the winners in the AI era will be the organisations with the most AI tools.

I think they’ll be the organisations that teach their people how to think clearly, define problems accurately and ask better questions.

Microsoft 365 Copilot is already showing us this reality. The technology is becoming easier to access every month. The differentiator isn’t the tool. It’s the person using it.

A poor question gives you a poor starting point.

A thoughtful question opens entirely different possibilities.

That’s why, when people ask me what skill they should develop to get more value from Copilot, my answer is increasingly simple:

Learn to ask better questions. Everything else gets easier from there.

Stop Feeding the Algorithm. Start Changing Minds.

I see a lot of businesses obsessing over content calendars.

They sit in meetings asking questions like: What should we post next week? Or How many videos do we need this month? Or Should we start a podcast?

In my experience, that’s usually the wrong conversation.

The better question is this:

What belief are you trying to create in the mind of your audience?

Because content by itself doesn’t do much. Anyone can produce content. AI has made that even easier. A few prompts and you’ve got articles, videos, social posts and newsletters ready to go.

The problem is that information is everywhere.

Attention isn’t.

Most Content Is Just Noise

An average marketer focuses on output.

More emails. More posts. More videos. More activity.

The assumption is that if you produce enough material, eventually customers will notice.

Sometimes they do. Most of the time they don’t.

What I’ve noticed when working with businesses is that content often becomes the goal rather than the vehicle. Teams celebrate publishing something without asking whether it changed how their audience thinks.

That’s a dangerous trap.

You can publish every day and still be invisible.

You can have thousands of followers and still struggle to generate sales.

Volume doesn’t create demand.

Changing beliefs does.

Leads Are Better. Beliefs Are Better Again.

A good marketer understands this and focuses on generating leads.

They want enquiries. Booked meetings. Website conversions. People entering the sales process.

That’s certainly an improvement.

But even then, many organisations are trying to persuade people who haven’t yet reached the point of wanting what they’re selling.

They are still fighting an uphill battle.

The strongest businesses I’ve observed create demand before the sales conversation even begins.

When a prospect turns up already convinced that a problem exists, already understands the cost of ignoring it, and already believes your approach is the right one, the sales process becomes dramatically easier.

The decision was largely made long before the sales call.

The content wasn’t there to inform.

It was there to shape perspective.

This Is Where Copilot Becomes Interesting

One of the reasons I’m paying close attention to Microsoft 365 Copilot isn’t because it helps create content faster.

Almost everyone talks about content production.

I think that’s the least interesting part.

The real opportunity is understanding what your audience already believes and what needs to change.

Imagine reviewing customer meeting notes stored in Teams, analysing recurring themes from emails in Outlook, and asking Copilot to identify the assumptions that repeatedly appear across conversations.

You might discover that clients believe AI is too expensive. Or too risky. Or only suitable for large enterprises.

Suddenly your next article isn’t chosen because somebody needed something to post on Tuesday.

It’s chosen because there’s a specific belief that needs challenging.

That’s a much more strategic use of both content and AI.

The Businesses Winning Attention Know Exactly What They’re Installing

The organisations that consistently attract customers don’t just share information.

They repeat ideas.

They reinforce viewpoints.

They help their audience see the world differently.

Over time those ideas compound.

Eventually prospects begin repeating those beliefs back to their colleagues, managers and peers.

That’s when demand starts appearing.

Not because your content reached more people.

Because it changed the way people think.

That’s a very different outcome.

The next time you’re planning content, I’d suggest stepping back from the calendar, the platform and the format.

Don’t start with the post.

Start with the belief.

Because when you know what belief you’re trying to create, the content becomes obvious.

And when enough people share that belief, demand tends to take care of itself.

AI Is Helping People Solve the Wrong Problem Faster

There’s something I’m seeing more and more as AI becomes embedded into everyday business processes.

People are getting dramatically faster.

The problem is that many of them are getting faster at solving the wrong problem.

Microsoft 365 Copilot can help create documents in minutes, analyse data in seconds and summarise information almost instantly. That’s impressive. But if the question you start with is flawed, all you’ve done is accelerate yourself in the wrong direction.

In my experience working with MSPs and SMBs, the biggest challenge isn’t usually a lack of answers.

It’s a lack of asking the right questions.

The Bottleneck Is Rarely Where You Think It Is

As service providers, we’re often called in to solve what a client thinks is the issue.

Their help desk is overloaded.

Sales aren’t growing.

Projects are delayed.

Customer enquiries aren’t being answered quickly enough.

The temptation is to jump straight into fixing the symptom.

Now AI makes that even easier.

A client says they need faster reporting. Copilot helps generate reports in minutes.

A manager says they need more customer communications. AI drafts them in seconds.

A business owner says they need more leads. AI builds marketing content all day long.

But what if none of those things are actually the constraint?

I’ve seen businesses spend months optimising processes that had almost no impact on growth because they were focused on the visible problem rather than the real one.

AI simply helped them arrive at the wrong destination faster.

Better Questions Create Better Outcomes

One of the most valuable uses of Microsoft 365 Copilot isn’t generating content.

It’s helping people think.

The organisations getting the most value aren’t treating Copilot as a faster keyboard. They’re using it as a tool to challenge assumptions and examine the business from different angles.

When someone tells me they have a problem, I often start with three simple questions:

Where are you now?
Where do you want to go?
What do you believe is stopping you?

Those questions sound basic, but they’re surprisingly revealing.

Recently I spoke with a business that was convinced slow customer response times were hurting growth. Their team wanted AI to automate messaging and increase communication volume.

After stepping back and examining the process, the real constraint wasn’t messages.

It was capacity.

There simply weren’t enough qualified people available to deliver the service being sold.

Automating communications would have made customers happier for a short period while simultaneously increasing pressure on an already constrained delivery team.

The wrong problem.

Another business owner believed lead generation was the issue. They wanted AI-generated marketing campaigns, social media content and automated outreach.

The reality?

Their existing funnel already produced enough opportunities.

The bottleneck was conversion.

Improving close rates by a small percentage would have generated far greater revenue than doubling marketing activity.

Again, the wrong problem.

Copilot Should Challenge Thinking, Not Replace It

This is where I believe many MSPs have an opportunity.

Too many AI conversations revolve around features, prompts and automation.

Those things matter, but they’re not the real value.

The real value comes from helping clients identify what actually constrains growth, profitability or productivity.

Copilot gives us the ability to analyse information faster than ever before. It can surface patterns, summarise discussions and identify trends that would otherwise take hours to uncover.

But someone still needs to ask the right question.

As MSPs, that’s where our experience becomes critical.

The businesses that will benefit most from AI won’t necessarily be the ones with the most advanced technology. They’ll be the ones with the clearest understanding of their business challenges.

AI can provide answers almost instantly.

What it can’t do is determine whether you’re asking the right question in the first place.

The Real Opportunity

Before using Copilot to generate another report, automate another workflow or create another piece of content, take a step back.

Define where you are.

Define where you want to go.

Identify what you genuinely believe is standing in the way.

Then challenge that assumption.

Because in many cases, the biggest business breakthrough doesn’t come from getting a better answer.

It comes from discovering you’ve been asking the wrong question all along.

And that’s a mistake AI can help you make much, much faster.

More Consumption Doesn’t Mean More Progress

I had a conversation last week that’s stuck with me. Someone was telling me, with real pride, how productive their week had been. They’d cleared every email, sat through nine meetings, skimmed four reports and watched a couple of webinars. By Friday they were wrung out. So I asked the obvious question: what actually changed because of all that? There was a long pause. The honest answer was, not much. They’d taken in a mountain of information and moved almost nothing forward.

That gap — between taking things in and actually getting somewhere — is something I keep coming back to.

Busy is not the same as moving

It has never been easier to consume. Open Outlook and Copilot will boil a forty-message thread down to a few lines before you’ve read the subject. Miss a Teams meeting and the recap is sitting there waiting for you. Ask Copilot in Word to turn a thirty-page document into five points and it’s done in seconds. All of it genuinely useful. But here’s the trap I keep watching people fall into: they mistake the speed of consuming for the act of progressing.

Reading a summary feels like work. It isn’t. It’s the warm-up to work. That summary only earns its place if it leads to a decision, a reply, a change of plan — something that wasn’t true before you read it.

The point of a summary is what you do next

When I use Copilot to catch up on a noisy channel, the value was never the recap itself. It’s the one thing the recap surfaces — the client still waiting on an answer, the date that quietly moved, the call only I can make. If I read the summary and slide straight into the next one, I’ve consumed, but I haven’t progressed a single step.

So I’ve started asking Copilot a different kind of question. Not “summarise this thread,” but “what here needs a decision from me?” In Outlook, instead of “what’s in my inbox,” I’ll ask what’s waiting on a reply from me specifically. It’s a small change in wording, but it shifts Copilot from a faster way to take things in into a prompt to actually act.

More input, fewer outcomes

The real risk in all this capability is sheer volume. Because we can now process more, we start to feel we should. More reports, more recaps, more dashboards, more catch-ups. But a business doesn’t run on how much its people have read this week. It runs on what they decided, finished and delivered.

I’d rather end a week having genuinely moved three things forward than having consumed everything that landed on my desk. Copilot is brilliant at clearing the path — pulling the signal out of a crowded SharePoint site, drafting a first version in Word, getting the numbers into shape in Excel. But once the path is clear, walking down it is still on me.

The tools will keep getting faster at feeding us information, and that’s not the part I’m watching. I’m watching whether all that speed actually changes what we do — or whether we just get very good at staying busy. Consumption is effortless now. Progress still needs a decision, and that’s the one thing no tool will make for you.

Comparing LLMs in Copilot services–Round 1 – Chat

Inspired by the recent soccer world cup, I have decided to create a Copilot LLM output comparison challenge.

The plan is to use the same prompt with all examples of different services in Copilot and then different models available in each service. After that, the idea is them to compare the winner of each round to determine the overall winner and to continue to do this on a regular basis as new models and services are added over time.

Thus, the methodology is to use the same complex prompt to generate the result from the model (a report) and then use a standard prompt to evaluate all the results to determine a winner. The easiest comparison method is to use Copilot in SharePoint but the aim will also to be to compare using other models as well.

So, for round 1 I’m going to compare all the models available in Copilot chat. Comparison generated by Copilot for SharePoint.

Rather than try and fit the reports here I will upload them to my Github repository here in markdown format:

https://github.com/directorcia/general/tree/master/Copilot/Comparisons

Comparative Assessment – Live Writer Paste

This first report is now directly available at:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Chat.md

The results where (out of 10):

1. Opus – 9.51

2. Sonnet – 9.45

3. GPT 5.6 Thinking – 8.76

4. GPT 5.5 Quick – 7.69

5. Auto – 6.93

So, the winner for Round 1 – Copilot Chat = Opus.

Onto Round 2

When Everyone Has the Expert in the Room

For most of my working life, expertise has been a bottleneck. The person who really understood how to structure a sound investigation, write a watertight scope of work, run a proper risk assessment, or close out a project without loose ends — that person was rare, busy, and usually expensive. Their knowledge lived in their head, in a handful of dog-eared templates, or in a methodology a consulting firm guarded like a recipe. If you wanted the good version of something, you waited for the expert, or you paid for them, or you simply went without and hoped your rough effort was good enough.

So here is the question I keep turning over: what happens to a business when those world-class practices stop being scarce? When the best way to do a thing is no longer trapped in one person’s experience, but packaged as a skill that anyone can call on, inside the tools they already use every day?

We are closer to that than most people realise.

The recipe leaves the head of the chef

Think about what a “best practice” actually is. It’s a sequence of good decisions someone learned the hard way, refined over years, and turned into a repeatable approach. The hard part was never the steps themselves — it was knowing them, and knowing when to apply them.

Copilot changes who has access to that sequence. With agents and custom skills, an organisation can take its genuinely good way of doing something — the proposal process the best salesperson uses, the onboarding checklist that actually works, the way the sharpest analyst pressure-tests a forecast — and make it available to everyone. Not as a PDF nobody reads, but as something you ask for in the flow of work. You’re in Word drafting a statement of work, and the expert method is right there. You’re in Excel staring at a model, and Copilot applies the same scrutiny your best analyst would, in the same Excel you already had open.

The recipe leaves the head of the chef. And once it does, the people who were never going to become experts can still produce expert-grade work.

The gap that quietly disappears

I find this genuinely interesting, because of what it does to the gap between the few and the many.

In most businesses there’s an enormous distance between your top performer and your average one. Not because the average person isn’t capable, but because they never had the top performer’s accumulated judgement. When that judgement becomes a skill anyone can invoke — when the new hire in their second week can ask Copilot to apply the company’s proven method and get most of the way there — that gap narrows fast.

I’ve watched a junior team member produce a client response that, two years ago, would have needed three rounds of review from someone senior. The senior person still added value. But the starting point was already good, because the method was baked in rather than carried around in someone’s memory. That’s a different shape of organisation. The floor rises. The distance between your best and your rest gets smaller.

And that should make business leaders pause, because so much of how we structure teams, pay people, and value experience assumes that gap stays wide.

When the answer is cheap, the action becomes everything

Here’s the part I think is easy to miss. If the best way to do something is available to everyone — including your competitors — then knowing the best practice stops being an advantage. Everyone has it. It becomes table stakes.

So what’s left? Judgement about which problem to point it at. Taste about what “good” actually means for your customers. And the plain willingness to act. When the expert method is in the room, the question shifts from who knows how to who actually does something with it.

I’ve seen two businesses with the same tools and the same access. One treats Copilot as a curiosity someone in IT is “looking into.” The other has quietly rebuilt how its people work — its real methods captured as agents in Teams, surfaced where decisions get made, used a hundred times a day. Same starting line. Wildly different outcomes. The difference wasn’t the technology. It was the decision to act on it.

That’s the uncomfortable, liberating truth of broadly available expertise. It doesn’t reward the people who hoarded knowledge. It rewards the ones who move.

What I’m watching

I don’t think this makes expertise worthless — I think it relocates it. The value moves from holding the knowledge to deciding what to do with it, and to having the judgement to know when the expert method is wrong for this particular case. Those things are harder to automate, and they’re suddenly worth far more.

What I’m watching for is which organisations notice the shift early. The ones who capture their best practices as Copilot skills and put them in everyone’s hands aren’t just becoming more efficient. They’re flattening a hierarchy that has shaped business for a very long time. The expert is no longer a bottleneck. The expert is in the room — for everyone, all the time.

The only question left is what you do now that they are.