Comparing LLMs in Copilot services–Round 2 – Researcher

Round 1 – https://blog.ciaops.com/2026/07/30/comparing-llms-in-copilot-services-round-1-chat/

Round 1 Winner – Copilot Chat = Opus

This is round 2 of my Copilot LLM battle, with the aim to determine the ‘best’ result from all the AI services and all the available LLMs used with Copilot in Microsoft 365.

The process involves taking a standard prompt and running it against all options. This prompt creates a multi page document requiring deep research and is quite involved. The results are then compared against each other using SharePoint Copilot and Gemini. Conclusions are then drawn.

This round is all the possibilities with Researcher. As before, rathet than duplicate the reports here I have uploaded them to my Githuv repository in markdown format:

SharePoint Copilot assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher.md

Gemini assessment:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Researcher-Gemini-eval.md

Results (out of 10):

Critique – 9.30

Claude – 8.88

Council – 7.83

Auto – 7.63

Notes – SharePoint and Gemini produced very different rankings. I’m sticking with SharePoint’s assessment for consistency. I will also point out that gettign Gemini to do this comparison and produce a simple markdown file with the result was pretty much impossible. It does a really poor job on this simple comparison ask as you can see from the results.

So the Round 2 winner – Copilot Researcher = Critique

Onto Round 3

Copilot said: What People Say, What They Think, and What They Actually Do

Many years ago, David Ogilvy made an observation that still rings true today:

“The trouble with market research is that people don’t think what they feel, they don’t say what they think and they don’t do what they say.”

The older I get, the more I see this playing out in technology.

If you spend any time talking to business owners, managers, or end users about artificial intelligence and Microsoft 365 Copilot, you’ll hear all sorts of opinions. Some people tell me they’re excited. Others say they’re cautious. Many claim they’re waiting to see where the technology goes before investing further.

Then I look at what they’re actually doing.

That’s where things get interesting.

Watching Behaviour Tells You More Than Listening to Opinions

Over the last year I’ve had countless conversations with organisations evaluating Microsoft 365 Copilot. The feedback is often remarkably similar.

“We’re not sure we’ll use it enough.”

“Our staff aren’t ready.”

“We don’t have enough use cases yet.”

“We want to be careful before spending money.”

All perfectly reasonable concerns.

Yet when I sit down with those same organisations and look at how people work, I see something different. Staff are already using AI tools on their phones, in their browsers, and in personal accounts. They’re pasting email drafts into consumer AI services. They’re asking AI to summarise documents. They’re using it to help write proposals, prepare presentations, and analyse spreadsheets.

In other words, their behaviour is already telling me they’ve accepted AI. They just haven’t consciously recognised it yet.

The actions are often ahead of the stated position.

The Same Thing Happens Inside Microsoft 365

One of the most revealing exercises is simply watching how people use the tools they already have.

Someone may tell me they don’t have enough information to write a client proposal faster. Then I ask Microsoft 365 Copilot in Word to create a first draft from existing documents stored in SharePoint.

Someone insists meetings take as long as they’ve always taken. Then Copilot in Teams produces a meeting recap, extracts actions, and highlights decisions in a few seconds.

Someone believes they’re spending hours processing email because that’s the nature of modern work. Then Copilot in Outlook starts drafting replies and summarising lengthy email chains.

The surprising part isn’t that the technology works.

The surprising part is how quickly people adapt once they see it working in their own environment.

What they thought they would do is often very different from what they actually end up doing.

Technology Adoption Has Always Been Emotional

I think we sometimes treat technology decisions as purely logical exercises.

They’re not.

People rarely make decisions based solely on facts, specifications, or business cases. Emotion plays a huge role. Fear of change. Fear of looking foolish. Fear of making the wrong investment. Sometimes even fear of becoming less valuable.

That’s why surveys and focus groups often struggle to predict outcomes accurately. People answer based on how they imagine they’ll behave in the future.

Reality arrives later.

The same pattern appears with Microsoft 365 Copilot. Many organisations start cautiously, perhaps with a small pilot group. The initial conversations focus on risks, limitations, and concerns.

A few weeks later the discussion changes.

Users begin sharing prompts with each other. Teams start discovering new ways to use Copilot in meetings. Managers realise they can get through information faster. Gradually the conversation moves from whether they’ll use AI to where they should use it next.

The behaviour changes first. The mindset follows.

Pay Attention to What Happens, Not Just What Is Said

When I’m helping organisations assess readiness for AI, I place far more value on observed behaviour than survey responses.

The question isn’t whether someone says AI will help them.

The question is whether they reach for it when they’re busy.

Do they use Copilot to prepare for a meeting?

Do they ask Copilot in Outlook to help clear their inbox?

Do they use Copilot in Excel when they need answers from a spreadsheet?

Those actions reveal far more than any questionnaire.

David Ogilvy understood something fundamental about human nature. People are often poor predictors of their future behaviour.

As Microsoft 365 Copilot continues to become part of everyday work, I think we’ll see the same pattern repeat itself. The organisations that succeed won’t necessarily be the ones that talk most enthusiastically about AI.

They’ll be the ones that quietly incorporate it into their daily habits.

Because when it comes to technology adoption, what people do has always mattered more than what they say.

The Colour of Your Calendar Tells the Truth

I’ve become a big fan of colour-coded calendars over the years. Not because they look neat. Not because they make me feel organised. And certainly not because I enjoy spending time categorising appointments.

I like them because they tell the truth.

The reality is that most of us have a very different perception of how we spend our time compared to what actually happens. Ask someone where their week went and you’ll usually get an answer based on memory, emotion or whatever felt busiest. Open a colour-coded calendar and suddenly the conversation changes.

The evidence is right there in front of you.

And sometimes it’s not what you expected to see.

Busy Doesn’t Mean Progress

One of the first things a colour-coded calendar exposes is whether you’re actually moving forward or simply staying occupied.

I’ve seen weeks where the calendar looked completely full. Every day packed from morning until late afternoon. At first glance it felt productive.

Then I looked at the colours.

Almost every block was internal meetings.

Status updates. Catch-ups. Reviews. Discussions about discussions.

There wasn’t much time allocated to activities that improved the business, created something new, built a process, documented knowledge or generated future value.

That’s a confronting realisation.

Many business owners and IT professionals spend their entire week responding to what’s already happening instead of creating what happens next. A simple colour system makes that visible very quickly.

The calendar doesn’t care how important the meeting felt at the time. It simply shows where the hours went.

Seeing the Cost of Context Switching

Another thing colour coding reveals is how fragmented your day has become.

You start with a customer conversation. Then jump into Outlook to clear email. Then sit through a Teams meeting. Then back to customer work. Then another meeting. Then more email.

By lunchtime you’ve changed focus so many times that it feels like you’ve been running all morning but haven’t actually completed anything meaningful.

I see this regularly with MSPs and small business teams.

The work itself isn’t necessarily the problem. It’s the constant switching between different types of work.

When each category has its own colour, the pattern becomes obvious. Instead of solid blocks of concentrated effort, the calendar looks like confetti scattered across the day.

Once you can see it, it’s much easier to fix.

You begin grouping similar activities together. Customer calls sit together. Administration sits together. Internal meetings get pushed into dedicated windows.

The day starts working for you instead of against you.

Your Best Hours Matter Most

The biggest lesson I keep coming back to is how valuable the first few hours of the day really are.

That’s when my thinking is usually at its sharpest.

It’s when I want to write, plan, create content, build presentations or work with Microsoft 365 Copilot to develop ideas and explore new approaches. Whether I’m using Copilot in Word to draft an article or reviewing strategy notes from a Teams meeting, those activities benefit from clear thinking and uninterrupted attention.

Yet many calendars tell a different story.

The most productive hours are often handed over to routine meetings, administrative work or conversations that could easily happen later.

When I review a colour-coded calendar, that issue jumps out immediately.

The colours show whether those high-energy hours are being invested or simply consumed.

That’s an important distinction.

The Colours Never Argue

What I like most about colour coding is its honesty.

It removes assumptions.

It removes excuses.

It removes that comfortable story we tell ourselves about where our time goes.

Open your calendar and look at the next seven days.

Which colour dominates?

Customer work? Internal meetings? Administration? Personal development? Strategic planning?

Whatever colour appears most often is receiving the largest investment of your time.

And over the long term, that’s probably where your business, career and attention are heading as well.

The colours won’t tell you what to do.

But they will show you the truth.

And that’s usually the first step towards making better decisions.

CIAOPS Need to Know Microsoft 365 Webinar – August

laptop-eyes-technology-computer_thumb

Now in our tenth year!

Join me for the free monthly CIAOPS Need to Know webinar. Along with all the Microsoft Cloud news we’ll be taking a look at Beyond Copilot Chat

Shortly after registering you should receive an automated email from Microsoft Teams confirming your registration, including all the event details as well as a calendar invite.

You can register for the regular monthly webinar here:

August Registrations

(If you are having issues with the above link copy and paste – https://bit.ly/n2k2608)

The details are:

CIAOPS Need to Know Webinar – July 2026
Friday 31st of July 2026
11.00am – 12.00am Sydney Time

All sessions are recorded and posted to the CIAOPS Youtube channel.

Also feel free at any stage to email me directly via director@ciaops.com with your webinar topic suggestions.

I’d also appreciate you sharing information about this webinar with anyone you feel may benefit from the session and I look forward to seeing you there.

Stop Waiting for Perfect Conditions

I hear versions of the same story all the time.

Someone wants to learn Microsoft 365 Copilot. Start a side project. Improve their business processes. Clean up their Microsoft 365 tenant. Build a new service offering for their clients.

But not yet.

They’re waiting for a quieter week. Waiting for a project to finish. Waiting for school holidays to end. Waiting for the inbox to settle down.

The problem is that week never arrives.

I’ve been in technology long enough to know that there is always another project. Another client request. Another issue needing attention. If you’re running an MSP or managing an SMB, the workload doesn’t magically disappear one day and leave you with hours of spare time.

That’s why I’ve come to believe that constraints aren’t the thing stopping you.

More often than not, they’re the thing that should be driving you forward.

The Reality of Limited Time

A lot of people treat constraints like roadblocks.

“I don’t have enough time.”

“I don’t have enough money.”

“I don’t have enough people.”

Those things may all be true. But they’re also true for almost everyone else.

The difference is how you respond to them.

When I work with organisations deploying Microsoft 365 Copilot, the most successful ones rarely have unlimited resources. They’re not sitting around with spare staff and empty calendars. They’re already busy.

What they do have is a willingness to work within the constraint rather than waiting for it to disappear.

Instead of trying to train every employee at once, they start with five people.

Instead of redesigning every process in the business, they pick one recurring task in Outlook or Teams and improve it.

Instead of creating the perfect governance framework, they begin with the basics and build from there.

The constraint forces focus.

And focus is often worth more than freedom.

Small Boundaries Create Better Decisions

One thing I’ve noticed over the years is that too many options can be just as dangerous as too few.

Give someone unlimited time and unlimited resources and they often spend months planning.

Give them one hour a week and suddenly they become remarkably efficient.

I’ve seen MSPs create entirely new AI service offerings because they only had a few hours available each week. The limited time meant they ignored all the distractions and concentrated on the activities that actually mattered.

The same applies to personal growth.

If you’ve only got thirty minutes available on a Friday afternoon, use it.

Open Copilot in Word and draft an article.

Use Copilot in Teams to summarise a meeting you missed.

Explore a new automation in Power Automate.

Read one Microsoft Learn article.

Thirty minutes won’t change your business overnight. But thirty minutes every week for a year absolutely will.

The point isn’t the amount of time.

It’s the consistency.

Stop Negotiating With the Future

What worries me most about the “I’ll start when things settle down” mindset is that it quietly trains us to delay action.

We convince ourselves we’re being sensible.

We’re really just postponing the uncomfortable part.

Learning something new feels hard.

Changing a process feels risky.

Starting a project creates uncertainty.

So we strike a deal with ourselves and push it into the future.

The future, of course, never objects.

It just keeps moving.

I’ve found that progress usually begins when we stop asking whether conditions are ideal and start asking what can be done with the resources available today.

That’s true whether you’re adopting Copilot, building an MSP business, improving your security posture, or simply trying to learn a new skill.

The Constraint Is the Opportunity

When I look back at the projects that made the biggest difference in my career, very few started under perfect conditions.

Most began when time was short, money was limited, or other priorities were competing for attention.

The limitation wasn’t the problem.

It was the catalyst.

So if you’re waiting for a quieter month, a bigger budget, or fewer commitments, I’d encourage you to rethink the situation.

The constraint you’re looking at today might not be the obstacle.

It might be the thing that finally forces you to focus on what matters most.

That’s usually where the real progress starts.

Writing Is Still Thinking (Even in the Age of Copilot)

A few weeks ago I caught myself doing something I suspect a lot of people are doing right now.

I had an idea. Instead of opening a blank document, I opened Microsoft 365 Copilot and started asking questions.

Within minutes I had summaries, suggestions, outlines, and options. It was impressive. It was also a little unsettling.

The reason was simple. I had an answer before I had really worked out what I thought about the question.

That’s something I’ve been reflecting on more as AI becomes a bigger part of our daily work. We often talk about AI helping us write, but I think the more important question is whether we’re still using writing to think.

The Value of a Blank Page

For most of my career, writing has been one of the ways I clarified my thinking.

Whether it was a client proposal, a blog post, meeting notes, or an internal strategy document, the process worked because it was slow enough to force decisions. You couldn’t hide gaps in your logic. When you sat staring at an empty page, every weak assumption became obvious.

That’s still true today.

When I open a Word or OneNote document and start typing, I often discover that the idea I thought was clear in my head isn’t quite as solid as I imagined. The act of putting words on a page exposes the fuzzy edges.

The danger with AI is that it can smooth over those edges too quickly. Copilot can generate something that sounds complete before you’ve done the hard work of deciding what you actually believe.

The output may be polished. Your thinking may not be.

Copilot Works Best as a Thinking Partner

This isn’t an argument against Microsoft 365 Copilot. Far from it.

What I’m seeing in organisations is that the most successful users don’t treat Copilot as a replacement for thinking. They use it as a way to challenge and refine their thinking.

A practical example might be drafting a proposal in Word. I often start with rough notes of my own. They are messy and incomplete. Only after I’ve worked through the problem do I ask Copilot to review the draft, identify gaps, suggest alternative approaches, or summarise the key points.

The sequence matters.

Thinking first. AI second.

If you reverse the order, there’s a risk that you’re evaluating someone else’s answer rather than exploring your own understanding.

That’s a subtle difference, but it’s an important one.

Faster Answers Aren’t Always Better Answers

Many of us spend our days moving between Outlook, Teams meetings, client conversations, and project work. The attraction of having instant answers everywhere is obvious.

The challenge is that genuine understanding usually takes longer than information retrieval.

Copilot can tell you what happened in a Teams meeting you missed. It can summarise a lengthy email thread in Outlook. It can pull key points from a SharePoint document in seconds.

That’s incredibly useful.

What it can’t do is decide what those things mean for your business, your clients, or your next decision.

That responsibility still sits with you.

I think that’s why many people who try AI for the first time are both impressed and disappointed. They’re impressed by the speed. They’re disappointed when speed alone doesn’t solve the problem they were really facing.

Knowledge is easy to access. Judgement is still hard work.

The Skill Worth Protecting

The organisations getting the most value from AI aren’t abandoning traditional skills. They’re strengthening them.

Clear writing. Critical thinking. Good questioning. Sound decision-making.

Those capabilities become more important as AI becomes more capable, not less.

The people who ask the best questions get better results from Copilot. The people who understand a problem deeply can spot when an AI-generated answer misses the point. The people who can write clearly can guide AI more effectively.

That’s what I’m watching most closely.

AI is changing how we work, but I don’t believe it changes the need to think carefully. If anything, it raises the value of that skill.

The blank page still matters.

Copilot may help us fill it faster, but the real value comes from understanding what deserves to be written there in the first place.

The Login Screen Looks Simple. The Decision Behind It Is Not.

I worry that sign-in security is still treated like a mystery box in too many Microsoft 365 environments.

A user enters a password. Something happens behind the scenes. Maybe MFA appears. Maybe Conditional Access blocks the request. Maybe the sign-in works because the policy did not apply as expected. Then, when something goes wrong, the first question is usually, “Why did Microsoft let that happen?”

That is the wrong question.

The better question is: did we understand the path that sign-in actually took?

That is why I like simple visual tools such as the Microsoft 365 login simulator. Not because they replace proper testing in Entra. They do not. I like them because they make the invisible visible. They let people walk through authentication logic without starting inside a dense admin portal.

Authentication is where theory meets reality

Most businesses think they have secure login because they have MFA turned on. That is a start, but it is not the full story.

In the real world, sign-in decisions are messy. A user might be on a managed device, a personal laptop, or public Wi-Fi. They might be coming from a known location, a strange location, or through a service that does not behave like a normal browser session.

This is where Microsoft Entra ID, Conditional Access, MFA, device compliance and sign-in risk start to matter. They are not isolated controls. They are a decision chain.

For an MSP, that chain needs to be explainable.

I have sat in enough client conversations to know that saying “we enabled MFA” does not always land. A business owner wants to know what happens when an account is attacked. A help desk person wants to know whether the issue is identity, device, location, policy, licensing or behaviour.

A simulator gives you a conversation starter. It turns the abstract into something you can point at.

Copilot still depends on identity hygiene

Microsoft 365 Copilot does not remove the need for clean identity controls. If anything, it raises the stakes. When someone asks Copilot in Teams to summarise a channel, or uses Copilot in Word to draft from files in SharePoint, the experience depends on the access that user already has.

That means sign-in is not just a security event. It is the front door to organisational knowledge.

If a compromised account gets through that front door, the issue is no longer just email. It may include documents, chats, meetings and shared files. For an SMB, that is a business risk, not a technical footnote.

So when I look at a login simulation, I am thinking about what the user can reach after access is granted.

Use the simulator to teach judgement

The best use of a tool like this is not to frighten people. It is to build judgement.

Run through a few scenarios with a client or your own team. What should happen for a global administrator? What should happen for a user on an unmanaged device? What should happen for a login from a location the organisation never normally uses? Where does the policy catch the risk?

Then compare that ideal path with what your tenant is configured to do.

That gap is where the work is.

For MSPs, this is a better advisory conversation. You are not just selling another security setting. You are helping the client understand how identity, access and data exposure connect. That moves the discussion away from checkbox compliance and towards resilience.

My view is simple. If you cannot explain the sign-in path, you probably do not control it as well as you think.

The login screen looks simple. The decision behind it is anything but.

The simulator is here – https://directorcia.github.io/Office365/m365-login-sim.html

and the documentation is here – https://github.com/directorcia/Office365/wiki/M365-Sign%E2%80%90In-and-Conditional-Access-Flow-Simulator

The Skill That Matters Most in the Age of Copilot: Asking Better Questions

The people getting the most value from Microsoft 365 Copilot today aren’t necessarily the most technical. They’re not the fastest typists, the best PowerShell writers or the ones with the biggest AI budget.

They’re the people who ask better questions.

That might sound obvious, but I think it’s one of the biggest mindset shifts happening right now. For years, many of us have been trained to search for information using a few keywords. Type something into Google, scan the results, click a link, repeat. The skill was finding information.

With Copilot, the skill is becoming framing the problem.

I see this regularly when working with SMBs and MSPs. Two people sit in front of the same Microsoft 365 Copilot environment. One gets a generic answer that doesn’t really help. The other gets a detailed summary, actionable recommendations and a useful first draft. The difference is rarely the technology.

It’s the question.

Most People Stop Too Early

A common pattern I see is someone opening Copilot in Teams or Outlook and typing a very short request:

“Summarise this.”

“Write an email.”

“Analyse this document.”

Copilot will do something, but not necessarily something valuable.

The better approach is to provide context. Why do you need the information? Who is the audience? What decision are you trying to make? What concerns do you already have?

The quality of the answer often improves dramatically when the question becomes more specific.

Think about a conversation with a trusted employee. If you walked into their office and said, “Help me with sales”, you’d get a confused look. If you said, “Review the last quarter of sales reports and identify the three biggest opportunities in our existing customer base”, you’d be much closer to getting a useful result.

Copilot works in a similar way.

Copilot Rewards Curiosity

One of the biggest mistakes I see is treating Copilot like a command-line interface.

People issue instructions instead of having a conversation.

The real value appears when you start exploring.

For example, after Copilot produces a meeting summary in Teams, don’t stop there. Ask what wasn’t discussed. Ask what risks were mentioned only briefly. Ask which actions have no assigned owner. Ask which topics are likely to create issues next month.

Each follow-up question improves your understanding.

What fascinates me is that the process starts to resemble working with a highly capable colleague. The first answer is rarely the destination. It’s often the starting point.

The people who get the best outcomes are usually the people who are naturally curious. They’re willing to ask one more question.

Then another.

This Changes How We Learn

I think we’re moving into a world where knowing everything becomes less important than knowing how to investigate effectively.

In the past, expertise often meant storing large amounts of information in your head. Today, much of that information can be surfaced instantly from SharePoint, OneDrive, Teams conversations and Outlook messages through Copilot.

The challenge becomes directing that capability effectively.

When I use Copilot in Word to help draft content or in Outlook to analyse a long email thread, I rarely accept the first result. I refine it. I challenge it. I ask for alternatives. I request different viewpoints.

In many cases, my role is becoming less about generating information and more about guiding the process that generates it.

That’s a different skill set altogether.

The Real Competitive Advantage

I don’t think the winners in the AI era will be the organisations with the most AI tools.

I think they’ll be the organisations that teach their people how to think clearly, define problems accurately and ask better questions.

Microsoft 365 Copilot is already showing us this reality. The technology is becoming easier to access every month. The differentiator isn’t the tool. It’s the person using it.

A poor question gives you a poor starting point.

A thoughtful question opens entirely different possibilities.

That’s why, when people ask me what skill they should develop to get more value from Copilot, my answer is increasingly simple:

Learn to ask better questions. Everything else gets easier from there.