Comparing LLMs in Copilot services–Round 1 – Chat

image

Inspired by the recent soccer world cup, I have decided to create a Copilot LLM output comparison challenge.

The plan is to use the same prompt with all examples of different services in Copilot and then different models available in each service. After that, the idea is them to compare the winner of each round to determine the overall winner and to continue to do this on a regular basis as new models and services are added over time.

Thus, the methodology is to use the same complex prompt to generate the result from the model (a report) and then use a standard prompt to evaluate all the results to determine a winner. The easiest comparison method is to use Copilot in SharePoint but the aim will also to be to compare using other models as well. 

So, for round 1 I’m going to compare all the models available in Copilot chat. Comparison generated by Copilot for SharePoint.

Rather than try and fit the reports here I will upload them to my Github repository here in markdown format:

https://github.com/directorcia/general/tree/master/Copilot/Comparisons

Comparative Assessment – Live Writer Paste

This first report is now directly available at:

https://github.com/directorcia/general/blob/master/Copilot/Comparisons/20260630-Chat.md

The results where (out of 10):

1. Opus – 9.51

2. Sonnet – 9.45

3. GPT 5.6 Thinking – 8.76

4. GPT 5.5 Quick – 7.69

5. Auto – 6.93

So, the winner for Round 1 – Copilot Chat = Opus.

Onto Round 2


The Recurring Problem: A Managed Services Story–Chapter 5

Previouslyhttps://blog.ciaops.com/2026/07/29/the-recurring-problem-a-managed-services-story-chapter-4/

image

The idea of selling Bridgepoint, or at least taking on a growth investor, had floated around Dave’s head for a couple of years, mostly as a retirement fantasy he indulged on long drives. In March, half out of curiosity and half because losing Lakeside had rattled him more than he wanted to admit, he agreed to a preliminary valuation conversation with an M&A advisory firm that specialized in IT services roll-ups. Marcus had a contact; the meeting cost nothing but an afternoon.

The advisor, a clipped, unsentimental woman named Renata Cole, walked Dave and Priya through Bridgepoint’s financials with the emotional affect of someone reading a weather report. Revenue: $9.4 million. EBITDA: roughly $1.6 million, a healthy-looking 17 percent margin. Dave had always been quietly proud of that number.

“Your EBITDA is fine,” Renata said. “Your multiple is going to be a problem.”

“Explain that to me,” Dave said.

image

“At your size, with your revenue mix, buyers in this market are going to look at three things before they look at anything else,” she said, ticking them off on her fingers. “First: what percentage of your revenue is contracted, recurring, and sticky, versus one-off project work that could evaporate next quarter. Second: how concentrated your revenue is in your largest clients. Third: whether you have any real differentiation, or whether you’re a generalist competing on relationships and price. On the first, you’re sitting around forty-six percent recurring revenue — buyers want to see north of eighty-five percent before they’ll pay a premium multiple. On the second, your largest client is nineteen percent of revenue, and buyers get nervous well before that; anything over twenty to twenty-five percent concentrated in one account is a red flag they’ll price into the offer. On the third — no offense, Dave — ‘we’ve been doing this for eight years and our clients like us’ isn’t a moat. It’s a headline that every generalist MSP could put on their own website.”

She turned her laptop around so they could see the slide. A business with Bridgepoint’s EBITDA and a strong recurring-revenue base, high client diversification, and a specialized niche could reasonably expect eight to twelve times EBITDA, even higher with a security specialization. A business that looked like Bridgepoint currently looked — generalist, project-dependent, concentrated — was more realistically priced at four to six times, and buyers would likely structure a meaningful chunk of even that as an earnout contingent on retaining the client base through the transition.

image

Four to six times $1.6 million was, doing the arithmetic Dave really didn’t want to do in front of Renata, somewhere between $6.4 million and $9.6 million — for a company he’d spent eight years and, by his own private accounting, most of his thirties and half his forties building. Not a number to be ashamed of, exactly. But nowhere near what he’d let himself believe it might be worth, and nowhere near enough for the number to represent what it was supposed to represent: proof that the thing had been worth doing.

“I’m not telling you to sell,” Renata said, more gently, watching his face. “I’m telling you what the market is actually pricing, so that if you ever do want to sell — or even if you just want to run a business that isn’t one bad quarter away from a crisis — you know exactly which three levers you’d need to pull. It’s not really about me, or about a buyer. It’s about whether your business model matches the world your clients are actually living in now. Right now, it doesn’t quite.”

image

Dave didn’t sleep much that night. He kept coming back to something Aisha had said in November, dismissed at the time as youthful overconfidence: You’re spending forty percent of your week resetting passwords. That’s not judgment. That’s just volume.

For the first time, he let himself actually finish the thought he’d been avoiding for a year: Bridgepoint wasn’t losing because it was doing anything badly. It was losing because it was doing, extremely well, a version of the job that increasingly nobody needed done that way anymore.

image

When Everyone Has the Expert in the Room

image

For most of my working life, expertise has been a bottleneck. The person who really understood how to structure a sound investigation, write a watertight scope of work, run a proper risk assessment, or close out a project without loose ends — that person was rare, busy, and usually expensive. Their knowledge lived in their head, in a handful of dog-eared templates, or in a methodology a consulting firm guarded like a recipe. If you wanted the good version of something, you waited for the expert, or you paid for them, or you simply went without and hoped your rough effort was good enough.

So here is the question I keep turning over: what happens to a business when those world-class practices stop being scarce? When the best way to do a thing is no longer trapped in one person’s experience, but packaged as a skill that anyone can call on, inside the tools they already use every day?

We are closer to that than most people realise.

The recipe leaves the head of the chef

Think about what a “best practice” actually is. It’s a sequence of good decisions someone learned the hard way, refined over years, and turned into a repeatable approach. The hard part was never the steps themselves — it was knowing them, and knowing when to apply them.

Copilot changes who has access to that sequence. With agents and custom skills, an organisation can take its genuinely good way of doing something — the proposal process the best salesperson uses, the onboarding checklist that actually works, the way the sharpest analyst pressure-tests a forecast — and make it available to everyone. Not as a PDF nobody reads, but as something you ask for in the flow of work. You’re in Word drafting a statement of work, and the expert method is right there. You’re in Excel staring at a model, and Copilot applies the same scrutiny your best analyst would, in the same Excel you already had open.

The recipe leaves the head of the chef. And once it does, the people who were never going to become experts can still produce expert-grade work.

The gap that quietly disappears

I find this genuinely interesting, because of what it does to the gap between the few and the many.

In most businesses there’s an enormous distance between your top performer and your average one. Not because the average person isn’t capable, but because they never had the top performer’s accumulated judgement. When that judgement becomes a skill anyone can invoke — when the new hire in their second week can ask Copilot to apply the company’s proven method and get most of the way there — that gap narrows fast.

I’ve watched a junior team member produce a client response that, two years ago, would have needed three rounds of review from someone senior. The senior person still added value. But the starting point was already good, because the method was baked in rather than carried around in someone’s memory. That’s a different shape of organisation. The floor rises. The distance between your best and your rest gets smaller.

And that should make business leaders pause, because so much of how we structure teams, pay people, and value experience assumes that gap stays wide.

When the answer is cheap, the action becomes everything

Here’s the part I think is easy to miss. If the best way to do something is available to everyone — including your competitors — then knowing the best practice stops being an advantage. Everyone has it. It becomes table stakes.

So what’s left? Judgement about which problem to point it at. Taste about what “good” actually means for your customers. And the plain willingness to act. When the expert method is in the room, the question shifts from who knows how to who actually does something with it.

I’ve seen two businesses with the same tools and the same access. One treats Copilot as a curiosity someone in IT is “looking into.” The other has quietly rebuilt how its people work — its real methods captured as agents in Teams, surfaced where decisions get made, used a hundred times a day. Same starting line. Wildly different outcomes. The difference wasn’t the technology. It was the decision to act on it.

That’s the uncomfortable, liberating truth of broadly available expertise. It doesn’t reward the people who hoarded knowledge. It rewards the ones who move.

What I’m watching

I don’t think this makes expertise worthless — I think it relocates it. The value moves from holding the knowledge to deciding what to do with it, and to having the judgement to know when the expert method is wrong for this particular case. Those things are harder to automate, and they’re suddenly worth far more.

What I’m watching for is which organisations notice the shift early. The ones who capture their best practices as Copilot skills and put them in everyone’s hands aren’t just becoming more efficient. They’re flattening a hierarchy that has shaped business for a very long time. The expert is no longer a bottleneck. The expert is in the room — for everyone, all the time.

The only question left is what you do now that they are.

How SharePoint environments can be improved for Copilot results

image

The short answer is this:

Copilot results are only as good as the SharePoint environment underneath them. If your SharePoint is messy, overshared, full of duplicate files, stale content, and inconsistent naming, Copilot will surface messy, duplicate, stale content. If SharePoint is well-structured, governed, and maintained, Copilot becomes dramatically more useful.

From everything I’ve seen in SMB environments, improving SharePoint for Copilot usually delivers a bigger productivity gain than buying additional AI licences.

1. Fix permissions and oversharing first

This is the most important step.

Copilot doesn’t magically know what information is important. It relies on Microsoft Graph and existing permissions. If users can access content they shouldn’t, Copilot can discover and surface that content.

Common problems

  • Everyone has access to everything

  • Legacy project sites never cleaned up

  • Anonymous sharing links still active

  • “Everyone except external users” permissions

  • Former employees still own sites

What to do

  • Review site permissions

  • Remove unnecessary access

  • Review sharing links

  • Remove broad access groups

  • Identify inactive sites

  • Implement site lifecycle management

SharePoint Advanced Management and Data Access Governance reports were specifically highlighted as ways to identify oversharing risks before Copilot rollout.


2. Create a proper information architecture

Many organisations have:

Documents
├── New Folder
├── Old Stuff
├── Final
├── Final V2
├── Copy of Final
└── Misc

Copilot struggles because the business itself has no structure.

Instead build:

Finance Hub
├── Budget Planning
├── Forecasting
├── Reporting
└── Policies

Sales Hub
├── Proposals
├── Customers
├── Pricing
└── Marketing

The clearer the structure, the easier it is for:

  • SharePoint Search

  • Microsoft Search

  • Copilot Chat

  • SharePoint Agents

  • Copilot Agents

to locate relevant content.


3. Improve file naming standards

Copilot does read document content, but filenames still matter.

Bad:

Proposal.docx
Proposal New.docx
Proposal Final.docx
Proposal Final Final.docx

Good:

CustomerName-Proposal-2026-07.docx
CustomerName-SOW-v1.docx
CustomerName-SOW-Approved.docx

In one of your SharePoint discussions, naming conventions were specifically called out as something that should be standardised across the site.


4. Use metadata instead of folders where possible

Metadata is one of the biggest Copilot improvements available.

Rather than:

Projects
 ├── Sydney
 ├── Melbourne
 ├── Brisbane

Use columns such as:

Column
Value

Client
ABC

Region
Sydney

Project Type
Migration

Status
Active

This gives Copilot richer context when searching and grounding answers.

Instead of finding a file based only on its location, Copilot can reason over:

  • client

  • project type

  • status

  • department

  • business owner


5. Remove stale content

One major challenge is outdated content appearing in search results.

Microsoft now provides features that recommend:

  • demoting inactive pages

  • identifying content gaps

  • fixing broken links

to improve discoverability and Copilot relevance.

Ask yourself:

  • Is this document still current?

  • Is there a newer version?

  • Does anyone own it?

  • Should it be archived?

A common issue is Copilot finding a policy from 2019 while a better one exists from 2026.


6. Identify authoritative sources

One of the newest improvements is the ability to mark SharePoint sites as authoritative.

Examples:

  • HR

  • Finance

  • Legal

  • Corporate Communications

Content from these sites can be prioritised in Copilot Search and Copilot Chat results.

For example:

Site
Why make it authoritative?

HR
Official policies

Finance
Budgets and governance

Legal
Contracts

Company Communications
Executive announcements

This helps reduce contradictory responses.


7. Apply sensitivity labels

Copilot respects sensitivity labels and information protection controls.

Typical labels:

  • Public

  • Internal

  • Confidential

  • Highly Confidential

Benefits include:

  • Better governance

  • Controlled sharing

  • AI visibility controls

  • Better compliance outcomes


8. Improve search quality

Copilot depends heavily on Microsoft Search.

Poor search equals poor Copilot.

Things that help:

Create quality pages

Instead of storing everything in Word documents:

  • Create SharePoint pages

  • Add summaries

  • Add FAQs

  • Add ownership information
Add page owners

Every key page should have:

  • business owner

  • review date

  • contact person
Use meaningful titles

Bad:

Welcome
General Information
Policies

Good:

Employee Leave Policy
Expense Claim Procedure
Remote Work Guidelines


9. Build hub sites

Hub sites create logical business groupings.

Example:

Corporate Hub
 ├── HR
 ├── Finance
 ├── Operations
 └── IT

Customer Hub
 ├── Sales
 ├── Marketing
 └── Service

Hub sites improve navigation, search context and content relevance for Copilot.


10. Create SharePoint agents for specialised knowledge

Where a user only needs answers from a particular area, create a SharePoint Agent.

Examples:

  • HR Agent

  • Policy Agent

  • Finance Agent

  • Project Agent

These agents ground themselves on specific SharePoint locations and often produce more accurate answers than tenant-wide searches.


11. Add more organisational context

Copilot works best when content explains:

  • who owns it

  • what it is for

  • where it applies

  • when it was reviewed

Bad document:

Procedure.docx

Good document:

Financial Approval Process
Owner: Finance
Review Date: July 2026
Applies To: Australia Operations

The extra context significantly improves grounding quality.


12. Measure and improve continuously

The best Copilot environments are not set-and-forget.

Establish a quarterly process:

Review
  • Oversharing

  • Inactive sites

  • Broken links

  • Orphaned content
Clean up
  • Duplicate files

  • Old projects

  • Stale policies
Improve
  • Metadata

  • Authority sites

  • Labels

  • Search experience

Data Access Governance reports and Content Management assessments are specifically designed for this ongoing process.

My practical SMB recommendation

If I were preparing a tenant for Copilot today, I’d prioritise:

  1. Permission cleanup

  2. Oversharing remediation

  3. Sensitivity labels

  4. Review inactive sites

  5. Standard naming conventions

  6. Create hub sites

  7. Add metadata

  8. Designate authoritative sites

  9. Build targeted SharePoint agents

  10. Quarterly governance review

In most SMB tenants, doing just those ten things improves Copilot results more than any prompt engineering or user training because you’re improving the quality of the information Copilot can see and trust.

The Recurring Problem: A Managed Services Story–Chapter 4

Previously –  https://blog.ciaops.com/2026/07/28/the-recurring-problem-a-managed-services-story-chapter-3/

image

Denise Okafor called on a Thursday in February, and this time there was no anger in her voice at all, which was somehow worse. “We’re moving our IT relationship to Meridian Health IT at the end of our contract term,” she said. “They only work with medical practices. Their entire pitch was built around HIPAA and nothing else. Dave, I like you. I’ve liked working with Bridgepoint for six years. But I can’t keep explaining to my board why our IT provider treats a medical practice exactly the same as a landscaping company. We need a security team that speaks our language natively, not one that’s learning it because we complained.”

image

Bridgepoint’s ninety-day notice clock started that afternoon. $640,000 in annual revenue — nearly seven percent of the company’s total — was walking out the door, and it was walking toward a competitor that had done exactly one thing Bridgepoint hadn’t: picked an industry, gone deep, and built a service around that industry’s specific compliance reality instead of a generic bundle stretched to cover everyone.

Dave took the loss personally, because it was personal — Lakeside had been one of his first ten clients — but the number that actually kept him up that week wasn’t the $640,000. It was something Marcus said almost in passing, trying to soften the blow: “At least it wasn’t Baptiste. If we lost Baptiste on top of this, that’d be real trouble.”

image

Baptiste Manufacturing was Bridgepoint’s single largest client, at just under 19 percent of total revenue — a family-owned metal fabrication company that Tom Baptiste had built up from his father’s two-man machine shop. Dave hadn’t thought about customer concentration in those terms before. He thought about it constantly after that.

Two weeks later, he got a call from Tom that gave him a preview of exactly how that could go. Baptiste’s plant manager had started using a free AI chatbot to draft supplier emails and, it turned out, to summarize confidential pricing data from a shared drive he didn’t fully understand the access permissions on. Nothing had leaked — yet — but Tom had found out from his own son, who worked on the shop floor and had seen it happening, and he was furious that nobody at Bridgepoint had ever mentioned that this was even a risk worth managing.

“You guys handle our computers,” Tom said. “Isn’t this exactly the kind of thing you’re supposed to be telling us about?”

image

Dave didn’t have an answer for that either. Bridgepoint had never offered anything resembling a policy for how clients’ own employees used AI tools, because as recently as eighteen months earlier, it hadn’t occurred to anyone that this was IT’s job to manage. It clearly was now. Somebody was going to own that conversation with clients. Dave just didn’t know yet whether it would be Bridgepoint or whoever Bridgepoint’s next competitor turned out to be.

image

You Always Get the Next Move

image

I had a conversation recently with someone who runs a small business, and somewhere in it he said, almost in passing, “I feel like everything’s just happening to me.” His inbox, his clients, his suppliers, the latest pricing change from a vendor — all of it landing on him like weather. He wasn’t lazy and he wasn’t short on ability. But somewhere along the way he’d started to believe the events of his day were the ones in charge, and he was only along for the ride.

That belief — that the world acts on you, rather than you acting on the world — is the most expensive idea a person can carry. And I’ve come to think the most important quality anyone can develop is its opposite: the quiet conviction that you have power. Not power over other people, or over outcomes, or over the economy. Power in the plainest sense of the word — the ability to act.

Here’s the part that’s easy to miss. No matter what is going on around you, you still get to decide what you do next. That’s the whole thing. Circumstances can narrow your options, sometimes brutally, but they never remove the choice entirely. There is always a next action, and you are the one who chooses it.

The wheel is closer than it feels

Most of the people I work with aren’t short on skill. They’re short on the sense that they’re driving. The day arrives as a flood and they meet it by reacting — answering whatever shouts loudest, putting out whatever’s burning. By five o’clock they’ve been busy for nine hours and couldn’t tell you what they actually decided. Reacting feels like working, but it isn’t the same as choosing.

I notice this most clearly in the morning, in front of a full inbox. Two hundred messages can feel like two hundred small demands, each one acting on you. But that feeling is a story, not a fact. The moment I ask Copilot in Outlook to summarise a long thread and tell me what genuinely needs me, the pile stops being weather and starts being a list — and a list is something you work through on your terms. The tool didn’t hand me more hours. It handed back the moment of choice.

Tools can’t give you agency, but they can return it

This is where I think the conversation about AI usually goes wrong. People talk about it as something that will do the work for them, as if the goal is to act less. I see it the other way around. The value of Copilot catching me up on a meeting I missed in Teams, or pulling the three figures I need out of a forty-page document in Word, isn’t that I get to disappear. It’s that the noise stops dictating my next move and I get to.

Agency was always mine. What gets in the way is the volume — the sheer weight of information and obligation that makes it feel like there’s no room to decide anything. Clear some of that weight and the choice that was there all along becomes visible again. You stop being a passenger in your own day.

So I’d offer this, for whatever it’s worth. When the next thing lands on you and the old feeling rises — this is just happening to me — pause on it. It almost never is. You still have the next move. You always do. Everything that matters tends to follow from remembering that.

The Recurring Problem: A Managed Services Story–Chapter 3

Previously – https://blog.ciaops.com/2026/07/27/the-recurring-problem-a-managed-services-story-chapter-2/

image

Aisha Malik had joined Bridgepoint two years earlier, straight out of a cybersecurity program, and had spent most of that time being the youngest, quietest person in every meeting she sat in. She stopped being quiet in November.

“I’ve been running a side project for about six weeks,” she told the leadership team, sliding a laptop across the conference table so Priya could see the screen. “I connected an AI ticket-triage tool to our help desk queue — just a trial license, I didn’t need budget approval, it’s free up to five hundred tickets a month. It auto-categorizes incoming tickets, resolves the routine stuff — password resets, printer errors, access requests — without a human touching them, and escalates anything that looks like a security event immediately, to an actual person, with a page, not an email that sits in an inbox.”

“And?” Dave said, arms crossed, in the posture Priya privately called his skeptical-uncle stance.

“And our median resolution time on the tickets it touched dropped from about nineteen hours to just over four,” Aisha said. “On the security-flagged subset — the stuff that mattered most after what happened with Lakeside — it dropped to under thirty minutes, because it doesn’t sleep, and it doesn’t leave things in a queue over a weekend.”

Jordan, who had been with Bridgepoint since year two and trained half the technicians in the room, didn’t wait to be invited into the conversation. “So what, we replace the overnight tech with a chatbot and call it a security program? I’ve been doing this eleven years. I know these networks better than any script ever will. You want to hand Lakeside’s compliance problem to a language model?”

“I don’t want to hand anything to anybody,” Aisha said, more evenly than her twenty-six years might have suggested. “I want the routine stuff off your plate so you can spend your time on the things that actually need eleven years of judgment. Right now you’re spending forty percent of your week resetting passwords. That’s not judgment. That’s just volume.”

The room went quiet in the particular way rooms go quiet when somebody has said something true that other people would rather not examine too closely.

Dave ended the meeting without a decision, which was its own kind of decision. Driving home that night, he found himself doing math he hadn’t done in years: what percentage of Bridgepoint’s revenue was actually predictable, contracted, recurring money, versus the unpredictable project work Marcus chased every quarter. He guessed, generously, somewhere around fifty percent. He would later find out he was wrong, and not in the direction he hoped.

image

For the next several months, Dave did what a lot of successful founders do when the evidence starts contradicting the story they’ve told themselves for a decade: he found reasons the evidence didn’t apply to him. AI was hype, aimed at enterprises with budgets Bridgepoint’s clients didn’t have. Sentio Cyber would burn out; boutique undercutters always did. Lakeside’s incident was a one-off, a process gap, not a symptom. He said some version of “we’ll keep an eye on it” in four consecutive leadership meetings, and each time, Priya wrote the date in a notebook she kept for exactly this purpose.

image

The Clock and the Compass

image

I watched two people come undone this year, in completely opposite ways.

One never stops moving. Every day is a sprint — inbox at zero by eight, three deadlines cleared by lunch, a task list that breeds faster than he can knock it down. He hits his targets and feels nothing, because there’s already another target waiting. He’s busy in a way that looks impressive and feels hollow. Ask him where all this running is heading and he goes quiet.

The other has the opposite problem. She knows exactly where she’s going. She’ll talk for an hour about the business she’s building five years out, the kind of work that genuinely matters to her. But the payoff sits so far away that nothing she does today feels like a win. Months drift past. The vision is lovely, and the week is empty of anything that actually moved her closer to it.

I’ve been both of these people. Most of us only ever pick one.

There’s a clock in all of us, and there’s a compass. The clock is urgency — it’s what gets you out of the chair. The compass is direction — it tells you whether all that motion is worth anything. On their own, each one quietly fails you. All clock, and you run hard in circles. All compass, and you stand in the driveway, certain of the destination, never turning the key.

Pick a direction you’d never want to finish

The first move is choosing something big enough that you can’t tick it off. Not a goal with a finish line, but a pursuit — getting sharper at your craft, serving a particular kind of client properly, building something that outlasts you. That’s the compass. It doesn’t expire the moment you hit a number.

This is where I lean on Copilot more than people expect. When the long game feels fuzzy, I open a blank Word document, bring up Copilot, and just talk to it — what am I actually trying to build, what does good look like in three years, what would I regret not starting. It drafts, I push back, it sharpens. Twenty minutes later I’ve got a direction I can re-read on a Monday morning when everything else is shouting for attention.

Then build short games you can actually win

A compass won’t get you moving on a Tuesday. You need something to chase this week. So I carve the long thing into small games with a real scoreboard — wrote this, rang that client, shipped one improvement. Proper finish lines, close enough to feel.

I keep mine in Planner, with Copilot doing the slicing. I’ll describe the quarter out loud and ask it to break the work into weekly chunks I can tick off. Then every Friday I get Copilot to pull together what genuinely got done across my Teams chats, my sent items and my tasks — not what I planned, what moved. That summary is my clock. It’s the small hit of winning that the compass can never give me, and it stops a good direction from slowly going nowhere.

The trick was never choosing between the two. It’s wiring them together — a direction you’d hate to ever finish, fed by a steady drip of small wins you can finish by Friday.

The people running on empty are short a compass. The people standing still are short a clock. Look honestly at your own week. Whichever one you’re missing — that’s the one to fix.