Comparing AI services–an objective analysis?

bp1

If you have been following my articles about comparing AI services, you’d know that, through some ‘rule of thumb’ reasoning I was able to determine the following ranking of Ai services:

1. Deepseek

2. M365 Copilot

3. Copilot Researcher

4. Gemini

5. Copilot Studio

6. ChatGPT deep research

7. ChatGPT

The problem is that I used the same AI services to potentially evaluate the results that they in fact generated. Could that result in bias? Unsure, but I’d suggest probably, if you look at the results.

What I therefore decided to do was have the original articles evaluated by two AI services that were not on my original list, Claude and Grok. Here’s the result of jus these two:

AI Service Claude Grok Total
M365 Copilot 7 4 11
Gemini 3 7 10
Copilot Studio 5 5 10
Deepseek 6 2 8
Copilot Researcher 2 6 8
ChatGPT Deep Research 4 3 7
ChatGPT 1 1 2

If I now incorporate these results in the overall results I get the following:

AI Service Researcher Gemini ChatGPT Claude Grok Total
M365 Copilot 7 3 4 7 4 25
Deepseek 5 4 7 6 2 24
Gemini 4 7 2 3 7 23
Copilot Studio 2 5 5 5 5 22
Copilot Researcher 6 6 1 2 6 21
ChatGPT Deep Research 3 2 3 4 3 15
ChatGPT 1 1 6 1 1 10

That changes the ranking slightly to:

1. M365 Copilot

2. Deepseek

3. Gemini

4. Copilot Studio

5. Copilot Researcher

6. ChatGPT deep research

7. ChatGPT

with the average score being 20, which most services exceed. ChatGPT still lags, even after this! Interesting, huh?

I think my original conclusion remains valid – most AI services, except for ChatGPT, seem to produce very similar quality on average when prompted in the same way.

Comparing AI Services–the final analysis

bp1

I started out to provide an indication of the differences between different AI services here:

Testing the differences between AI services

I did a quick comparison here:

An analysis of how AI services vary

I then did a deep analysis of all the generated articles using:

Copilot Researcher

Gemini Deep Thinking

ChatGPT Deep Thinking

If you now take those three results and assign a score of 7 = highest and 1 = lowest recommendations of each and total them up, you end up with this ranking table:

AI Service Researcher Gemini ChatGPT Total Score
Deepseek 5 4 7 16
M365 Copilot 7 3 4 14
Copilot Researcher 6 6 1 13
Gemini 4 7 2 13
Copilot Studio 2 5 5 12
ChatGPT Deep Research 3 2 3 8
ChatGPT 1 1 6 8

 

The winner then appears to be, on average, Deepseek. However, you will note that most AI services tested, except ChatGPT have similar scores, with the ‘average’ score being 12, which most services, except again ChatGPT, scored at or above.

This analysis is far from perfect or ideal or for that matter without bias. There are so many variables that possibly come into play that it very difficult, if not impossible, to get a true ‘apples vs apples’ comparison of AI services. However, I think this result still does provide value if you are looking to answer the question of the ‘best’ AI service. That answer seems to largely be that most AI services, apart from ChatGPT, are pretty much the on par when it comes to prompting, so choosing from amongst these simply based on their response to prompts, doesn’t seem to matter all that much.

Of course, there are plenty of other factors, aside from prompt results, that should be considered. The quality of the generated results also is greatly affected by the actual prompts used and I am sure that also varies across the AI services as well.

What I’ll now be interested to see is what the ‘click’ rate is on each article after a period of time. Will the Google AI service generate more article ‘hits’ than the other articles? Time will tell and I’ll report back once enough time has elapsed. These results also make a good benchmark to potentially test again down the track to see if things have changed at all and the progress these AI agents have made.

Interesting time ahead.

Comparing AI services – a third analysis

bp1

Recently, I have been analysing the results produced from the same prompt in carious AI services. After having the various AI services generate answers I compared their value using Copilot Researcher and Gemini. To provide a final alternate analysis of the articles I used ChatGPT Deep Research and received the following analysis, summarised here:

————————–

Rankings (Value to SMB Owner): Based on clarity, practicality, and depth of recommendations, we rank the articles as follows (1 = most valuable):

  1. Deepseek (Aug 31) – Provides a step-by-step guide on replacing firewall functions with M365 features, with concrete examples (Safe Links, web filtering, Conditional Access) and even a cost comparison. Its clear bullet format and action-oriented advice make it highly accessibleblog.ciaops.comblog.ciaops.com.

  2. ChatGPT (Sep 2) – Gives an ultra-concise Q&A answer summarizing the essentials. The “Answer in short” explicitly states that for most SMBs a basic firewall suffices and expensive NGFWs add little compared to built-in M365 protectionsblog.ciaops.com. Its brevity makes it very easy to digest (2‑minute read), though it is simple.

  3. Copilot Studio (Aug 30) – A bullet-point summary that quickly lists the key protections in M365 (endpoint EDR, email filtering, conditional access, DLP, etc.)blog.ciaops.comblog.ciaops.com. Its Q&A style (“Short answer”) is user-friendly and covers the main points succinctly.

  4. M365 Copilot (Sep 3) – Offers a short answer plus a configuration blueprint (steps to enforce MFA, Intune baselines, Defender settings, etc.) focused on identity/device security. This balances brevity with practical steps. It clearly states that for remote-first SMBs, a basic router is enough and effort should go into Intune/M365 policiesblog.ciaops.comblog.ciaops.com.

  5. ChatGPT (Aug 28) – A traditional blog-style article with multiple sections on built-in security, Zero Trust, host firewalls, etc. It thoroughly covers many features of Business Premiumblog.ciaops.comblog.ciaops.com. While detailed, its long paragraphs (7‑min read) and technical tone may be harder for a busy non-expert to quickly follow.

  6. Gemini (Sep 1) – A formal, research-style analysis (13‑min) with executive summary, threat modeling, TCO, and compliance discussionblog.ciaops.comblog.ciaops.com. It is extremely comprehensive (notably covering things like PCI DSS) but in heavy academic tone. This depth is impressive but likely more than a typical SMB owner needs.

  7. Copilot Researcher (Aug 29) – An exhaustively detailed report (20‑min) with an executive summary, configuration guide, feature comparisons, and cost analysisblog.ciaops.comblog.ciaops.com. While it contains valuable info, its length and technicality make it the hardest to consume. It also concludes (like the others) that M365 BP can often replace expensive firewallsblog.ciaops.com, but the sheer volume can overwhelm small teams.

In summary, the shorter Q&A and bullet-list articles (Deepseek, ChatGPT Sep2, Copilot Studio, M365 Copilot) score highest for SMB owners because they deliver clear guidance quickly. The longer, more technical write-ups (Gemini, Copilot Researcher) score lower despite depth.

Overall Best: Deepseek’s article stands out as the most practical: it clearly explains why and how to use M365 features instead of buying pricey firewalls, with concrete steps and examplesblog.ciaops.comblog.ciaops.com. This level of clarity makes it particularly valuable for a small-business owner deciding on cost-effective security.

An alternate analysis of AI services

bp1

Recently, I have been analysing the results produced from the same prompt in carious AI services. After having the various AI services generate answers I compared their value using Copilot Researcher. To provide an alternate analysis of the articles I used Gemini Deep Research and received the following analysis, summarised here:

————————–

The seven articles provided by the user all arrive at a similar, well-reasoned conclusion regarding the strategic superiority of M365 Business Premium for the typical SMB. However, they vary in their clarity, depth, and how well they frame the discussion for a business-focused audience. The ranking below is based on an assessment of these qualitative factors, determining which AI tool provided the most valuable and actionable report for an SMB owner.

Ranking Criteria
  • Clarity & Structure: How easy is the argument to follow? Is the report well-structured and free of jargon?
  • Comprehensiveness: Does the report cover all key aspects of both solutions and the modern security landscape?
  • Depth of Insight: Does the report go beyond a simple feature list to discuss total cost of ownership, strategic alignment, and the new security model?
  • Actionability: Does the report provide a clear, practical recommendation for the SMB owner?

AI Tool Rankings – M365 vs Hardware Firewalls

AI Tool Rank Justification Key Strength
Gemini (Deep Research) 1 The article is a near-perfect strategic analysis, framing the decision from a business perspective. It provides the crucial total cost of ownership (TCO) discussion and, uniquely, identifies the critical PCI DSS exception, which demonstrates a high degree of nuance and awareness of a real-world business constraint. Exceptional strategic framing and business-centric analysis.
Copilot Researcher 2 The article provides a highly structured and detailed summary. Its strength lies in its comprehensive breakdown of the limitations of traditional firewalls, using clear, bulleted points that are easily digestible for a busy executive. The logical flow of the argument is excellent, making it very easy to understand why the old model no longer works. A comprehensive and highly structured breakdown of the problem.
Copilot Studio (with GPT-5 reasoning) 3 This output provides a very strong, detailed breakdown of the specific M365 Business Premium components. It excels at explaining how each component directly replaces a traditional perimeter security function. The argument is well-reasoned and authoritative, linking the new tools to the Zero Trust model. Outstanding detail on the M365 BP components and their function.
Deepseek (Deep Research) 4 This article is notable for its use of the powerful “firewall in the cloud” metaphor, which effectively communicates the strategic shift to a non-technical audience. It clearly outlines how M365 BP provides specific feature replacements, such as web content filtering and intrusion prevention, making the report very practical. Effective use of a powerful metaphor to simplify a complex topic.
M365 Copilot (GPT) 5 This article introduces the “thin-edge, strong-endpoint” security model, a key strategic concept that refines the discussion beyond a simple comparison. It provides a solid summary of how cloud and endpoint controls outperform older, perimeter-based setups in terms of TCO and risk reduction. Introduction of a key strategic model for modern security.
ChatGPT (Deep Research) 6 This article is a solid, albeit slightly more generic, summary of the core arguments. It is comprehensive and includes the key components of M365 BP that are relevant to the comparison, but it lacks the more granular, strategic framing seen in the higher-ranked outputs. It’s a good starting point but not as definitive as the others. A solid, comprehensive overview of the core points.
ChatGPT 7 This output is the most list-based and least narrative of the group. While it accurately lists the advanced security controls in M365 BP, it is more of a feature catalog than a strategic report. It does not provide the same level of business context or direct analysis of the TCO and strategic alignment as the others, making it less valuable for an executive decision-maker. Good at categorizing and listing the specific security controls.

Need to Know podcast–Episode 353

in this episode I talk about a recent series of blog posts I wrote about some analysis I did on various AI services available today as well as my thoughts on these. I also cover off the latest news and information in the Microsoft Cloud for you. Listen along.

Brought to you by www.ciaopspatron.com

you can listen directly to this episode at:

https://ciaops.podbean.com/e/episode-353-ai-services-analysis/

Subscribe via iTunes at:

https://itunes.apple.com/au/podcast/ciaops-need-to-know-podcasts/id406891445?mt=2

or Spotify:

https://open.spotify.com/show/7ejj00cOuw8977GnnE2lPb

Don’t forget to give the show a rating as well as send me any feedback or suggestions you may have for the show.

Resources

CIAOPS Need to Know podcast – CIAOPS – Need to Know podcasts | CIAOPS

X – https://www.twitter.com/directorcia

Join my Teams shared channel – Join my Teams Shared Channel – CIAOPS

CIAOPS Merch store – CIAOPS

Become a CIAOPS Patron – CIAOPS Patron

CIAOPS Blog – CIAOPS – Information about SharePoint, Microsoft 365, Azure, Mobility and Productivity from the Computer Information Agency

CIAOPS Brief – CIA Brief – CIAOPS

CIAOPS Labs – CIAOPS Labs – The Special Activities Division of the CIAOPS

Support CIAOPS – https://ko-fi.com/ciaops

Get your M365 questions answered via email

Welcome to the Microsoft Incident Response Ninja Hub –

https://techcommunity.microsoft.com/blog/microsoftsecurityexperts/welcome-to-the-microsoft-incident…

Listen to an audio recap of your meetings in Teams –

https://techcommunity.microsoft.com/blog/Microsoft365InsiderBlog/listen-to-an-audio-recap-of-your-m…

Introducing Surveys Agent, your personal survey expert –

https://techcommunity.microsoft.com/blog/microsoft365insiderblog/introducing-surveys-agent-your-per…

What’s New in AI for Security from Microsoft Entra? –

https://techcommunity.microsoft.com/blog/microsoft-entra-blog/what%E2%80%99s-new-in-ai-for-security…

Microsoft ranked number one in modern endpoint security market share third year in a row –

https://www.microsoft.com/en-us/security/blog/2025/08/27/microsoft-ranked-number-one-in-modern-endpoint-security-market-share-third-year-in-a-row/

Securing and governing the rise of autonomous agents –

https://www.microsoft.com/en-us/security/blog/2025/08/26/securing-and-governing-the-rise-of-autonomous-agents/

How systems integrators are scaling innovation with Microsoft 365 Copilot and agents –

https://partner.microsoft.com/en-US/blog/article/copilot-partner-spotlight-august-2025

Microsoft deployment blueprint – Address oversharing concerns for your M365 Copilot deployment –

https://techcommunity.microsoft.com/blog/healthcareandlifesciencesblog/microsoft-deployment-blueprint—address-oversharing-concerns-for-your-m365-copi/4434598

Staying Ahead of Compliance: Keep Up with Key Insights from our Quarterly Compliance Update –

https://techcommunity.microsoft.com/blog/microsoft365copilotblog/staying-ahead-of-compliance-keep-up-with-key-insights-from-our-quarterly-complia/4448011

Microsoft Security Copilot in Intune deep dive – Part 1: Features available in public preview –

https://techcommunity.microsoft.com/blog/intunecustomersuccess/microsoft-security-copilot-in-intune-deep-dive-%E2%80%93-part-1-features-available-in-pu/4406244

What’s New in Microsoft Intune: August 2025 –

https://techcommunity.microsoft.com/blog/microsoftintuneblog/what%E2%80%99s-new-in-microsoft-intune-august-2025/4445612

OneNote for Windows 10 support is ending –

https://techcommunity.microsoft.com/blog/microsoft365insiderblog/onenote-for-windows-10-support-is-ending/4445230

Think before you Click(Fix): Analyzing the ClickFix social engineering technique –

https://www.microsoft.com/en-us/security/blog/2025/08/21/think-before-you-clickfix-analyzing-the-clickfix-social-engineering-technique/

Deep Dive: DLP Incidents, Alerts & Events – Part 1 –

https://techcommunity.microsoft.com/blog/microsoft-security-blog/deep-dive-dlp-incidents-alerts–events—part-1/4443691

Deep Dive: DLP Incidents, Alerts & Events – Part 2 –

https://techcommunity.microsoft.com/blog/microsoft-security-blog/deep-dive-dlp-incidents-alerts–events—part-2/4443700

New SKUs available for M365 Business premium – https://techcommunity.microsoft.com/blog/microsoft-security-blog/deep-dive-dlp-incidents-alerts–events—part-2/4443700

Testing the differences between AI services – CIAOPS – https://blog.ciaops.com/2025/09/06/testing-the-differences-between-ai-services/

An analysis of how AI services vary – CIAOPS – https://blog.ciaops.com/2025/09/07/an-analysis-of-how-ai-service-vary/

Comparison of AI-Generated Articles – CIAOPS – https://blog.ciaops.com/2025/09/08/comparison-of-ai-generated-articles/

Comparison of AI-Generated Articles

bp1

Recently, I’ve been researching different Ai tools and the results they generate when give the same prompt. For the next piece in the analysis I have asked Microsoft 365 Researcher to compare, rate and rank them all. Here are the results:

———————————————–

Seven articles – each authored by a different AI tool – examine whether Microsoft 365 Business Premium’s security features can replace traditional hardware firewalls for small/medium businesses (SMBs). Below, we compare these articles across key dimensions (depth, accuracy, relevance, clarity, and unique insights) and rank them by overall value to an SMB decision-maker. Despite different styles, all the articles reach a similar conclusion: for most cloud-focused SMBs, an expensive next-gen firewall provides diminishing returns if Microsoft 365 Business Premium is fully utilized[1][1]. The differences lie in how comprehensively and clearly each article makes its case.

Depth of Analysis

Depth of analysis ranges from succinct overviews to exhaustive reports. The Copilot Researcher (Aug 29) article is by far the deepest dive – a 20-minute read with an executive summary and a full breakdown of traditional firewall functions vs. M365’s capabilities[2]. It details everything from legacy VPN issues to Zero Trust principles, providing extensive background and even historical context (e.g. how remote work “dissolves” the network perimeter)[2]. Similarly, the Gemini (Sep 1) piece offers a structured 13-minute strategic analysis with numbered sections (I, II, III, etc.), multiple subheadings, and footnoted references supporting each point[3][3]. This gives it considerable depth as well, exploring business implications and technical details in tandem.

In contrast, the ChatGPT standard (Sep 2) article is very shallow – a 2-minute quick read structured as a 6-point list that hits the high notes without delving into specifics[1][1]. It’s essentially a summary of conclusions and key factors. The Deepseek (Aug 31) article is also relatively brief at ~4 minutes, but still manages to cover multiple points in a numbered list format, making it concise yet informative (e.g. points 1 through 3 map M365 features to firewall functions)[4][4]. ChatGPT (Deep Research, Aug 28) and Copilot Studio (Aug 30) fall in the middle: around 6–7 minutes each. The ChatGPT (Deep Research) piece provides a moderate level of detail, describing M365’s built-in layers and giving examples (like how Conditional Access extends the perimeter to trusted devices)[5], but it doesn’t have the full formal structure of the longer articles. Copilot Studio’s article (~6 minutes) is packed with content – it reads like a practical checklist with references – thereby achieving significant depth in condensed form (for example, it enumerates 7 configuration steps for using Business Premium as a “firewall” replacement, under headings like 1) Identity and access, 2) Device onboarding, etc.[6][6]). Overall, Copilot Researcher has the greatest depth, followed by Gemini and M365 Copilot, whereas ChatGPT’s basic version provides the least depth.

Technical Accuracy

All seven articles demonstrate high technical accuracy, describing Microsoft 365 Business Premium’s security features correctly and in line with known Microsoft documentation. Several articles explicitly bolster their accuracy by citing sources or using official terminology:

  • Copilot Studio (GPT-5) and M365 Copilot articles integrate direct Microsoft Learn references. For example, Copilot Studio’s piece links out to docs for Defender for Business, Safe Links, Conditional Access, etc., in-line[6][6], ensuring factual correctness about what each feature does. The M365 Copilot article (Sep 4) likewise uses footnotes referencing Microsoft guides and latest services (e.g. Microsoft Entra Global Secure Access) – it mentions these services as not included in Business Premium but available as add-ons[1], which is an up-to-date detail. This indicates a strong grasp of current Microsoft offerings.
  • The Gemini (Deep Research) article uses many footnote references as well, implying data points like “MFA alone blocks 99.9% of account attacks”[2] and other stats were taken from authoritative sources. Its discussion of PCI DSS requirements for firewalls is accurate (PCI DSS does require a dedicated firewall if cardholder data is on-prem)[3]. Including such specifics shows trustworthy accuracy and nuance.
  • ChatGPT (Deep Research) and Copilot Researcher provide technically correct content (e.g. listing included features like Defender for Office 365 P1, Intune, Azure AD P1 – all indeed part of Business Premium[5][5]). Copilot Researcher’s long article is thorough in explaining technical limitations (like the challenge of inspecting encrypted traffic with a firewall)[2], demonstrating accurate understanding of network security issues beyond just Microsoft’s domain.
  • Even the short ChatGPT summary hits accurate points: for instance, it notes that NGFW features (like deep packet inspection, sandboxing) are overkill if using M365 and reiterates that identity/endpoints are the real focus now[1][1]. It doesn’t cite sources, but nothing in it appears incorrect or misleading.

In summary, all articles are technically accurate. The differences are more about thoroughness than correctness. The articles that cite specific guides or statistics provide extra confidence in accuracy (Copilot Studio, Copilot Researcher, Gemini, M365 Copilot), whereas the more narrative ones lean on general knowledge which still aligns with known best practices.

Relevance to SMB Decision-Makers

When judging relevance for an SMB owner/decision-maker, we consider how well the article addresses business needs (cost, simplicity, risk trade-offs) in understandable terms. In this regard, some articles explicitly frame their content for decision-makers:

  • Copilot Researcher (Aug 29) opens with an Executive Summary that directly poses the SMB’s dilemma (“expensive firewall appliances vs. M365’s security features”) and gives a bottom-line finding[2]. It continues to compare features and costs, which is highly relevant for making a purchase decision. Despite its length, the executive summary and conclusion guide an SMB reader to the key takeaways without requiring a full read.
  • Deepseek (Aug 31) and ChatGPT (Sep 2) are very on-point for SMBs due to brevity and focus. Deepseek’s article explicitly speaks about spending budget wisely, using an analogy (“fortress-like firewall to protect an empty castle”) that a business owner can relate to intuitively[4]. It also highlights that money is better spent on securing identities/data and even mentions investing in user training as a “human firewall” in the conclusion[4] – practical advice a non-technical manager would find relevant. The ChatGPT short article similarly cuts straight to what an owner cares about: do I still need to buy a big firewall or not? Its final “Answer in short” is practically a direct recommendation to the SMB: a basic router plus M365 is enough in most cases; put your money into M365’s security, not a $10k appliance[1].
  • The M365 Copilot (GPT, Sep 4) article is tailored to both audiences – it starts with a “Short answer” summary in plain language that clearly states you usually don’t need a high-end firewall if Business Premium is well-configured[1]. This is immediately useful to an SMB decision-maker. It then transitions into very detailed guidance that an IT specialist would use. The presence of that summary means an owner can read one paragraph to get the gist, and optionally have their IT staff act on the detailed blueprint.
  • ChatGPT (Deep Research, Aug 28) stays relevant by emphasizing the SMB scenario throughout – it begins by noting SMBs have shifted to Zero Trust and cloud, and explicitly states how a “traditional on-premises perimeter… (expensive firewall) becomes far less critical”[5]. It also includes a “Cost vs. Benefit of Dedicated Firewalls” section that plainly argues a $2K firewall yields little extra security for a remote-centric SMB[5]. Discussing cost-benefit in business terms makes it quite relevant to decision-makers.

The more technical or formal pieces, like Copilot Studio’s step-by-step guide and Gemini’s strategic analysis, are slightly less accessible to a non-technical owner. Copilot Studio’s content is extremely useful for an IT admin setting up security (lots of configuration detail), but an SMB owner might skim the “Short answer” at the top and glaze over the rest. The Gemini article reads like a strategy whitepaper – great for a CIO or consultant who wants to deeply justify a decision, but an average small business owner might find it too dense (it doesn’t boil things down as succinctly, though it does have an executive summary and “Key Findings at a Glance” section highlighting business-centric points like TCO)[3][3].

Bottom line: Articles that address cost, compliance, and clear recommendations (Copilot Researcher, Deepseek, ChatGPT short, M365 Copilot) score highest for SMB relevancy. Those that are heavy on technical implementation or academic tone, while valuable, might need an IT intermediary to translate for a business owner.

Clarity and Readability

Clarity varies with writing style and structure:

  • The ChatGPT (Sep 2) list-style article is extremely clear and easy to read. It’s structured with numbered points 1–6, each with a bolded heading and brief explanation[1][1]. There’s no fluff or jargon overload, making it digestible for any reader. Similarly, the Deepseek (Aug 31) article uses a simple numbered list (1, 2, 3, …) with short paragraphs under each, plus a concluding recommendation. Its language is straightforward (“investing thousands in a firewall to protect an empty office is a misallocation” is a plain-English, memorable statement[4]). These two are probably the most readable for non-experts.
  • Copilot Studio (Aug 30) is clear for technical readers. It’s basically a well-organized checklist with sub-sections and even sub-bullets for recommendations (it reads like documentation). Every important term is explained or tied to a reference link. However, because it’s dense with IT terms (MFA, ASR rules, TLS, VPN, etc.), a non-technical reader might find it less clear. The format (short answer, then lots of steps) at least separates the high-level idea from the details.
  • Copilot Researcher (Aug 29) and Gemini (Sep 1) employ formal report structures. Copilot Researcher’s clarity benefits from headings and an executive summary; it’s long, but you can navigate it easily. It defines concepts as it goes (e.g., listing firewall capabilities and then immediately their limitations in today’s context)[2][2], which improves understanding. The Gemini article is arguably the most dense in prose style – it reads like an analyst report with complex sentences and heavy use of adjectives (e.g., calling the hardware firewall “a relic of a bygone era” in the conclusion)[3]. It’s well-written and precise, but requires careful reading. For a detail-oriented reader, it’s clear; for a quick skim, it might be challenging.
  • ChatGPT (Deep Research, Aug 28) has an accessible narrative style. It flows like a blog post, not a dry report, and uses real-world logic (“if fully configured, the need for an expensive firewall is greatly reduced”[5]). It doesn’t explicitly label sections with numbers or bullet points, but transitions through topics (Zero Trust, host firewalls, when to still use a firewall) in a logical order. Many sentences are short and to the point, aiding clarity.
  • The M365 Copilot (Sep 4) article balances clarity with completeness. It starts with a very clear short answer (literally labeled “Short answer”) stating the thesis in one sentence[1], then uses bold subheadings for each major part of the discussion (which are numbered 1–5 in the text). It also uses call-out formatting like for the summary recommendation, which in the blog stands out visually[1]. The presence of footnote numbers in the text could slightly clutter readability, but those can be ignored if one is just reading the main text. Overall it’s well-structured and reader-friendly, providing clarity for both high-level and detail-level readers.

In terms of overall readability, the shorter, list-driven articles (ChatGPT standard, Deepseek) are clearest. The longer ones are still clear but demand more attention. None of the articles is poorly written; it’s more a question of audience fit – technical folks will find all of them clear, while a layperson will gravitate to the simplest presentations.

Unique Insights and Recommendations

Each article adds its own flavor of insight beyond the basic argument (“use M365 security, not just firewalls”):

  • Deepseek (Aug 31) stands out for its visual cost-benefit comparison. It literally provides a mini table comparing the traditional approach vs. modern approach for each security layer[4]. For example, it contrasts “High-end enterprise firewall ($3k+ + annual fees)” with “Basic firewall ($500–$1k) for the office,” and “Firewall subscription for DNS filtering” with “Defender for Endpoint Web Content Filtering (Included)”[4]. This side-by-side approach, plus explicit dollar figures, is a unique and very practical way to show value. This article also uniquely emphasizes user security training as part of the solution[4], something others only hint at.
  • Gemini (Sep 1) brings a strategic business perspective. It explicitly discusses Total Cost of Ownership (TCO) and makes a point that M365’s subscription model is more predictable and consolidated than buying separate security appliances[3]. It also uniquely highlights SMB resource constraints – noting that SMBs often lack in-house expertise to manage complex firewalls, which is a strong argument for a simpler cloud solution[3]. Additionally, Gemini is the only one to strongly call out compliance exceptions: if you handle credit card data (PCI DSS), a high-end firewall might be mandated despite the general advice[3]. That nuance adds credibility and is a helpful caveat for specific readers.
  • Copilot Studio (GPT-5, Aug 30) provides a granular “how-to” that others don’t. Its step-by-step list of how to configure Business Premium in lieu of a firewall (covering MFA, device compliance, Attack Surface Reduction rules, etc.) is essentially a mini implementation guide[6][6]. This is invaluable for IT personnel who want to follow the recommendation – it bridges the gap between theory and practice. It also enumerates clear criteria for when a higher-end firewall could still be justified (like specific on-prem needs or compliance mandates)[6][6], similar to some other articles but presented succinctly in a “consider if…” list.
  • Copilot Researcher (Aug 29) offers breadth of context: it deeply explains legacy vs. modern security in SMB terms – for instance, it describes how forcing all remote traffic through VPN/firewall is cumbersome and often not done, exposing those users[2]. It basically reads like a mini-research paper on SMB network security, which can enlighten readers on why the shift is happening (not just that M365 has features). Its breadth (from firewall functions, to Zero Trust, to specific Microsoft features, to a recommended policy checklist toward the end) provides a one-stop knowledge source. One particularly insightful part is how it underscores the “beyond the firewall” trend – quoting that firewalls were built for a perimeter that no longer exists[2] – framing M365’s approach as the future-ready one.
  • M365 Copilot (GPT, Sep 4) is notable for mentioning Microsoft’s latest Security Service Edge (SSE) offerings. It suggests that if one still wants centralized web traffic control without hardware, Microsoft Entra Internet Access (a cloud-based secure web gateway) and Entra Private Access (for VPN-less app access) are options[1]. No other article mentions this new Microsoft solution. This forward-looking insight could be very useful for readers considering the cutting edge of cloud security. The M365 Copilot piece also introduces the catchy concept of “thin edge, strong endpoint” model[1], neatly summarizing the philosophy of relying on cloud/endpoint security rather than a heavy perimeter – a phrasing that might stick with readers.
  • ChatGPT (Deep Research, Aug 28), while covering points also seen elsewhere, emphasizes a balanced view: it clearly states a basic firewall/router is still recommended for certain roles (segmentation, VPN, etc.)[5] and gives examples of how Azure AD Application Proxy or Azure VPN can replace traditional firewall functions[5]. It might not have one singular unique feature, but it’s strong in tying all pieces together in a concise way.
  • The ChatGPT (standard) article’s unique aspect is essentially its extreme brevity and focus. It doesn’t introduce new technical insights, but one could say its value is showing how an AI (ChatGPT) can compress the answer into a very actionable summary. It’s the kind of thing an SMB might read as a quick answer or that you’d find as a summarized answer on a forum.

To sum up, each article adds value beyond the overlap in core message. From cost tables to compliance notes, from implementation checklists to new cloud services, these insights differentiate the articles and reflect the strengths of the respective AI tools that generated them.


Comparison Table of Articles by Key Criteria

Below is a side-by-side comparison of the seven AI-generated articles, evaluating how each performs in various dimensions:

M365 Business Premium vs. Hardware Firewalls – Article Review

Article (AI Tool) Depth of Analysis Technical Accuracy Relevance to SMBs Clarity & Readability Unique Insights / Recommendations
ChatGPT (Deep Research)
“M365 Business Premium vs. Hardware Firewalls for SMBs”
Aug 28, 2025
Moderate depth. ~7-minute read covering major M365 security layers and firewall roles. Descriptive narrative but not exhaustive. High. Accurately describes built-in features (Defender AV, MFA, Intune, etc.) with links to Microsoft docs. No obvious errors; aligns with best practices (e.g., enabling OS firewalls). High. Directly addresses SMB context (remote work, cost) and draws a clear conclusion about reducing firewall spend. Mentions cost vs benefit plainly. Good clarity. Flows logically in plain language. No heavy jargon; uses real-world examples (coffee shop Wi-Fi scenario). Easy for a general audience to follow. Balanced advice. Emphasizes setting up M365 security properly to replace firewalls. Notes a basic firewall is still useful for certain network functions. Underscores Zero Trust mindset and device-based protection.
Copilot Researcher
“Security Without the High-Priced Firewall: M365 vs Traditional Firewalls”
Aug 29, 2025
Very deep. ~20-minute detailed report. Covers traditional firewall capabilities and limitations, then systematically covers M365’s equivalents and setup. Includes executive summary and in-depth analysis (akin to a whitepaper). High. Well-researched and source-backed (numerous footnote references). Cites stats (e.g., MFA stops 99.9% of attacks) and Microsoft sources. Comprehensive and technically sound; explains concepts like VPN pitfalls and SSL inspection accurately. High. Framed for decision-making: Exec summary + cost/effectiveness comparison guide an SMB reader. Clearly highlights the shift needed for remote-work security. Perhaps longer than busy owners prefer, but key points are upfront. Clear but lengthy. Organized with headings and bullet lists. Reads somewhat like a formal report, but key messages are reiterated for clarity. Plain subheadings aid navigation. Extremely thorough. Provides a full feature-by-feature comparison and a recommended mitigation checklist. Highlights often-missed points (e.g., firewalls can’t verify device health). Strong “don’t invest in big firewall, invest in M365 security” message with justified reasoning.
Copilot Studio (GPT-5)
“Why Business Premium can replace most perimeter security for typical SMBs”
Aug 30, 2025
Detailed. ~6-minute read that is content-dense. Provides a “Short answer” summary followed by a step-by-step guide (7 numbered steps) to implement M365’s security configuration. Also lists scenarios when a bigger firewall is needed. Very high. Every claim is backed with references to official documentation. Accurately lists M365 features/capabilities and how they correspond to firewall functions. Essentially an accurate compilation of Microsoft’s own guidance, tailored to SMB needs. Medium for owners, high for IT pros. The “short answer” upfront is useful to anyone, but the bulk is a configuration roadmap more relevant to IT staff than a business owner. Clear advice on when an advanced firewall is justified helps strategic decisions. Structured & technical. Clarity is good due to numbered sections and concise points, but it’s written in IT language. Non-technical readers might skip details, but overall it’s well-organized and not verbose. Actionable insights. Provides a practical blueprint for replacing firewall functions. Explicitly delineates basic vs advanced firewall use cases, aiding decision-making.
Deepseek (Deep Research)
“How M365 redefines the need for expensive hardware”
Aug 31, 2025
Concise. ~4-minute read, but covers a lot via a structured list. Each point is focused on a key argument with a few concrete examples. High. Captures the essence of M365’s capabilities correctly. Uses simple, correct analogies (identity is the new perimeter). Includes accurate product names and features. High. Tailored to SMB realities: directly states that fully remote SMBs shouldn’t invest in “fortress” firewalls and budget is better spent elsewhere. Cost-saving argument resonates strongly. Very easy to read. Subheadings and even an ASCII network diagram illustrate points. Short, punchy sentences. Clear conclusion in plain terms. Distinct visuals & cost focus. Includes a cost vs. benefit table contrasting traditional vs. M365-centric approaches. Stresses training users as a “human firewall,” a practical non-technical tip.
Gemini (Deep Research)
“Cybersecurity for the Modern SMB: A Strategic Analysis of M365 vs High-End Firewalls”
Sep 1, 2025
Comprehensive. ~13-minute analytical piece with multiple sections (Executive Summary, findings, etc.). Covers policy, cost, and context thoroughly, but less implementation-heavy. High. Very thorough and well-referenced. Describes Zero Trust principles and M365 features accurately. Notes specific compliance cases like PCI DSS that require firewalls. Moderate-High. Strategic content is highly relevant. Somewhat formal/academic tone, requiring focus, but key points are very pertinent. Formal but structured. Numbered sections and logical flow aid clarity. Executive Summary condenses arguments for quicker reading. Strategic insights. Highlights cost efficiency and workforce realities. Strong recommendation against top-tier firewalls except in compliance scenarios.
ChatGPT (Standard)
“M365 Business Premium includes so many advanced security controls that previously required on-premises appliances”
Sep 2, 2025
Minimal depth. ~2-minute read. Summarizes content in 6 succinct points. Good for a quick overview, but lacks nuance. Good. Factually correct points about included M365 features. No incorrect statements, just not deeply detailed. Very high. Focused on the SMB’s decision about firewalls and alternatives. Clearly addresses cost considerations. Excellent clarity. Simple numbered Q&A format with short sentences. Easy for any reader to understand quickly. No new insights, but effectively reiterates key conclusions and briefly mentions exceptions when a hardware firewall is needed.
M365 Copilot (GPT)
“Why the perimeter is no longer the control that matters most”
Sep 4, 2025
High depth. ~8-minute read with both executive summary and detailed blueprint. Covers endpoint, identity, and advanced optional services. Very high. Accurately reflects current Microsoft features and security settings. Recommendations align with best practices. High. Thoroughly addresses SMB needs and scenarios where premium firewalls are still justified (e.g., VPN, regulatory). Well-structured. Combines tl;dr summary with detailed sections. Subheadings and bolded key points improve readability. Cutting-edge advice. Introduces Microsoft Entra Global Secure Access (SSE) as a cloud-based alternative to firewalls. Provides a full implementation plan and rollout timeline.

Table Legend: M365 = Microsoft 365 Business Premium; NGFW = Next-Generation Firewall; EDR = Endpoint Detection & Response; ASR = Attack Surface Reduction; SSE = Security Service Edge (cloud-delivered network security).


Ranking of Articles by Value to SMB Owners

Finally, here is a ranked list of the seven articles (from most to least valuable) for a small business owner seeking guidance on M365 Business Premium vs hardware firewalls:

1. M365 Copilot (GPT) – “Why the perimeter is no longer the control that matters most”Top pick: This article provides the best all-around value. It gives a clear initial answer for quick understanding and then backs it up with a comprehensive plan. An SMB owner gets the immediate recommendation (skip the pricey firewall, leverage M365) in plain language[1], and their IT team gets a detailed roadmap to implement that strategy[1][1]. It’s up-to-date (even mentioning new Microsoft solutions) and covers “when you still might need a firewall” caveats. This dual approach of brevity + depth, and its forward-looking insights, make it extremely useful.

2. Copilot Researcher – “Security Without the High‑Priced Firewall: M365 vs Traditional Firewalls”Runner-up: A deep dive with executive summary that nails the question from both managerial and technical perspectives. For an SMB owner, the Executive Summary and conclusion clearly state the recommendation and rationale[2]. If more convincing is needed, the body provides a wealth of detail (feature comparisons, cost considerations, real-world scenarios) to support the decision. It’s essentially a mini research report advocating for M365’s security, which can be persuasive for stakeholders who want all the evidence. The only downside is length – not everyone will read 20 minutes – but the clarity of its introductory and closing sections ensures the main message is delivered even on a skim.

3. Deepseek (Deep Research) – “How M365 redefines the need for expensive hardware”Highly valuable: This short article is laser-focused on SMB benefits and cost-effectiveness. It articulates the core argument in simple terms (why buy “a fortress to protect an empty castle”?)[4] that any decision-maker can grasp. The inclusion of a cost comparison table is a standout feature, directly showing what you pay for in a firewall versus what you get with Business Premium[4]. For a time-pressed small business owner, this piece provides quick clarity and appeals to the practical mindset (security outcome vs cost). It lacks the extensive detail of others, but as a decision tool, it hits the bulls-eye succinctly.

4. Gemini (Deep Research) – “Cybersecurity for the Modern SMB: A Strategic Analysis…”Valuable for thorough strategy: This article offers a comprehensive strategic perspective that can be very convincing to a thoughtful SMB owner or an IT consultant advising one. Its discussion of TCO (total cost of ownership) and compliance is directly relevant to business considerations[3][3]. It effectively says: not only is the cloud approach effective, it’s also more economical and aligned to modern work – except in specific regulated cases. An owner reading this gets a full understanding of “why” the investment should shift. The formality and length keep it just shy of the top three; it’s best for those willing to invest time or for use in making a board-level case. In terms of content value, it’s excellent – just a bit dense.

5. ChatGPT (Deep Research) – “M365 Business Premium vs. Hardware Firewalls for SMBs”Solid and straightforward: This article is a well-rounded explainer that covers both technical and business points in a relatively brief format. It clearly enumerates the security features of M365 Business Premium and directly correlates them to the functions of a firewall, coming to the conclusion that a high-end firewall is largely redundant[5]. It’s written in an accessible way and includes a specific Cost vs. Benefit discussion[5] that resonates with business owners. While it doesn’t have the structured polish of some others, it is likely to leave an SMB reader convinced and with a good basic understanding of what Microsoft 365 offers. It ranks slightly below the more specialized or depth-intensive articles above simply because it doesn’t have a flashy unique element (like a table or step-by-step plan), but it certainly does the job well.

6. Copilot Studio (with GPT-5) – “Why Business Premium can replace most perimeter security…”Great for implementation, slightly less for pure decision-making: This piece is extremely useful if the SMB owner has an IT background or an IT admin to interpret it. It essentially provides the “how” after the “why,” including a detailed checklist for configuration[6][6]. Its upfront summary does answer the main question clearly (“a high-priced UTM is rarely cost-effective…”[6]), so the owner gets the recommendation. However, much of the content is technical guidance (Intune policies, ASR rules) that a non-technical owner might not use directly. Thus, its overall value to the owner alone is a bit lower, but it’s a fantastic resource to hand to their IT person once the decision is leaning that way. In a sense, it’s slightly niche in audience compared to the more narrative arguments higher on this list.

7. ChatGPT (Standard) – “M365 Business Premium includes so many advanced security controls…”Quickest answer, but least depth: This ultra-brief article delivers a straightforward verdict which is certainly valuable – an SMB owner could read the entire post in two minutes and walk away with the gist (use M365, don’t overspend on firewall)[1]. It’s ranked last not because it’s “bad” – on the contrary, it’s clear and correct – but because it provides the least new information or justification. A cautious decision-maker might find it too thin, possibly wanting a bit more “why” or real examples to be fully convinced. Essentially, it’s a summary of what others explained in detail. It’s very useful if the owner already had a hunch and just wanted confirmation. As a standalone persuasive article, it’s just okay. Think of it as a cheat-sheet: great for recall, but one might still look to the above articles (especially #1–#3) for a richer understanding before making a final decision.


Conclusion: All seven AI-generated articles agree on the main point – Microsoft 365 Business Premium’s security suite can replace most functions of a dedicated SMB firewall, except a basic router/firewall for fundamental network needs. They differ in presentation: some cater to those who want a quick answer, others to those who want a deep justification or a technical gameplan. For a small business owner, the most actionable and convincing pieces are the ones that blend clarity, business reasoning, and actionable guidance (as reflected in our top-ranked choices). By reviewing the comparison above, an SMB decision-maker can confidently choose which resource format suits their needs – or better yet, use the collective insights of all these perspectives to make an informed decision about investing in cloud security vs. on-premises firewalls.

References

[1] September 2025 – CIAOPS

[2] Security Without the High‑Priced Firewall: M365 Business Premium vs …

[3] Cybersecurity for the Modern SMB: A Strategic Analysis of M365 Business …

[4] How M365 redefines the need for expensive hardware – CIAOPS

[5] Microsoft 365 Business Premium vs. Hardware Firewalls for SMBs

[6] Why Business Premium can replace most perimeter security for typical …

An analysis of how AI services vary

bp1

Recently, I detailed how I used a variety of Ai services with the same prompt to compare the generated results. You can read that initial article here:

Testing the differences between AI services

The next step in this journey is now to start comparing them. For that I have used Microsoft 365 Copilot with GPT5 enabled. Here’s a concise summary of the key differences across the seven CIAOPS posts:


1. Audience & Tone
  • Aug 28 (ChatGPT DR): Short, practical overview for SMB IT/MSPs.
  • Aug 29 (Copilot Researcher): Formal, report‑style for CIO/CFO decision‑makers.
  • Aug 30 (Copilot Studio GPT‑5): Practitioner‑centric with actionable steps.
  • Aug 31 (Deepseek DR): Conceptual, explains “de‑perimeterization.”
  • Sep 1 (Gemini DR): Strategic, cost‑focused for board/C‑suite.
  • Sep 2 (ChatGPT): Feature‑inventory framing for non‑technical buyers.
  • Sep 3 (M365 Copilot GPT): Visionary/philosophical capstone on Zero Trust.

2. Unique Angle
  • Aug 28: Emphasizes App Proxy and host firewall as VPN/WAF alternatives.
  • Aug 29: Compares traditional firewall roles vs. modern limits; cost lens.
  • Aug 30: Provides a hardening checklist and “when you still need a firewall.”
  • Aug 31: Maps NGFW functions to M365 features; identity/data as perimeter.
  • Sep 1: Argues high‑end firewalls are financially inefficient for SMBs.
  • Sep 2: Highlights advanced security controls now built into M365 BP.
  • Sep 3: States perimeter is no longer the main control—identity/device/app is.

3. Depth & Practicality
  • Most actionable: Aug 30 (step‑by‑step baseline + decision criteria).
  • Most strategic: Sep 1 and Aug 29 (budget and governance framing).
  • Most conceptual: Aug 31 and Sep 3 (Zero Trust philosophy).
  • Most feature‑focused: Sep 2 (inventory of built‑in controls).

Here’s a comparison table summarizing the key differences across the seven CIAOPS posts

 
Post & Date Audience & Tone Unique Angle Depth & Practicality
Aug 28 – ChatGPT (Deep Research) SMB IT / MSPs; concise App Proxy & host firewall as VPN/WAF alternatives Moderate detail; quick read
Aug 29 – Copilot Researcher CIO/CFO; formal report Traditional firewall roles vs. modern limits; cost analysis High-level strategy; structured
Aug 30 – Copilot Studio (GPT-5) Admins/MSPs; hands-on Hardening checklist + “when you still need a firewall” Most actionable; step-by-step
Aug 31 – Deepseek (Deep Research) SMB leaders; conceptual Identity/data as the new perimeter; function mapping Conceptual depth; less prescriptive
Sep 1 – Gemini (Deep Research) Board/C-suite; strategic Financial inefficiency of high-end firewalls for SMBs Strategic recommendation
Sep 2 – ChatGPT Non-technical buyers Inventory of advanced security controls in M365 BP Feature-focused; overview
Sep 3 – M365 Copilot (GPT) Vision/strategy leaders “Perimeter is no longer the main control” (Zero Trust) Philosophical capstone

Testing the differences between AI services

bp1

If you are a regular reader of my blog, and I hope you are, you may have noticed a number of articles around a similar topic recently. A very common question these days is ‘What is the best AI service to use?’.

It turns out that the answer to that question is not straightforward. The reason is that AI models produce results ‘probabilistically’. This means, the answers are generated using probability based on the prompt that was made. Thus, even if you use exactly the same prompt, in exactly the same service, it is unlikely that you’ll get exactly the same answer, thanks to probability.

Thus, to provide some answers hopefully, I used the same prompt in a number of different AI tools and results can be found here:

Chatgpt (Deep Research) – https://blog.ciaops.com/2025/08/28/microsoft-365-business-premium-vs-hardware-firewalls-for-smbs/

Copilot Researcher – https://blog.ciaops.com/2025/08/29/security-without-the-high%e2%80%91priced-firewall-m365-business-premium-vs-traditional-firewalls-for-smbs/

Copilot Studio (with GPT5 reasoning) – https://blog.ciaops.com/2025/08/30/why-business-premium-can-replace-most-perimeter-security-for-typical-smbs/

Deepseek (Deep Research) – https://blog.ciaops.com/2025/08/31/how-m365-redefines-the-need-for-expensive-hardware/

Gemini (Deep Research) – https://blog.ciaops.com/2025/09/01/cybersecurity-for-the-modern-smb-a-strategic-analysis-of-m365-business-premium-vs-high-end-hardware-firewalls/

ChatGPT – https://blog.ciaops.com/2025/09/02/m365-business-premium-includes-so-many-advanced-security-controls-that-previously-required-on-premises-network-appliances/

M365 Copilot (GPT) – https://blog.ciaops.com/2025/09/03/why-the-perimeter-is-no-longer-the-control-that-matters-most/

Also, where possible, I used the same AI tool to create the image for the post, although not all tools provide this capability. I also used the ‘deep research’ option of the tool if it was available.

So, you can go and look at each results and judge the results for yourself and I’d love you to share what you think or the differences you have seen between different tools out there.

My plan going forward with these ‘baseline’ results is to use AI once again to compare and contrast them against each other to find the similarities and differences and report back.