AI In Performance Reviews Didn’t Matter—But the Culture Signal Will. Here’s What That Means for SEO, Content, and Execution.
Shopify’s CEO says peer ratings on AI use in reviews “didn’t matter”—but the question itself shaped behavior. For SMEs and agencies, the lesson isn’t to grade “AI adoption.” It’s to operationalize AI with clear quality gates, measurable outcomes, and an execution system that prevents slop and ships improvements.
Shopify’s CEO recently said something that should make every founder, marketing lead, and agency owner stop and think: the peer rating question about AI usage in performance reviews “didn’t matter.” Not because AI doesn’t matter—but because the question was never the point. The point was the cultural signal: this is what the company values, this is how work gets done here, and this is the direction of travel.
That comment comes from reporting by Search Engine Journal on remarks made by Shopify CEO Tobi Lütke about an AI-related performance review question and the broader expectations Shopify set in 2025 around “reflexive AI usage.” You can read the original coverage here: Shopify CEO: Peer Ratings On AI Use ‘Didn’t Matter’ (Search Engine Journal).
For most businesses, the headline isn’t “Should we ask employees how they use AI?” The headline is: if AI becomes normal at work, you must redesign how you produce, verify, publish, and improve your website—because AI changes the speed and volume of output, which changes the risk profile of your brand.
I’m Marius Dosinescu, and at AYSA we spend our time on the messy middle: turning strategy into site changes that actually ship. AI has accelerated ideation, drafting, and analysis. It has not magically solved execution, quality control, or accountability. If anything, it has made those gaps more expensive.
Concise summary
Shopify’s “AI usage” review prompt is a sharp example of a broader truth: adoption metrics are a weak proxy for outcomes. Businesses that win in AI Search won’t be those who “use AI the most.” They’ll be the ones who:
- Define what “good” looks like (quality, trust, usefulness, brand voice).
- Set quality gates to prevent unchecked AI output (“slop grenades”).
- Measure outcomes that matter (qualified traffic, leads/sales, visibility in AI-driven results).
- Build an execution system that monitors, proposes, gets approvals, and ships changes continuously.
Key takeaways
- Culture signals beat survey questions. If leadership wants AI-native behavior, workflows and incentives must follow.
- “AI adoption” is not a KPI. The KPI is business impact, plus quality and compliance outcomes.
- Unchecked AI output is an operational risk. A “slop grenade” isn’t just embarrassing—it creates rework, legal exposure, and brand erosion.
- Search is shifting toward AI answers. That increases the premium on clear entity signals, structured information, and content that is verifiably accurate and genuinely helpful.
- Execution is the bottleneck. The companies that improve fastest will be those with Monitoring + Approved Execution, not those with the flashiest prompts.
Table of contents
- What changed: from “AI question in reviews” to “AI-native culture”
- The real lesson from Shopify: questions don’t matter—systems do
- Why “reflexive AI usage” became a baseline (and why that’s not the point)
- The “slop grenade” problem: AI output handed off unread
- Why adoption metrics fail (and what to measure instead)
- AI search reality check: AEO/GEO and why your website is now a dataset
- How to operationalize AI safely: governance, QA, and workflow design
- A concrete SME scenario: a local clinic + an ecommerce store
- What agencies should rethink: deliverables, pricing, and accountability
- Where AYSA fits: monitor → prepare → approve → execute
- What to do next: a practical 30-day plan
- Sources and further reading
What changed: from “AI question in reviews” to “AI-native culture”
The piece of news that matters isn’t a single performance review item. It’s the pattern behind it:
- Shopify signaled that “reflexive AI usage” should be baseline behavior.
- They experimented with a peer review prompt about how colleagues show up with AI prototypes.
- The CEO later said the answers “didn’t matter,” and that the presence of the question can create “a reputation to live up to.”
Whether you love or hate that management approach, it highlights a pragmatic lever: if you want behavior change, you need social proof, defaults, and systems—not just policy.
In marketing and SEO, we’ve been here before. Organizations tried to “make people more data-driven” with dashboards and KPIs. But unless the meeting cadence, incentives, and decision rights changed, the dashboards became wallpaper.
AI is following the same path:
- Step 1: People adopt tools.
- Step 2: Output volume increases.
- Step 3: Quality and governance become the limiting factors.
- Step 4: The winners are the teams who can ship improvements continuously, without breaking trust.
The real lesson from Shopify: questions don’t matter—systems do
If you’re a business leader, the idea that “the answers didn’t matter” can sound reckless. But in context, it points to a deeper truth: you can’t audit your way into a modern workflow. You design a workflow, you reinforce it culturally, and then you measure outcomes.
Here are two approaches many companies are currently choosing between:
Approach A: rate AI usage
- “Did you use AI this quarter?”
- “How often do you use AI tools?”
- “Did you try an AI agent?”
These questions are easy to ask and easy to game. They also confuse means with ends.
Approach B: operationalize AI outcomes
- What outcomes are we trying to improve (cycle time, quality, conversion, support deflection)?
- What are the non-negotiables (brand voice, compliance, citations, human review)?
- Where does AI sit in the workflow (drafting, analysis, testing, monitoring)?
- Who approves what, and what gets logged?
Approach B is harder. It’s also how you avoid “slop grenades,” protect trust, and build a repeatable advantage.
For websites and search growth, this is critical because AI lets teams publish more. If your editorial and QA systems are weak, you can now publish more wrong things faster. Volume will not save you. It will sink you.
Why “reflexive AI usage” became a baseline (and why that’s not the point)
The phrase “reflexive AI usage” resonates because it describes what’s already happening in many workplaces: people increasingly treat AI like spellcheck, search, or a calculator—an ambient tool rather than a special project.
But for SMEs, “reflexive AI” can become a trap if it’s interpreted as “always use AI” instead of “use AI where it helps, then verify and ship responsibly.”
In practical terms, here’s where reflexive AI is genuinely useful for SEO and content teams:
- Research acceleration: faster outlines, question expansion, topic clustering (followed by human validation).
- Drafting and rewriting: producing first drafts and variants for pages you already know you need.
- Internal enablement: transforming product knowledge into drafts for FAQs, help docs, and internal SOPs.
- Pattern detection: summarizing recurring issues from audits or performance trends (followed by data checks).
And here’s where reflexive AI can harm you if you don’t have guardrails:
- Medical/legal/financial claims drafted without citations or review.
- Thin “SEO pages” generated at scale, creating duplication and brand erosion.
- Overconfident technical recommendations implemented without staging, QA, and rollback plans.
So the point isn’t “use AI.” The point is: build a business system that uses AI to move faster without degrading truth, trust, and customer experience.
The “slop grenade” problem: AI output handed off unread
Search Engine Journal’s reporting references Lütke’s earlier remark about “slop grenades”—a term Shopify staff use for AI output that gets passed to a colleague without being read. That’s not just a funny internal phrase. It’s a diagnostic label for a failure mode that’s showing up everywhere:
- Marketing hands a draft to compliance without verifying claims.
- SEO hands a list of “recommended changes” to dev without validating intent or risk.
- An agency sends AI-written content to a client for approval without reading it carefully.
When AI increases speed, humans instinctively shift effort away from review—especially under deadline pressure. That’s exactly when review becomes more important.
Why slop happens
- Ambiguous ownership: no one is accountable for accuracy once AI is involved.
- Rewarding velocity: teams celebrate shipping, not verifying.
- Broken handoffs: drafts move between roles without a shared checklist.
- Tooling mismatch: AI generates content faster than CMS workflows can handle safely.
Why slop is expensive
- Brand cost: customers spot generic, inaccurate, or contradictory content quickly.
- Operational cost: editors, lawyers, or customer support end up cleaning up messes.
- Search cost: low-quality pages can cannibalize better pages and confuse relevance.
- Trust cost: once internal teams lose confidence in content, everything slows down.
This is why “AI use” is the wrong thing to rate. You should be rating (and reinforcing) behaviors like verification, clarity, documentation, and customer usefulness.
Why adoption metrics fail (and what to measure instead)
Leaders love adoption metrics because they’re tidy: percentage of staff using tools, number of prompts, number of AI drafts created. But adoption is not value. Adoption is activity.
If you’re running SEO and content for a business, what you need are metrics that connect AI-assisted work to outcomes without incentivizing spammy volume.
Better metrics than “did you use AI?”
Consider measuring these instead (pick what fits your model):
- Cycle time to publish (with QA): how long from brief to live, including review steps.
- Update cadence: are key revenue pages improved continuously, or left to rot?
- Content accuracy incidents: number of corrections, customer complaints, or compliance flags.
- Conversion impact: form fills, calls, purchases, demo requests (measured carefully and honestly).
- Search visibility for priority intents: not vanity rankings, but whether you show up where buyers look.
- AI search inclusion: evidence that your brand/pages are being used or cited in AI-style answers (where measurable).
The theme: measure what matters to the business, and measure the health of the system (quality, errors, rework), not just output volume.
What to avoid
- Counting pages published as a primary KPI.
- Rewarding “shipping at all costs” without quality gates.
- Setting goals like “everyone must use AI daily” (creates performative behavior).
AI search reality check: AEO/GEO and why your website is now a dataset
Even if you don’t run a tech company, you’re now operating in a search environment shaped by AI systems. The practical implication for SMEs is simple: your website isn’t just a brochure—it’s training data and reference material for machines that summarize answers.
That’s why we talk about AEO and GEO:
- AEO (Answer Engine Optimization): structuring and writing content so it can be reliably used in direct answers.
- GEO (Generative Engine Optimization): increasing the likelihood that generative systems choose your brand as a trusted source.
Regardless of the acronym you prefer, the mechanics are converging: systems try to extract meaning, identify entities, and present synthesized responses. Vague, contradictory, or generic content performs poorly in that world. Clear, specific, consistently updated content performs better.
At AYSA, we treat this as a visibility and execution problem, not a “prompting” problem. Your advantage comes from:
- Having the right pages (and the right structure) for your buyers’ questions.
- Keeping those pages accurate and updated.
- Making changes continuously based on monitoring signals.
If you’re new to these concepts, start here: AYSA AI Search Visibility.
How to operationalize AI safely: governance, QA, and workflow design
The most important shift leaders need to make is to stop treating AI as a “tool rollout” and start treating it as a workflow redesign. That requires governance—not in the bureaucratic sense, but in the operational sense: who does what, when, and with what checks.
1) Define your non-negotiables
Before you scale AI-assisted work, decide what can’t be compromised. For most SMEs, the list includes:
- Accuracy of claims (especially for health, finance, legal, safety).
- Brand voice and positioning (no generic filler, no contradictions).
- Customer usefulness (content must answer real questions, not just “rank”).
- Compliance and privacy (don’t leak sensitive info into tools or pages).
2) Build quality gates (small, enforceable checklists)
A “slop grenade” happens when there is no shared definition of “done.” Solve that with gates that are quick and clear. Example gates for content:
- Claim check: any factual claim must be validated or removed.
- Source check: if a claim depends on external facts, cite reputable sources (or don’t state it).
- Internal alignment: does it match your pricing, services, and policies elsewhere on the site?
- Intent match: does the page answer what the searcher actually wants, in plain language?
- Approval step: who is the accountable editor/owner?
Example gates for technical changes:
- Staging review before production.
- Rollback plan.
- Measurement plan: what will improve if we do this?
3) Log decisions and changes
When AI increases speed, you need an audit trail. Not because you want red tape, but because you want to know:
- What changed?
- Why did it change?
- Who approved it?
- What happened afterward?
This is a core principle behind AYSA’s approach: monitor what matters, propose changes, request approval, then execute only what’s accepted. It’s the difference between “AI as chaos” and “AI as compounding advantage.” Learn more here: AYSA Monitoring.
4) Train your team on examples, not slogans
“Use AI more” is a slogan. “Here’s what good looks like” is training. Create a small internal library:
- 2–3 examples of excellent pages in your niche and why they work.
- 2–3 examples of bad AI content and what made it bad (generic, inaccurate, contradictory).
- Approved outlines and templates for key page types (service page, category page, FAQ, comparison page).
A concrete SME scenario: a local clinic + an ecommerce store
Let’s make this real with two businesses that don’t have time for AI theater.
Scenario 1: A local clinic that wants more booked appointments
A clinic hears “AI search is changing everything” and tells the marketing assistant to “use AI to create more content.” The assistant produces 20 blog posts in two weeks. The posts sound professional but contain subtle inaccuracies about treatment timelines and eligibility, and they repeat generic advice found everywhere.
What happens next?
- The front desk gets calls from patients confused by what they read.
- A provider complains that the content doesn’t reflect how the clinic actually practices.
- The clinic pulls the posts down, embarrassed—and now “AI content” becomes taboo internally.
The failure wasn’t AI. The failure was the workflow: no medical review gate, no accuracy checklist, no clear priority pages, and no measurement plan tied to appointment bookings.
A better approach:
- Start with 5 high-intent service pages (the ones that convert).
- Use AI to draft updates and FAQs, but require provider review for claims.
- Monitor performance and iterate monthly.
Scenario 2: An ecommerce store trying to defend organic revenue
An ecommerce brand uses AI to expand category descriptions and product FAQs. The content is “fine,” but it’s not grounded in the brand’s real differentiators, and the internal linking becomes inconsistent. The site grows in page count but loses clarity.
A better approach:
- Define a category page standard: who it’s for, what it should answer, what proof it should provide.
- Use AI to generate first drafts, then inject real differentiators: shipping terms, materials, sizing guidance, returns, care instructions.
- Use a monitoring system to flag pages with declining performance or engagement, then update systematically.
These are not hypotheticals. This is what “AI at work” looks like in normal businesses. The winners will build repeatable, safe publishing and optimization loops.
What agencies should rethink: deliverables, pricing, and accountability
Agencies are under unique pressure: clients now assume AI makes everything faster and cheaper. That’s partially true for drafting. It’s rarely true for outcomes—because outcomes depend on strategy, prioritization, and execution quality.
Stop selling outputs; sell improvement loops
If your deliverable is “X blog posts per month,” AI will compress your perceived value. The more durable offer is:
- A monitored backlog of opportunities.
- Prioritized changes tied to business goals.
- Approved execution (so changes actually ship).
- Learning cycles that improve what works and kill what doesn’t.
This is why AYSA’s model matters for agencies too: it turns “recommendations” into “approved changes” into “executed updates,” with traceability. If you’re building a modern agency stack, explore: AYSA AI SEO Tools.
Make QA part of the product
In an AI-heavy world, QA isn’t overhead—it’s a differentiator. Agencies that formalize quality gates will outperform those who race to publish.
Examples of QA you can productize:
- Claim verification checklists for YMYL-adjacent industries.
- Brand voice and positioning review gates.
- Internal linking and cannibalization checks before publishing.
- Change logs and approval trails clients can audit.
Update reporting: adoption metrics are irrelevant
Clients don’t need to know how often you used AI. They need to know:
- What changed on the site.
- Why you changed it (the hypothesis).
- What the outcome was (or what you learned).
If your report contains “we generated 40 AI drafts,” you’ve already lost the narrative.
Where AYSA fits: monitor → prepare → approve → execute
The Shopify story highlights a reality I see constantly: most organizations don’t fail because they lack ideas. They fail because they can’t consistently turn ideas into shipped improvements.
AYSA is built for that gap. The model is straightforward:
- Monitor: keep a continuous pulse on site health, visibility, and opportunities. (See monitoring)
- Prepare: generate recommended changes in a structured, reviewable way—content updates, technical fixes, internal linking improvements, schema guidance, and more.
- Ask for approval: humans stay accountable. Nothing “auto-publishes” into your brand without oversight.
- Execute accepted changes: ship improvements so your site compounds over time.
That last step is the one most businesses miss. They live in audits and roadmaps. They drown in tickets. They talk about AI, but the website stays stale.
If you want the big picture view of how we think about this shift, start with our blog and resources: AYSA Blog.
And if you’re evaluating whether an execution system is economically viable for your business, pricing matters; you don’t want a “platform” that requires a second full-time hire to operate: AYSA Pricing.
What to do next: a practical 30-day plan
You don’t need a new performance review question. You need a workable operating system for AI-assisted growth.
Days 1–7: Decide what “good” means
- Pick 5–10 “money pages” (service pages, category pages, lead gen pages) that matter most.
- Write down your non-negotiables: accuracy, compliance, brand voice, customer usefulness.
- Create a lightweight QA checklist for AI-assisted drafts (one page, not a manifesto).
Days 8–15: Build a monitored backlog
- Inventory your key pages: which ones are outdated, thin, or unclear?
- Define a simple prioritization framework: impact, effort, risk.
- Set up monitoring so you don’t rely on vibes (traffic drops, page performance changes, new issues).
Days 16–23: Ship improvements with approvals
- Update 3–5 priority pages with AI-assisted drafts plus human review.
- Improve internal linking and FAQs where it helps users.
- Document what changed and why.
Days 24–30: Measure outcomes and tighten the system
- Review what improved (or didn’t): leads, sales, engagement, visibility for target queries.
- Look for quality signals: complaints, corrections, confusion, support tickets.
- Refine your checklist and approval workflow based on what went wrong.
What to do next (action list)
- Remove “AI adoption” from your KPIs and replace it with outcome + quality metrics.
- Create a “no slop grenades” rule with a required checklist for handoffs.
- Start with updating revenue-driving pages before scaling new content production.
- Implement monitoring so you can detect harm quickly and learn faster.
- Adopt approved execution so good ideas become shipped improvements safely.
Sources and further reading
- Search Engine Journal: Shopify CEO Says AI Performance Review Answers ‘Didn’t Matter’
- Search Engine Journal: AI Search coverage
- Search Engine Journal: SEO coverage
- Search Engine Journal: SEO News
- Search Engine Journal: Technical SEO
- AYSA: AI Search Visibility
- AYSA: AI SEO Tools
- AYSA: Monitoring
- AYSA: Pricing
- AYSA: Blog
Note on sourcing: The Shopify remarks referenced above are based on Search Engine Journal’s reporting of an interview and related comments. Where this editorial extends beyond the reported details, it is analysis and operational guidance drawn from common failure modes we see across SMEs and agencies.
Continue the AI search topic inside AYSA.
Use these pages to connect the article with AI SEO tools, AI visibility monitoring, AI Overviews and approved website execution.
Turn this topic into a website action plan.
Use these AYSA hubs to move from reading to technical fixes, AI visibility monitoring, research, glossary context and approval-first SEO execution.
