ChatGPT vs. Human in SEO Blog Marketing

During my Master's degree in Technology Entrepreneurship, I was building my own startup and constantly searching for cost-effective ways to promote it.

That's when I started experimenting with Large Language Models. I'd like to share my journey and findings of my scientific paper about comparing AI-generated with human-written blogs. Still you can download the full paper for free or the business white paper below.

My first encounters with LLMs

I first played around with GPT-2 and it was impressive for its time, but the outputs still felt rough and required heavy editing.

Then halfway through my thesis, GPT-3 dropped. Everything changed. The quality leap was massive and suddenly using AI for content creation felt like a real possibility. Which was also quite nice because it felt that my thesis was now more relevant than ever and I felt like an early adopter of this new technology.

I was wondering: Could AI-generated content actually compete with human-written content for SEO purposes? I decided to find out.

Goal

Beyond finishing my thesis, I wanted to push my startup's marketing efforts further. But as a student, you also have to frame your thesis with a proper research question and scientific methods.

So I tried combined both: scientifically evaluate the SEO performance of AI-generated blog posts versus human-written ones while building a useful tool I could actually use for my own marketing.

I'll walk you through my experiment setup, the results of it, break down the cost-benefit ratio, and show you the AWS architecture I used to deploy everything.

Experiment

I set up an experiment aimed to measure SEO KPIs from blog posts: impressions, clicks, and average position on search engine results pages.

Better performance in these KPIs increases the likelihood of acquiring more customers. I created 10 AI-generated blog posts using our own developed SEO-optimized ChatGPT wrapper and 10 human-written blog posts as a control group.

Both sets were indexed and ranked on Google for over four months to gather KPI data.

Results

Each day's KPIs were recorded for each blog post and aggregated into a dataset used for comparison.

The Wilcoxon-Mann-Whitney test was used to confirm the data's statistical significance, proving the results are not random and have statistical validity.

Findings reveal that human-written blog posts outperform AI-generated ones across all recorded KPIs.

Impressions: The number of times a website appears in search results, indicating how often users may have seen a blog post.

Clicks: The number of times users click on a blog post’s search result, reflecting how many are redirected to the blog from the search engine.

Average Position: The ranking position of a blog post on search results pages. A position closer to the top increases the likelihood of users seeing and clicking on it.

Cost-Benefit Ratio

Creating human-written blog posts requires effort, while AI generates posts much faster. To evaluate the economic viability of using AI for customer attraction, I calculated the cost-benefit ratio based on time and costs.

The calculation shows that AI is more economically efficient for achieving impressions and clicks.

AWS Architecture

To generate the AI-generated blogs, I built my own SEO-optimized ChatGPT wrapper as a Software-as-a-Service tool and deployed it on AWS. I kept the architecture simple and focused on development speed.

Recommendation & Conclusion

Human-written blog posts perform better in SEO, but AI-generated posts offer a better cost-benefit ratio.

A semi-automated approach combining advantages could create AI-generated blog posts regularly, while measure performance for monthly reports.

This data-driven approach can identify high-performing posts, which human authors can edit to further optimize SEO to archive best results.