How Good is OpenAI at Understanding Content?
ChatGPT from OpenAI has only been available for just over a year.
During that year, OpenAI has got people really excited about artificial intelligence tools. OpenAI is impressive with its ability to create content. It can produce blog posts, write school essays, and even tackle advanced exams.
Recently, we added a new feature to the TaxoPress plugin called “TaxoPress AI“. This works with OpenAI and helps you find the best Tags and Categories for your content.
TaxoPress AI uses OpenAI to scan your content and understand the meaning. In this post, I’m going to explore just how well OpenAI understands different types of content. Can you trust OpenAI to scan your content and add useful terms?
Table of Contents
OpenAI and Different Types of Content
Previously, we ran some tests using OpenAI on WordPress posts in Portuguese, Spanish, Turkish and many other languages.
The results showed that OpenAI is a wonderful tool, but its understanding of language is inconsistent. It does brilliantly with some languages, and makes good guesses with some other languages, but sometimes it fails completely.
TaxoPress users have been testing TaxoPress AI with lots of languages, and also different types of content. We’ve seen them using the OpenAI integration to analyze news articles, reviews, products, technical guides and more. So we ran a new set of tests to see how good OpenAI is at understanding different types of content. I ran tests by copying the text into WordPress and scanning them with the OpenAI scanner in TaxoPress AI.
How Well Does OpenAI Understand News Articles?
I first tried OpenAI on news articles, starting with an article about the Australian Open tennis final, written by the BBC. This is a very standard sports article about famous people at a popular event. OpenAI has probably been trained on thousands of similar articles. OpenAI produced 15 suggested tags and they were all accurate. I’d be happy to add OpenAI’s suggestions to the post.
Next, I looked another news article. This was a Bloomberg story about Intel. Again the suggested tags from OpenAI were excellent and relevant.
My third test was a movie review from The Guardian. OpenAI was outstanding again. The suggested tags contain the main actors and topics for the movie. I think it’s clear that OpenAI has a very good understanding of news articles.
How Well Does OpenAI Understand eCommerce?
Next, I looked at eCommerce products. I checked a boat review which contained a lot of technical details. OpenAI did OK in this test. It highlighted the key topics of the review, but also included some less relevant tags such as “hull design” and “powerboat wakes”.
Continuing the eCommerce tests, I analyzed a bike review. These results were also OK, but not outstanding. The first tags are the type of bike and brand name. But afterwards the tags, such as “Price”, “Weight” and “Brand”, are less useful and related to the content.
My third eCommerce test was an Amazon product which contained a lot of technical details, and also some typos. OpenAI didn’t do very well here and the IBM Watson results were significantly better. OpenAI failed to identify the brand name or even the key topic, which was “phone case”.
How Well Does OpenAI Understand Other Content Types?
My next group of tests focused on other types of popular content. I scanned a recipe for making salad and the results were excellent. I presume that OpenAI has been trained on a huge number of food blogs.
Next, I checked a tech-heavy article from Google’s research blog. This should be easy for OpenAI because it was an article about Artificial Intelligence. However, these results were mixed. You’ll find the main topic of the article on Mobile Diffusion but also some irrelevant tags, such as “sub-second” and “$1”,
My final test was a science research paper on prostate cancer. With the exception of a couple of extra long tags about “Pten knockout mice”, these suggested terms were excellent.
Summary of the OpenAI results
OpenAI is a wonderful tool, and its understanding of content is generally very good. OpenAI produces excellent results on news articles and other subjects where there’s a lot of online content. It does struggle a little more with subjects that are less common or more technical.
TaxoPress currently has integrations with four different AI services: OpenAI and IBM/Watson, plus also Dandelion and LSEG/Refinitiv. Each service has different pros and cons. We recommend testing them to see which is the best choice for your site.
In case you’re wondering, let me finish by showing you OpenAI’s analysis of this article you’ve finished reading:
I wrote a GPT for my site to find tags. Here are the result for your article: Steve Burge, February, Year, Artificial Intelligence, Blog Post, School, Exam, TaxoPress, Plugin, Tag, Category, WordPress, Portuguese, Spanish, Turkish, Language, News Article, Review, Product, Technical Guide, Australian Open, Tennis, Final, BBC, Intel, Bloomberg, Movie, Guardian, ECommerce, Boat, Technical Detail, Bike, Amazon, Phone Case, Recipe, Salad, Google, Research, Mobile Diffusion, Scientific Paper, Prostate Cancer, Pten Knockout Mouse, IBM Watson, Dandelion, LSEG, Refinitiv,
I don’t think it’s better, but I prefer to get too many results as it’s easy to delete some of them.
Thanks Miriam. We aim to add options in the next big TaxoPress release to give you more control over how many terms are suggested for each post.