Several plugins have been created to improve the default search, such as Relevanssi, which builds a separate index in the database to search so that results can be ordered by relevance. But this still only works if the text matches exactly. So, for example, “paternity leave” wouldn’t match a post containing “time off for new child”, even though they are about the same topic. This is where the power of AI can be helpful. AI can detect that these two phrases are related by meaning, or semantics, even though they don’t contain the same words. Let’s take a look at how we can use AI to get a better semantic search experience in WordPress.
Embeddings
We can distill the meaning of some text, such as a WordPress post, by generating its “embedding.” From OpenAI, “An embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness. Small distances suggest high relatedness and large distances suggest low relatedness.” Think of an XY plot with arrows going out from the origin.
Source: https://nexla.com/ai-infrastructure/vector-embedding/
The apple and banana vectors are close to each other, but Apple the company is further away because it’s not related to literal fruit.
With semantic embeddings, the vectors are not just two dimensional, they are often hundreds of dimensions. This is impossible for us to visualize, but in terms of math, it doesn’t matter how many dimensions the vectors are, the same vector math can be used to calculate how close the vectors are.
Embeddings in WordPress
So how do we use these embeddings in WordPress search? The general idea is:
- Pre-generate the embeddings (or vectors) for all of your posts
- When a search query comes in generate its vector
- Find the post vectors near the query vector
To get more specific, we can generate the embeddings using the OpenAI embeddings API, and store them in the postmeta table, then use the same API to get the embedding for the search query, and then order the results by how closely the vectors align (see limitations below for this step). In fact, this is exactly what the WordPress AI Search plugin does, and it works quite well.
We now have a search system in WordPress that finds posts based on the text meaning, rather than an exact text match.
Limitations
The primary limitation with the method described above is WordPress’s MySQL database. WordPress uses the MySQL database, which does not support vector operations until version 9. Google Cloud SQL also supports vector search. Unless we want to use one of these other solutions, we have to load all the vectors from the database into memory, and run the comparison using PHP. This is fine for a reasonable number of posts, about 500, but this is not really what PHP is meant for, so going beyond that would lead to site instability and performance issues on standard hosting.
Alternative Semantic Search Tools
An alternative to doing the vector comparisons ourselves would be to use a third-party service that offers a vector database which can run the vector comparison natively in the database. This could find results in milliseconds, even with thousands of posts, instead of 1-5 seconds with PHP with only hundreds of posts.
- ExpertRec – $49/mo – This appears to be a decent option for a reasonable price.
- WP Engine Smart Search – Starts at $140/mo (but you’ll need to be on premium/custom plan). This would probably be the most seamless to integrate if you were already on a premium WP Engine hosting plan.
- Algolia AI Search – Only available on top-level “Elevate” plan for which pricing is not public. I’d only explore this option if ExpertRec wasn’t good enough for you.
Conclusions
Semantic search can provide a better user experience than the default WordPress search. If you have a small site, try out the AI Search plugin. If you have a big site, try out a service that offers a vector database.
