Using Next.js, WebDevStudios Built a 1,000 Page Headless WordPress Website

Last spring, I started dabbling with Next.js and it didn’t take long for “dabbling” to turn into, “Whoa! This is awesome!” I found myself totally immersed. A big part of that immersion is the simplicity of Next.js. For me, it has the perfect amount of abstraction, somewhere between Create React App and Gatsby. You don’t have to know how to set up Webpack or routing, but you’re also not locked into an opinionated way of doing React things mixed with Node based JavaScript.

Next.js just blends into the background, so you can focus on building components and ship a JAMStack website in no time.

Background

Around that same time, I was rolling out our Gutenberg First initiative to help level-up our engineers and start shipping Gutenberg-based websites for our clients. JavaScript was on a lot of our engineers minds as well as potential clients. Sales leads were starting to come in with requirements around building headless WordPress. Admittedly, we did not have a solid process in place for building Headless WordPress websites.

Part of my role as Director of Engineering is to write those very processes; and in order to do that, I had to learn as much as I could about Next.js and WordPress. I had already made some contributions to the Next.js docs but noticed that they were building a WordPress Example so I offered to help. The team at Vercel was excited to collaborate, and a few weeks later, we shipped the official WordPress Example.

The official Next.js WordPress example is not very feature rich on purpose. It’s there to provide a low barrier of entry, so it wasn’t long after the official Next.js WordPress example shipped that we decided to fork it and started to add functionality. When our engineers had downtime, they were helping contribute to an alpha version of the WebDevStudios Next.js WordPress Starter.

We aimed to support advanced features like Menus, Custom Post Types, Gravity Forms, Custom 404 Pages, Yoast SEO, and more. Right in the middle of development, things got busy at WebDevStudios—so busy, that I was asked to jump in and lead some client projects, including a massive site rebuild with a headless frontend.

A Real Headless Project!

In August 2020, we began work on a huge project that involved moving a client from a traditional WordPress frontend to a decoupled one. The client’s website is very popular and their mission is to share educational content to educators and students across the United States. Their website was ranked on the first page of Google for several prominent keywords, and as such, received a large amount of traffic.

They were also managing over 1,000 pieces of educational, page-based content as blog posts (yikes). So, a total overhaul of the data architecture was needed. We paused work on our Next.js WordPress Starter, and took what we learned so far over to this new 1,000+ page Headless WordPress project.

The primary requirement was speed. The website had to have static content served from the JAMStack, but it also needed to dynamically update pages in the background without having to run a build. Other requirements included serving images from Amazon S3, 301 redirect management in WordPress, and over a dozen custom post types.

As Lead Engineer on the project, I was responsible for coming up with the technical stack and chose Next.js as the framework, since one of the key features of Next.js is Incremental Static Regeneration. This would help us meet the primary requirement. Other parts of the frontend stack included:

• Axios
• DayJS
• TailwindCSS
• Algolia
• ESLint, Stylelint, and Prettier
• Storybook
• Next Sitemaps
• Next Auth0
• Hosted on Vercel

The backend (WordPress) stack included:

• Advanced Custom Fields Pro
• CPTUI
• Formstack
• WP GraphQL (and several other addons)
• WP Offload Media
• WP Search with Algolia
• Yoast SEO Premium
• Vercel Deploy Hooks
• Hosted on Pantheon

In early September 2020, I assembled my team and away we went….

Challenges

Complex Data Structures

The previous implementation of educational content was poor at best. They were using posts, categories, and tags, along with a complex mixture of parent and child pages to manage relationships between those posts. The content team was mostly interns, and one of the challenges was onboarding them as content editors while trying to explain the relationship between everything.

After meeting with the migration team, we worked on restructuring their content into 17 different custom post types. This would make creating and managing content much easier for interns, since Lesson would be under “lesson,” a Resource would be a “resource,” and so on.

The challenge on the frontend would be querying all of these data types and their relationships via GraphQL.

Data Fetching

Because of the complex nature of the queries, one of the biggest challenges was server timeouts (status code 5xx errors) during builds.

This was in large part due to how the complex relationship between data in WordPress was structured, but the tooling was also a problem…Take a look at the screenshot below which was taken during a build. There were 1.25 million requests to WP GraphQL in one minute alone!

While Incremental Static Regeneration is amazing, it doesn’t come free! The whole “updating in the background” concept really means dozens of server(less) functions (spread over 1,000+ pages) querying WordPress and WP GraphQL constantly, which would literally take down the Pantheon server.

To help, we increased the Incremental Static Regeneration delay from one minute to 10 minutes. This update alone was a game changer. Additionally, we added the Axios Retry package, which also helped reduce 500 errors because retry requests are based on exponential backoff. Finally, we activated Redis caching and verified that GraphQL queries were being saved.

Was Axios the best choice for this project? No. In hindsight, I should have recommended Apollo Client, since it supports in-memory caching of queries. Given the complexity of the data structure, this could have reduced server load and build issues right out of gate.

Dependency Hell

Because we were using two stacks (WordPress and Next.js), it meant double the dependency hell.

Like WordPress plugins, NPM packages can easily be installed to help solve problems. I mean, why build your own date formatting system, when you can install DayJS instead?

The trouble was maintaining both WordPress Plugins and NPM Packages. Dependency updates for both stacks had to be carefully tested, and sometimes the site would build and test perfectly on our locals only to break the development environment. In the end, much of my time was devoted to maintaining and troubleshooting dependencies.

Error Handling

We still haven’t fully cracked the code on this one, but Next.js simultaneously has some of the best and worst error handling we’ve ever seen. Normally during development, Next.js will throw an error in the terminal and it’s very clear what’s going on. It can even display links to error codes for more information. This is awesome. However, during builds, this sorta falls on its face.

To quote one of the engineers from the project team:

Tracking down bugs was a nightmare. Documentation for both (and WP GraphQL) was decent on some things, entirely lacking in others, and sometimes wholly misleading. A lot of my work I had to just figure out on my own with no direction or reference points from the creators of the particular tool/API.

In fact, as I was writing this blog post, the client reported a section of the website was 404-ing. I quickly spun up my local and ran a build. No errors in the terminal nor in the console on the browser. But when I’d visit the page route? The page would 404! Again, no errors. Super frustrating! It took about 30 minutes to discover that there was an issue with a GraphQL query and pushed up a hot-fix. How can a statically generated website build successfully, yet still 404?!

Successes

Even though there were some challenges, the positives definitely outweigh the negatives. Remember when I mentioned the client’s requirement for speed? Well, Next.js websites are fast, like really fast! As expected with JAMStack based frontends, we definitely met the primary requirement for speed! Check out the screenshots from GTmetrix and WebPageTest.org.

Frontend Tooling: Github, Vercel, and Linting

We leveraged ESLint, Stylelint, and Prettier to enforce coding standards and deal with code formatting. Additionally, we used Husky + Lint-Staged to double check staged on the pre-commit git hook. This meant during pull request reviews, we could focus on how the engineers built their feature, instead of squabbling over semi-colons and spacing.

In addition to linting, Vercel has really, really impressive integration with Github. Each pull request was given its own unique build and URL for testing. This was great! An engineer could work locally, then open a draft pull request (which would kick off a testing build) so they could code in a Vercel-powered environment.

Finally, Chromatic made it possible to do UX/UI reviews on React components at the pull request level too. This was incredibly helpful as a Lead Engineer doing reviews. In the screenshot below, you can see the changes highlighted in bright green (bottom right):

I wish we could use this frontend workflow on every project; it was an absolute pleasure to work with.

Core Web Vitals

Google announced a new set of metrics for Lighthouse and PageSpeed Insights coming this year. They are:

• Largest Contentful Paint (LCP)
• First Input Delay (FID)
• Cumulative Layout Shift (CLS)

These metrics inform the “search signals for page experience,” which is a fancy way of measuring the perceived experience of a web page. The bottom line is that your website needs to load fast, be quick to accept interactivity from the user, and be visually stable.

Using Vercel Analytics, you can see how amazing our Next.js website (and Vercel as a host) are measuring with Core Web Vitals over a seven-day period.

Meeting these requirements will ensure the client’s website will retain its high rankings on Google.

Storybook

The first four weeks of the project was dedicated to building out atomic design-based components in Storybook. At first glance, that probably seems like way too much time, but once we were done? Implementing components throughout the React codebase just worked.

As one of the frontend engineers mentioned, working on components in isolation via Storybook made his developer experience a pleasure. The best part is now the client (and future developers) have access to a library of over 242 components.

Site Map and 301 Redirects

I feel like this topic doesn’t get enough attention among the development community, but it’s really important. As seen in the screenshot (earlier in this article), the client’s website is ranked on the first page of Google for several keywords; and migrating from a traditional WordPress frontend to a decoupled frontend (plus all the other data restructuring) would mean different URLs, thus the need for a new sitemap and 301 redirects to preserve their rankings.

Using Yoast SEO’s Redirect Manager, the client was able to import over 1,000 URLs and then manage every aspect of where they wanted them to point. Since Next.js supports redirects, we wrote a Node-based script that queries the list of 301 redirects via GraphQL, and writes them to redirects file at build time, which is then imported into next.config.js.

We also leveraged an NPM package named Next Sitemap, which also runs at build time, and creates a standard sitemap.xml as well as robots.txt, which we re-submitted to Google Search Console.

As you can see below, there was an initial dip since Google needed to re-crawl 1,000+ pages, but it quickly recovered, and the new website is tracking back toward the average position and click-through ratio it had prior:

Algolia

Leveraging our WP Search With Algolia plugin, we were able to easily index all of the client’s content and custom fields and push those indices to Algolia. On the frontend, we relied on Algolia’s React InstantSearch widget to create a live search field. We also used Algolia’s faceted search widget to refine the results.

One of the team members, Darren Cooney, shared some tips and tricks for using the WP Search With Algolia plugin based on the lessons we learned on this project.

Wrap Up

We learned a lot on this project. Sure, it had its moments, but in the end? I’m really proud of the work the team did and the partnership we’ve formed with the client. The engineering team has leveled up and has an entirely new skillset; and the client is especially thrilled about keeping the content management system they love and have a blazing fast JAMStack frontend.

We took all the lessons we learned and poured them into our Next.js WordPress Starter, an open-source project which we plan to use on future Headless WordPress projects. We love open-source at WebDevStudios, and invite you to give it a star on Github.

If you’re looking for an experienced team to develop your next Headless WordPress website, check out our headless solutions and services. If you’re ready to get started, contact us today.