Near the end of 2019, I realized that this website had become a bit of a mess. Posts were scattered all over the place, categories were poorly labeled and poorly organized, and the permalinks had a mind of their own. The website was not set up well for either hosting or sharing my content, and it was making me reluctant to post new things. So I decided to fix it.
Over the past 12 hours I’ve taken the website offline and completely overhauled both the category taxonomy and the post permalinks, adding custom redirects to prevent 404 errors from the old URLs.
Let’s go over the changes!
All New Categories
When I started this website I had no clear roadmap for what I wanted it to become, or how I wanted to organize the content. Because of that I started adding categories willy-nilly as I thought they were needed. Over the past three years, this has created a massive taxonomic mess of both overly-broad and overly-specific categories with little direction for where to put any given post.
The categories list has been completely reworked. The previous 18 top-level categories have been reduced down to just 4: projects, tutorials, meta, and articles. In addition to this, I’ve added several new sub-categories under “Projects”, including a “Multi-Part Projects” category to organize all of the larger and more comprehensive projects.
Reworked: Projects
I say “reworked”, but even though this has been a menu item for years, the truth is that “Projects” was never actually a category! When I started this blog I didn’t know enough PHP and CSS to create a custom page for a subset of posts, so instead I created a static page and updated it by hand whenever I finished a project. It goes without saying that this was clunky, and often times projects just got left out in the cold until I remembered to update that page.
Now this is a dedicated category within the site’s taxonomy, and will update automatically whenever I publish a new “Projects” post. This category also includes a few sub-categories that are closely related to the previous top-level ones, such as “Custom Controllers” (formerly “AltCtrl”), “Props” (formerly “Propmaking”), and “Programs”. These new categories will also be added to the main navigation menu for ease of access.
Reworked: Tutorials
The original “Tutorials” category was created only a few months ago once I realized that I should collect some of my “How To” posts together so they were easier to find. Alongside my own DIY projects, creating easily accessible tutorials is going to be one of my focuses going forward. Not only does this type of content help others but it’s a good way to solidify my understanding of a topic and to refresh my knowledge down the line. As with the new “Projects” sub-categories, this category is also going to be added to the navigation menu.
Meta (Unchanged)
This is the same old ‘meta’ category as before: posts about the website for the sake of the website (like this one!). This is a DIY website and the website itself is an ever-evolving DIY project – so this is staying as is. Posts in the “Meta” category tend to deal with minutiae and nitty-gritty, and will be filtered out of most of the dynamic feed sections like the landing page.
New: Articles
In WordPress, all posts must have some sort of category assigned to them. If posts aren’t assigned a category, they are put into the catch-all category “Uncategorized”. Catchy name, I know.
To take advantage of this quirky feature of the platform, I’m renaming this (previously unused) catch-all from “Uncategorized” to “Articles”! Now instead of creating a new sparse sub-category for each ill-fitting post, miscellaneous posts with no obvious category will just become generalized “Articles”. Works for me!
New: Multi-Part Projects (Sub-Category)
Many of the projects I work on are multi-faceted and complex, and require a lot of documentation in order to be thorough. Instead of putting all of this information into one lengthy and comprehensive post which is hard to read (and hard on the server!), I instead like to split things into well-organized, bite-sized chunks.
Previously each multi-part project had its own top-level category, which quickly gets unfeasible the more projects I complete. Instead, all of these multi-part projects are going to be collected under the new “Multi-Part Projects” category! The overview / summary posts are going to live directly in this category, and then each large-scale project will then have its own sub-category for all of the related posts.
To make this a little cleaner, I wrote some custom PHP functions to modify the category views for the “Projects” and “Multi-Part Projects” categories in order to exclude posts that are only listed in children of the “Multi-Part Projects” category. This way the “Projects” view will hide posts that are part of a larger project but not complete projects in-and-of themselves. This also limits the “Multi-Part Projects” category page to only show overviews, and not all of the posts all at once – hopefully making things both well organized and easy to find.
All New Permalinks
Next on the docket: ALL NEW PERMALINKS! The permalink structure for the website has been changed from /%category%/%postname%/ to just /%postname%/.
But… Why?
I’m glad you asked!
By default, WordPress uses ‘ugly’ permalinks, which identify posts by their internal ID number. This is confusing to the user and is bad for search engine optimization (SEO). Not long after I created the website I set up a substitute ‘pretty’ permalink structure that used the category as part of each post’s permalink: /%category%/%postname%/.
At the time this seemed like the best structure:
- “Pretty” Links: No ID. Easier to read, better for SEO.
- Category Included: Better for SEO, organization, and provides breadcrumbs for the user.
- No Date: Doesn’t prematurely age the content and allows evergreen content to thrive.
This has worked pretty well for the past few years, but there’s one critical issue with it that’s causing major problems.
The Category Conundrum
Remember how I said the website categories have become a mess? The permalink structure had a significant role in that fiasco.
Because I had no idea how to organize my content, I also had no idea which categories to effectively “marry” to each post. Instead, I started to design my categories around how I wanted the permalinks to look. And because these categories are reflected in the permalinks, I couldn’t change them after the fact without creating a redirect. So this was a semi-permanent decision for every piece of content I created.
For instance, one of the early projects I completed was a DIY Ambilight using WS2812B LEDs. This project was made up of 8 posts, and I thought it would look nice if each was prefixed by /ambilight/. Hence, the creation of the top-level “Ambilight” category for all of the posts.
This also meant I tried not to nest categories for fear of needlessly elongating the URL. “Arduino” probably should have been a sub-category under “Electronics”, but since the former invokes the latter /electronics/arduino felt almost redundant. So it was kept as is, further diluting the category list.
Organizational Anxiety
Eventually this led to a great deal of anxiety about how these categories, emphasized in the permalinks, would present my content to the world. Although posts could be listed in multiple categories, there could only be one “main” category that was reflected in the permalinks. This led to some messy questions without clear solutions:
- If I built a “Prop” that also had “Electronics” in it, which category does it fit “best”?
- If I build a microcontroller project, is that more “Electronics”, or “Programming”?
- If I write a “Tutorial” that is about “MIDI”, which category would be better for SEO?
This reached a head last October, where I finished a project and the documentation but held off on posting it because I didn’t want to create yet another top level category for all of the posts, and then have to create more redirects for that category and the posts within it knowing that the structure would eventually be changed.
Something had to be done.
The Solution
Originally, I thought this was all going to be “solved” once I figured out a good category structure and finally got my house in order. Instead, I’ve come up with a simpler solution:
I’m just eliminating the category from the URL altogether.
This means that even if the category structure is imperfect, I can rework it later on and only have to worry about the structure internally. I can move posts between categories freely without having to build a rewrite rule. And if I were to change or delete a category altogether, I would only need to create redirects for the category itself, rather than for every post and sub category under it.
It also means that posts won’t be married to or primarily represented by a single category. If a post is a “Project”, a “Custom Controller”, has a bunch of “Programming”, and is a “Prop” it can live happily in all of those categories without needing to choose just one as its representative for the URL.
As far as SEO goes this is a wash. Losing the category in the URL is theoretically worse, but without the category base the permalinks will be shorter, which is theoretically better. So I’m not losing sleep over it.
Executing the Changes
Down for maintenance! The offline screen while these changes were being made.
Before anything else, I created a full backup of my website just in case anything went horribly wrong during this transition. Then I installed a “Maintenance mode” plugin in order to take the site offline while I was tweaking things. I also disabled the YoastSEO plugin, just so it wouldn’t try to auto-magically fix categories and permalinks while I was in the process of changing them.
I did this entire process start to finish in the middle of the night U.S. time, while traffic would be at its lowest point. Belated apologies to anyone who was inconvenienced!
Permalink Modifications
First up was the permalink structure. This is a simple change in the “Permalink Settings” options from the “Custom Structure” to the common “Post Name” option. Click one radio button, hit “Save”, and that’s it.
Due to the way WordPress’ canonical redirects work, the old URLs for the posts that contain the category prefix are redirected automatically. For the handful of posts where the permalink wouldn’t make sense without the category prefix, I changed the permalink slug and will need to create a manual redirect. More on that in a minute.
Old Categories to Tags
The next step was dealing with the obsolete categories. Most of these that were based around a type of technology (“3D Printing”, “Arduino”, etc.) are best served as tags. In fact most of these categories are tags already for that very reason, so I just needed to pick up the slack and create the missing ones: “computers”, “electronics”, and “reverse engineering”.
After creating the tags I then needed to map the posts from the “category” version to the “tag” version. Since WordPress allows you to manage posts in bulk through the admin interface, this was a piece of cake:
- From the “Categories” page, click on the link in the “Count” column for a given category
- “Select all” using the selection box in the top left corner
- Using the menu at the top: “Bulk Actions” -> “Edit”
- Add the tag(s) and click “Update”
I repeated this process for every category that was going to be removed, which only took five minutes or so.
Creating New Categories
With the ‘old’ categories ready to retire, it was time to create the new ones.
After renaming “Uncategorized” to “Articles”, I then created the “Multi-Part Projects” category as a child of “Projects”, and set it as the parent for the existing “Ambilight” category. Next I added the multi-part project children (“McCree Hammershot”, “Time Circuits”, etc.) and the extra categories under “Projects”. Once the new categories were in place, I added the relevant posts using the same bulk edit process I used for reassigning the categories as tags.
At this point it was time to say “goodbye” to the categories of yore. All of the old categories were removed in one fell swoop. Any posts that were not yet listed in a “new” category were automatically moved into “Articles”. After a second pass to verify that all of the posts were where they were supposed to be, it was time to set up the redirects.
Redirection
The plugin I’m using to handle the redirects is called Redirection. It’s widely used (1 million+ installations) and well reviewed. I decided to go with a plugin rather than modifying things at a lower level (.htaccess) just to keep things simple. Plus these will be straightforward to modify later on if need-be.
Redirection allows you to import and export your redirect rules as JSON or CSV, so in preparation for this rework I built a comprehensive spreadsheet of all of my posts, their parent categories, their original slugs, and their updated slugs in the new structure. This allowed me to generate a CSV file to import into Redirection so I didn’t have to write any rules by hand. As WordPress’s canonical redirect system took care of vast majority of the posts, I only needed to generate a total of 56 rewrite rules: 26 for the categories (including rules with and without the ‘category’ prefix), and 29 for posts that had their slug changed.
When doing a trial run of these changes, I realized that there was an issue with trailing slashes. Specifically, WordPress includes trailing slashes on URLs by default, but most people entering URLs or linking externally don’t include them. If I navigated to one of the old links without a trailing slash, Redirection would not pick it up – leading to a 404.
I was just about to re-generate the redirects using a regex expression ([\/]?) when I found out that as of version 4.0 (2/23/19), Redirection includes an option to ignore trailing slashes! A quick toggle “on” in the plugin’s global settings and everything was ready to go.
Semi-Automated Testing
All of the changes were completed! I re-enabled the YoastSEO plugin and got ready to disable maintenance mode and greet the world anew.
Just one more thing left to do: make sure everything worked!
By my count there were approximately 190 URLs affected by this rework (that I could remember, at least). Rather than copying and testing each “before” link one at a time, I decided to build a script to do it for me.
# Project PartsNotIncluded Semi-Automatic URL Tester
# @author David Madison
# @link partsnotincluded.com/site-taxonomy-rework-2020
# @license MIT - Copyright (c) 2020 David Madison
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
#
import webbrowser
import csv
def main():
filename = "PNI Testing URLs.csv"
chrome = webbrowser.get("C:/Program Files (x86)/Google/Chrome/Application/chrome.exe %s") # chrome installation directory
urls = [] # list of URL dictionaries
nrows = 0 # number of rows, counter
print("\nStarting URL Tests... using \"{}\" as input\n".format(filename))
print("--------------------------------------")
# open the CSV and save the rows as dictionaries into the 'url' list
with open(filename, newline='') as f:
reader = csv.DictReader(f)
for row in reader:
urls.append(row)
nrows += 1
test_groups = urls[0].keys() # groups of URLs being tested, as dictionary keys
test_number = 1 # tracking number of tests complete
total_tests = len(test_groups) # total number of tests
# for each group of URLs to test:
# * open up URL in a new chrome tab
# * when 5 URLs are opened, pause and wait for confirmation
# * when a testing group has finished, wait for confirmation
for test in test_groups:
print("Testing Group: \"{}\"\n".format(test))
urls_open = 0
for number, row in enumerate(urls):
if row[test] is not "":
print("Opening:", row[test])
chrome.open_new_tab(row[test]) # open tab in browser
urls_open +=1
if urls_open >= 5:
input("\t(Press Enter to continue)\t\"{}\" (Test {}/{}) {} out of {} rows processed" \
.format(test, test_number, total_tests, number, nrows) )
urls_open = 0
print("\n\tTest \"{}\" Complete! ".format(test), end="")
if test_number is not total_tests:
input("Press Enter to continue...\n")
test_number += 1
print("\n\n-----------------------------\n")
print("All tests complete!")
if __name__ == "__main__":
main()
This short Python script reads a CSV file containing URLs to check. It will open up 5 URLs in Chrome at a time, waiting for user input to verify that the webpages opened properly and the redirects worked. This is done one column at a time so that different URL sets and rewrite rules can be tested together in groups.
I could have used cURL or another library to fully automate this and check the redirect headers, but looking at the pages themselves also allowed me to triple-check that posts were in the right categories and had their breadcrumbs configured properly. Since I’m not planning on making a habit of this, a visual check worked just fine.
I just threw this script together quickly, but it was a useful little tool so I’m sharing it here for posterity. On the off chance that anyone has any use for it it’s licensed under MIT.
While testing the redirects I also ran into a few issues where URLs would redirect to the home page rather than to their respective posts. After chasing a few red herrings this turned out just to be an issue with browser caching; clearing my local cache completely solved the problem. Although for good measure I double-checked all posts using Chrome in Incognito mode (by adding --incognito to the Chrome filepath in the script) just to be safe.
Moving Forward
Even though I did a trial run of this process on a local copy of the website, it was still quite harrowing to make such a significant change to the live version. The total process including testing took around 3 hours, and I have my fingers-crossed that I did everything properly. The Redirection plugin has 404 logging enabled, and so far it’s already caught 4 old URLs that slipped through the cracks. I’m going to keep a close eye on it over the next few days just to make sure I didn’t miss anything else.
I have high hopes that these changes are going to be healthy for both the website and myself moving forwards. For 2020 I want to focus more of my energy on creating things rather than organizing them.
The permalink structure should (“should”) be a permanent change. The category structure, although it seems good at the moment, may end up being tweaked as more content is added. Although by breaking the permalink’s reliance on categories, changing the site’s taxonomy in the future will be far less of an ordeal.
In the meantime, I’ve got at least one more major website change to make in the near future and two fun propmaking projects to post. Let’s get to it!
Projects in the Featured Image
One last thing! I never quite know what to do for the featured images in these “Meta” posts, so I decided to have a little fun with this one. I took a spare cardboard box, wrote a big “Junk” label on the side, and then stuffed it full of packing peanuts and past projects. Just in case something caught your eye, here are all of the projects in the box:
