Graphing Wikipedia 3: Lessons Learned

The Final Setup

After spending some time I was able to produce a single sqlite database containing every single article that is dumped by Wikimedia. Before generating indexes, I was able to see around a 60% reduction of space (from 65 GB to 26 GB) when only considering articles.

Lessons Learned

There are some challenges that I did not propery fix, which caused this project to take longer that I wanted to. I’ll briefly talk about them here.

USE SMALL BATCHES

I actually did create smaller versions of my final database in the early stages of the project, which was useful to validate my generation scripts as well as my API. Unfortunately, as I was getting closer to the final steps of this project, when I was performing fulls runs using real data, I failed to test with smaller batches first, causing me to lose days of compute, due to a minor mistake.

Filter/Organize as Much as Possible

The hardest part of handling large amounts of data is the fact that a lare portion of the data will not be used and can was storage space. This biggest example of this comes with the pagelinks table. The raw pagelink sql file comes in at 34 GB, but when filtered to only include articles, this goes down to 17 GB. While this is an extreme example here, these incremental reductions are always helpful, especially for IO heavy workflows. Aside from filtering, making sure your data is quickly readable is also as important. This can be done in a variety of ways, such as indexing.

While not directly obvious, another form of optimzation comes into play when looking up articles through a search bar. Because not everyone will know the title of exact title of the article they are looking for, there needs to be some method to provide close matches to the user. Building a complex search engines is way beyond the scope of this project, but basic indexings such as full text searches (I used ft5) perform better than direct string comparisons.

Make Scripts to Automate Tedious Operations

One of the reasons that I had trouble running small batches when performing end to end tests, what that I never developed a proper way to create a proper containerized testing envrionments. While I did utilize Docker to test specific sections of the database generation scripts, I never went any further.

How Well I Met My Goals

Throughout this series, I wrote down a couple of goals that I wanted to see. Here I talk about each one I mentioned, and how I tried to meet them.

Goal	Comments
Provide a visualization tool to view the connections for all Wikipedia sections	This is the end product.
Figure out a way to handle visualizing every single article and its outbound links	I knew visualizing all 7 million nodes would be impossible with my set of skills as well as requiring too much network bandwidth, so to handle this, I designed the user interface in a way such that the user can “discover” the outbound links, instead of providing all of them upfront.
Learn Go	Some of the script was written in Go, so I got some of the basics
Use no more than 6 GB of memory when generating the database	Not measured, but its not relevant here, as the project became a IO bandwidth problem
Minimize the raw MediaWiki dumps into something usable	I was able to generate a single sqlite3 file that contains all nessesary information.

Where Can You See this

Unless it changed, you can see the final visualization app using this link. Every so often I’ll try to update to the latest version of the provided dumps.