Graphing Wikipedia 1: Understanding the Data

Collecting Data

In the previous section, the biggest concern was that because we needed to interact with Wikipedia’s servers, the amount of requests needed to be minimized. Fortunately, there are some ways to get data dumps¹.

MediaWiki Content File Exports

Within all of the options present, there is a directory called enwiki/. Which should contain all of the articles we are looking for (as well as some other stuff). As of writing this, the latest dump is from August 2026. All of these files are offered in an XML format.

Understanding the Data

What is XML

XML is a markup language which has a similar structure to HTML. It’s sole goal is to provide a way to transport data in an software/hardware independent format. W3Schools provides examples of what this data looks like. What the data is trying to represent is indutive enough to not need explaining.

<!-- From https://www.w3schools.com/xml/xml_usedfor.asp (truncated) -->
<?xml version="1.0" encoding="UTF-8"?>
<bookstore>

  <book category="cooking">
    <title lang="en">Everyday Italian</title>
    <author>Giada De Laurentiis</author>
    <year>2005</year>
    <price>30.00</price>
  </book>

  <book category="children">
    <title lang="en">Harry Potter</title>
    <author>J K. Rowling</author>
    <year>2005</year>
    <price>29.99</price>
  </book>

  <!-- Truncated -->

</bookstore>

XML as a Database

Because XMl files are effectively a document database, a useful way to understand their structure is to rely on their XML Schema. enwiki exported files each rely on the same schema file which can be found online.

Potential XML Challenges

Because of the XML syntax rules, some characters might not usable. This could include:

• Characters that cannot be used in HTML such as ‘&’, which needs to translated to &. This might be an issue, especially if an article title uses these characters.
• Character encoding might be an issue. The extent of this issue will be evaluated when it arises.

The Current Goal

Now that the archive data is known, the goal now is to:

1. Iterate over every single XML file given in the archive, filtering out everything that isn’t an article.
2. For each article we iterate over:
   1. Extract all article text and parse out every link. MediaWiki, the underlying software used by Wikipedia, provides documentation on how links are formatted.
   2. Store all links inside of a database for later stages. This will be important to quickly generate the graph file (or whatever will be used in the end visualization tool).

Considerations

The biggest challenge generating the link database will be that each XML file is too large to fit into memory² and needs to be streamed. While Go does provide an easy way to parse and XML file, it requires the entire file to be loaded in at once. A much more streamable package will need to be used in addition to lower level XML decoding operations.

1. I knew this beforehand, I just want to act like I didn’t because it is important to talk about. ↩︎

2. A single file is around 13 GB and the entire collection of files is 121 GB ↩︎