Graphing Wikipedia 0: Getting Started

Wikipedia is a massive article database, and is widely known across the world. One key aspect of this database is that it is highly connected via hyperlinks (and a lot of fun toys can be made with this fact).

The Goal

Throughout this series I want to be able to graph out the entirety of Wikipedia and present in a visual format to allow myself and others to see how closely related every article is. For example, in the graph shown here:

• “Pear” is directed to “Apple”
• “Apple” is directed to “Pear”
• “Horse” is directed to “Apple”
• There are some other edges not shown here

While this could be a potential way to represent all the nodes produced in later states, because the graph can be highly connected, ths final visual might be different than the one shown here.

Subgoals

There are some other goals and contraints that I want to acheive out of this as well. These are:

• Learn a new programming langauge. I think that because of the “backend heavy” work required to produce the final graph, I think that Go will be a good one here.
• When constructing the graph, No more than 6 GB of memory should be used (as thats all I have available). Note that is not the memory required to show the graph¹.
• Minimize the amount of storage being used, especially for the final constructed graph. The final graph should be as small as possible because the client will eventually need to fetch this graph from where its hosted.

Considerations

There are some things to consider when developing in this project:

• Data will need to be fetched from Wikpedia’s servers. Before any work is done, the goal is to try and be as nice as possible to them. Maybe some archives exists, so that work can be done locally
• There is no way that the entirety of Wikipedia can fit within memory. The data will need to be worked on in chunks.

1. Using 6 GB of memory to render a graph in a web browser would be absolutely horrible. This is a later issue. ↩︎