One day project: Wikiloc exporter

Last weekend, I had some time to kill, and I thought I would tackle a minor pet peeve of mine.  

My wife and I are trying to go on a hike at least once a month, we like to try different trails around our area. In Spain, the dominant site for trail information is wikiloc. Its an awesome sime, with hundreds of trails ranging from beginner trails to very hard trails.

However, the site also uses  freemium scheme that doesnt resonate well with me.  See, you can see the trail on your browser for free, but for navigation functionalities, you have to pay X a month, which is a bit steep for a casual user like me. This means when you are walking the trail, you have to use your orientation skills to figure out where the hell you are on the map, or risk getting lost, which has happened to us more than once.

I thought, wouldnt it be great if I could extract the trail information into a map service with proper geolocation?  How about MapHub, one of the simplest free map providers that exist?

Turns out it was easier than expected, and here is the code if you want to use it yourself.

Exploring the site

First thing I did was to explore wikiloc. Its a site choke full of functionalities, most of them behind a paywall unfortunately.

As we can see on the bottom left of the map, Wikiloc uses leaflet, an awesome OSS library for map building. I have used it a few times for personal projects (here is an example).

What that means is that somewhere on the javascript side, there is a geojson that is being used to generate the leaflet trail (as a polyline). Let's see how to do that.

Extracting the data

By inspecting the site's code, we can see that the waypoints (meaning, the individual marquers we can see highlighting points of interest) are defined as Json linked data inside the site.

That is great, we can explore the site to see how to fetch these points. Chances are, there is a javascript variable somewhere that makes use of those points to display them on the leaflet map.

We can inspect the variables on the site inside the developer console.  I always use this simple snippet to print the variables defined at the window scope level, because the website devs have probably used a reasonable,meaningful name to define the map variables.

var variables = {}
undefined
for (var name in this) {
    variables[name] = name;
    variables[name]=this[name]
}

After running this snippet and printing the variable names, we find 2 objects that provide us the information that we need, mapData and trailMap

mapData contains the waypoints with their coordinates, names, and so on. We will fetch that information and add it to our exported map.

trailMap contains the reference to leaflet itself (that is imported as a separate library). This means we can export the trail information (meaning, the trail course ) as a geojson easily since thats what Leaflet uses internally.

In particular, we can export the layers of the leaflet map with this simple JS snippet.

var collection = {'type':'FeatureCollection','features':[]}; trailMap.eachLayer(function (layer) {if (typeof(layer.toGeoJSON) === 'function') collection.features.push(layer.toGeoJSON())}); 

That snippet will export a variable with valid geojson representing the trail.

Exporting the data

Now that we know how to extract the data from the site, we just have to copy all the js code into a python script that will run the extraction for us.

Since we need to execute Javascript to extract the data, we will use a headless browser (playwright) to execute the js.

Then we need to push the data into MapHub.  This is super easy, as the only thing we need to do is to create a maphub account, make a api token, and use that token to submit http requests to their api  endpoint to create the map.

We can do some final aesthetic improvements too, for example its nice to have the initial waypoint for a trail colored in green, we can do that by specifying the point's properties. For example, to change the color of the point we just have to set the property marker-color to a different hexcode color.

And voila, here is the exported map in MapHub!