Manugarri's blog

Manugarri's blog

Latest — Apr 23, 2024

Four lessons from managing a company wide Airflow plugin

At my current company (Carpe Data, we are hiring!) , one of my tasks is to maintain an internal Airflow plugin (common-airflow-utils) with Airflow related utilities. The library is used by 5 different teams, powering a big part of our production worfklows. and it provides higher abstractions on top of the

More issues

Thing i learned migrating from Digital Ocean to AWS Lightsail

Its been 9 years since I spin up my Digital Ocean instance. It has been hosting my personal site, blog, random personal projects, and even faced one or two Hacker news front page traffic spikes. Why named dokku you wonder, oh young reader? Well, back before kubernetes was a thing,

NOTES: Setting up git after a fresh install.

Recently I did a fresh install of Ubuntu 20.04 via WSL2 (which i don't love yet but its growing on me), and I had to do the following steps to set git up: 1.Install git (duh!)Im just putting it here for completion sake sudo apt-get update sudo

Making a simple, better weather and traffic conditions map for Spain's roads

TL;DR I made a map displaying weather and traffic road conditions for Spain that is easier to use, nicer and faster than the official Spanish Government map. As usual (1, 2), I keep being disappointed with the Spanish Government Open Data policies. You can check out the map here.

Airflow UI: How to trigger a DAG with custom parameters?

Airflow is one of the most widely used Schedulers currently in the tech industry. Initially developed at Airbnb, a few years ago it became an Apache foundation project, quickly becoming one of the foundation top projects. It is a direct competitor of other schedulers such as Spotify's Luigi or newer

Airflow UI: How to trigger a DAG with custom parameters?

Airflow is one of the most widely used Schedulers currently in the tech industry. Initially developed at Airbnb, a few years ago it became an Apache foundation project, quickly becoming one of the foundation top projects. It is a direct competitor of other schedulers such as Spotify's Luigi or newer

Note to Self. Installing LightGBM in Ubuntu 18.04

These are the steps I took to install Microsoft's cool Gradient Boosted Models library, LightGBM Step 1. Install CUDA I am not going to explain this step because it is easy to find. Step 2. Install Boost sudo apt-get install libboost-all-dev Step 3. Clone LightGBM and build with CUDA enabled

Note to self. Pyspark failling with "Error while instantiating ‘org.apache.spark.sql.hive.HiveSessionState’"

If you run pyspark and see this error (it happens in scala-shell as well): Error while instantiating ‘org.apache.spark.sql.hive.HiveSessionState’ The solution is easy, yet ridiculous. 1. Create the folder /tmp/hive 2. Give it chmod permissions sudo chmod -R 777 /tmp/hive Found here

Note to self: Fixing encoding in Golang ascii85

Yesterday I spent a few hours dealing with what I like to call "the edges of StackOverflow". By that I mean those situations in which you are trying to solve a programming problem (mostly a bug) and you have no idea why its happening, and even worse, no amount of

Note to self:Print statements not showing up on systemd logs? Do this

Let's assume we have a service set up as follows: [Unit] Description=systemd_microservice [Service] User=USER Group=GROUP WorkingDirectory=systemd_working_directory ExecStart=/usr/bin/python python_scripts.py SuccessExitStatus=143 TimeoutStopSec=10 Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target And inside python_script.py you have a