Automating tasks with R

James Laird-Smith

Data Scientist, Bank of England

• Link to slides: talk-automation-R.jameslairdsmith.com/
• Github repo for talk: jameslairdsmith/talk-automation-R

Agenda

• Why learn to automate?
• What tools are available for automation?
• The future?

Why learn automation?

Automation is the logical extension of coding.

Given that you have some operation.

Step 1: Re-write in code

• Pain: Learn to code
• Gain: Can then re-run at zero effort (among other things)

Step 2: Automate

• Pain: Learn an automation tool
• Gain: Can then re-run at zero effort even when you are not there

I see automation as the logical extension of coding. It is particularly important for organisations.

How can you automate?

Note

R itself is not an automation tool. Rather, it interfaces with other purpose-built tools.

• Cron
• Apache Airflow
• Windows Task Scheduler
• RStudio Connect
• Jenkins
• Github actions
• Many more…

R does have tools for interfacing with these.

No programming language is actually an automation tool.

To have an automation tool, you need an always on server with a daemon running in the background.

I’m betting that for many of you, automation is something hidden away, mysterious.

Varying levels of maturity and sophistication.

These each have their pros and cons.

What do we want?

What does an ideal automation tool look like?

• Simple to use.
• Feature rich. Eg. have triggers (or listeners).

• Developer friendly. Eg. Have an API
• User friendly. Eg. Have a good graphical interface (GUI).

• Free and open source with no vendor lock-in.
• Language agnostic? Or at least work with R?

I want to prime you to think about what you would want out of an automation tool.

All automation tools will have a way to trigger events on a schedule, only some will allow you to create triggers.

Cron

In the beginning God created the heavens and the earth … but also the Cron utility for Unix.¹

• Very old, widely used and well understood.
  • If you are running MacOS or Linux, it’s likely already on your computer.
• Minimal and easy to get started with.
• Cron syntax is the lingua franca of automation. Other tools will accept it as input.

These things make Cron a good tool to get started with.

1. Gospel according to James.

Cron expressions

Cron has it’s own syntax for job frequency and schedule.

* * * * *
| | | | |
| | | | +---- Day of the Week   (range: 0-6, 0 is Sunday)
| | | +------ Month of the Year (range: 1-12)
| | +-------- Day of the Month  (range: 1-31)
| +---------- Hour              (range: 0-23)
+------------ Minute            (range: 0-59)

crontab file

# Comment lines start with hastags
0 0 * * * echo 'Hello midnight!'
0 0 1 1 * echo 'Hello New Year!'
0 0 25 * * echo 'Hello payday!'
0 0 * * 6 echo 'Hello Saturday!'
* * * * * Rscript -e 'print("Hello R!")'
* * * * * python -c 'print("Hello Python!")'

Cron’s syntax is richer than this. To learn more you can go to crontab.guru or consult the spec here.

This is a simplified version. Cron has a richer syntax than this.

Adapted from: https://www.maximehuran.fr/en/create-a-cron-job-with-magento-2/

cronR

• Originally by Kevin Ushey, but maintained by Jan Wijffels.
• Helpers for writing Cron expressions and writing to crontab files.

library(cronR)

cron_add(
  command = "Rscript -e 'print(\"Hello R!\")'",
  frequency = 'daily', 
  at='7AM',
  dry_run = TRUE,
  ask = FALSE
)

Adding cronjob:
---------------

## cronR job
## id:   2d2a128c4688230eb141b52d86f3befa
## tags: 
## desc: 
0 7 * * * Rscript -e 'print("Hello R!")'

There is something similar for Windows with taskscheduleR, also by Jan Wijffels.

The drawback of this is that it’s not friendly to version control.

I know there are people out there who have tried to program around Cron.

Apache Airflow

Airflow is a platform […] to programmatically author, schedule and monitor workflows¹

• Perhaps the most mature and well understood modern automation tool for data.
• Fully featured and extensible.

---

• Started by Airbnb.
• Open source and with a community of developers.

---

• Powerful GUI.
• Notable for it’s use of Directed Acyclic Graphs (DAGs).

Strong focus on data engineering.

1. https://airflow.apache.org/

Apache Airflow (2)

Any set of tasks is going to have dependencies. Without a DAG, you are going to have to manage those dependencies manually. This means you are going to have to program your own trigger/sensor.

Image source: https://pypi.org/project/apache-airflow/

Apache Airflow (3)

from datetime import datetime
from airflow import DAG
from airflow.operators.python_operator import PythonOperator

def print_hello():
    return 'Hello world from first Airflow DAG!'

dag = DAG(
  'hello_world', 
  description = 'Hello World DAG',
  schedule_interval = '0 12 * * *',
  start_date = datetime(2017, 3, 20), 
  catchup = False
)

hello_operator = PythonOperator(task_id='hello_task',
                                python_callable = print_hello, 
                                dag = dag)

Airflow is really the industry giant of automation.

Written in Python, but deeply intertwined with Python.

You can execute R, but you do so as command line.

Adapted from: https://progressivecoder.com/airflow-dag-example-create-your-first-dag/

GitHub Actions

• Widely used as part of package testing (sometimes called CI/CD).
• Triggers can be repo events (like push and merge) or Cron schedules.
• Workflows are specified in YAML.
• Good support for R from work done by RStudio:
  • See: https://github.com/r-lib/actions

name: GitHub Actions Demo
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: ubuntu-latest
    steps:
      - run: echo "🎉 Job automatically triggered by ${{ github.event_name }}."
      - run: echo "🍏 This job's status is ${{ job.status }}."

There is also GitLab, the free version of which you could run on a Server.

Existing tools: Others

• Windows Task Scheduler
  • Heavily tied into Windows.
  • Your PC always has to be on.
• RStudio Connect
  • Good at managing R dependencies.
  • Scheduling still very GUI based.
• Nothing really has the feature set of Airflow.

Existing tools: summary

	Cron	Airflow	GitHub Actions	RStudio Connect
Easy to use	✅	🟡1	✅	✅
Trigger and DAG support	❌	✅	❌	❌
Good R support	🟡2	❌	✅	✅✅
Language agnostic	✅	🟡3	✅	❌
Has an API	❌	✅	✅	🟡4
Nice GUI	❌	✅✅	✅	✅
Free & open source	✅	✅	❌	❌
Not tied to a vendor	✅	✅	❌	❌

1. It’s quite a lot of setup and you need to know Python.

2. We do have cronR.

3. The API is language agnostic, but you can’t create DAGs with it right now (although this might be changing).

4. It exists, but I’m not sure of scheduling bindings?

The future¹

Focus on the open source technologies.

• Cron for the simpler automations
  • Synchronise crontab from a YAML file.

• Apache Airflow for the large scale deployments
  • R bindings to the API.
  • Create DAGs in R? Transpile them to Python?

1. …or at least what I think we should do.

Thank you!

Questions?