One-Off Tasks with Kubernetes

In PaaS language one-off means running a task once until it exits, independet of its success. Such tasks should not be restarted by the scheduler in comparison to normal tasks (aka services) which should always run and be restarted in failure cases. A good example for a one-off task is the database migration rake db:migrate in Rails.

Interestingly PaaS/cluster solutions often provide fantastic support for executing services and keeping them running but don’t have a good implementation of one-off tasks. Kubernetes is no different, one-off tasks are not yet implemented as a native concept. But with the restartPolicy it is possible to define a pod which does not get restarted.

Pod definition for one-off task one-off.json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{ "id": "one-off-curl", "kind": "Pod", "apiVersion": "v1beta1", "desiredState": { "manifest": { "version": "v1beta1", "id": "one-off-culr", "containers": [{ "name": "one-off-curl", "image": "tutum/curl", "command": ["/bin/sh", "-c", "sleep 30 && curl -X POST -d 'one-off=possible' http://requestb.in/18cvw901"], "cpu": 100 }], "restartPolicy": { "never": {} } } }, "labels": { "name": "one-off-curl" } }

Please create a new Requestbin URL for your own test and replace it the pod definition above. The one given is outdated and does no longer work.

Now we can create and start the one-off task:

1
$ kubecfg.sh -c one-off.json create pod

When getting the Pod, you will see that the container has been started and is in Waiting state as the image has to be downloaded first:

1
2
3
4
5
$ kubecfg.sh get pods/one-off cluster/kubecfg.sh get pods/one-off ID Image(s) Host Labels Status ---------- ---------- ---------- ---------- ---------- one-off tutum/curl 10.245.2.2/10.245.2.2 name=one-off Waiting

When the provisiong is complete the state should change to Running and shortly afterwards it changes again to Terminated as the task exited.

1
2
3
4
5
$ kubecfg.sh get pods/one-off cluster/kubecfg.sh get pods/one-off ID Image(s) Host Labels Status ---------- ---------- ---------- ---------- ---------- one-off tutum/curl 10.245.2.2/10.245.2.2 name=one-off Terminated

As we added a sleep before executing our curl command the Pod should stay in the Running state for at least 30 seconds.

Now we can open the inspect page for our Requestbin to see if it received a request by our one-off task: http://requestb.in/18cvw901?inspect. If all worked well we should see one post request.

So far everything worked. But some gotachs exist:

  1. We need to clean up the created Pod after it terminated by deleting it:

    1
    $ kubecfg.sh delete pods/one-off

    This means the Pod needs to be monitored to wait until it terminates. And probably you are interestd in the exit code as well which you can get by running kubecfg.sh -json=true get pods/one-off.

  2. When deleting the Pod and recreating it, often the Pod’s state is reported as Terminated first which is not true. It should be in state Waiting. Anyway Kubernetes is actually provisioning our container and shortly after it changes to the Running state. But this behaviour makes 1. more difficult when we want to automate the one-off task. It requires already waith for the Running state and then again until it reaches the Terminated state.

Hopefully Kubernetes will soon provide a native concept for one-off tasks where the Pod is automatically cleaned up. What is unsolved then is how to receive back the exit state of the container.

Georg Kunz
I'm a freelancer specialized on infrastructure and web performance with the goal to improve developer's life. Founder and organizer of the Swiss Rails User Group. Former product manager and software architect at local.ch.

 
Do you need help with DevOps, CloudFoundry, Docker, Ruby or Rails? I'm available, please get in touch with me to discuss a collaboration or check out my website.

comments powered by Disqus