1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
Simple Text Queue
#################
Why and what it is
==================
`teaqueue` is a command able to process sequentially items such as files from
a list of filenames in a text files by calling a given command for each of
them.
It can be used to:
- transcode a batch of video files such as your preferred technical
presentations to a format accepted by your mobile phone,
- retrieve a bunch of URLs or other remote operation,
- sequential operation on anything that can be encoded on a single line (json
data, base64 data, etc).
It is really useful for operations taking a lot of time when you want to be
able to stop processing items in the queue and resume it, optionally changing
order of items or adding/removing items from it before.
Why `tea` ???
-------------
Because I love tea.
Usage
=====
Usage is linked directly to manipulation of simple text files. You use text
files to make the queue, to add or remove items from it, to change priority of
items. *UNIX is the way*.
I give some examples in this section. Take into account `teaqueue` takes
`queue.txt` in the current directory by default as queue file modifying it
when interrupted to remove processed items. By default, processed items goes
to `done.txt`. This files serves to remove processed items from `queue.txt`
when the queue is interrupted and serves as a log file for processed items.
Video batch transcoding
-----------------------
Firstly list filenames in the queue::
find -type f -iname '*.webm' > queue.txt
Then ask `teaqueue` to process each files with a transcoding command::
teaqueue transcode_my_file.sh
This is all if you just want to process each file and do not interrupt it. If
you want to interrupt processing of the queue, modify it and resume it, first
press `Ctrl-C` and modify `queue.txt` accordingly::
find ../another_dir -type '*.webm' >> queue.txt
Or reorder some items::
vim queue.txt
Then run again `teaqueue`::
teaqueue transcode_my_file.sh
`transcode_my_file.sh` is a simple script which calls `ffmpeg` or another
program to convert your videos. The important point is the script should be
able to be interruptible without interrupting the underlying job.
.. code:: shell
#!/bin/sh
[ "${FLOCKER}" != "$0" ] && exec env FLOCKER="$0" flock -e "$0" "$@"
ffmpeg -i "$1" -arg -arg &
The first `ffmpeg` command will go to background then, the second will wait
until the first `ffmpeg` finishes. If the `teaqueue` command is interrupted,
it can be run again without effectively interrupting the current `ffmpeg` job.
History
=======
Normally, if you have a worker writing to standard output every file
processed, you could write something like this::
find -iname '*.ext' > queue.txt
cat queue.txt | worker > done.txt
# here you can interrupt the service at any time (Ctrl-C/SIGTERM)
comm -23 queue.txt done.txt > remaining_queue.txt
But if this is something you want to do again and again, it might be
convenient to write a little wrapper to do it for us. This is how `teaqueue`
was born.
|
