1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
|
Simple Text Queue
#################
Why and what it is
==================
**teaqueue** is a command pair able to process sequentially items such as
files from a list of filenames in a text file by calling a given command for
each of them. You use **GNU** tools like `find` to populate your queue, or use
**vim** to reorder it.
It can be used to:
- transcode a batch of video files such as your preferred technical
presentations to a format accepted by your mobile phone,
- retrieve a bunch of URLs or other remote operation,
- sequential operation on anything that can be encoded on a single line (json
data, base64 data, etc).
It is really useful for operations taking a lot of time when you want to be
able to stop processing items in the queue and resume it. Thus, when stopped
you can optionally change the order of items, add items, or remove items from
the queue.
Why `tea` ???
-------------
Because I love tea.
Usage
=====
Usage is linked directly to manipulation of simple text files. You use text
files to make the queue, to add or remove items from it, to change priority of
items. *UNIX is the way*.
I give some examples in this section. Take into account `teaqueue-server`
takes `queue.txt` in the current directory by default as queue file modifying
it when interrupted to remove processed items from it. By default, processed
items goes to `done.txt`. This files serves to remove processed items from
`queue.txt` when the queue is interrupted and serves as a log file for
processed items.
Video batch transcoding
-----------------------
Firstly list filenames in the queue and serve the queue::
find -type f -iname '*.webm' > queue.txt
teaqueue-server
Then ask `teaqueue-client` to process each files with a transcoding command::
teaqueue-client transcode_my_file.sh
This is all if you just want to process each file and do not interrupt it. If
you want to interrupt processing of the queue, modify it and resume it, first
press `Ctrl-C` in the `teaqueue-server` console and modify `queue.txt`
accordingly::
find ../another_dir -type '*.webm' >> queue.txt
Or reorder some items::
vim queue.txt
Then run again `teaqueue-server`, clients will continue to ask new items from
the queue::
teaqueue-server
`transcode_my_file.sh` is a called a worker and is a simple script which calls
`ffmpeg` or another program to convert your files. You can find examples of
workers inside the `worker_examples` directory.
Write a worker
==============
In order to be useful you certainly have to write a worker script. Writing
a worker is simple, all you have to do is to store the standard input line and
work with it.
Echo worker
-----------
For example, a useless worker, but simple to understand the principle is just
a worker echo each item it receive:
.. code:: shell
#!/bin/sh
line="$(cat)"
echo "Printing line '$line' and sleeping $1 seconds"
sleep $1
You can use it with a `teaqueue-client`::
teaqueue-client ./echo_sleep.sh 10
Remote files
------------
If you want to be able to use `teaqueue-client` on another host than the
`teaqueue-server` you can download the files (sharing it on the server side is
done through anything you want) from the host, then work on it and push back
the transformed files. Just write the worker in consequence:
.. code:: shell
#!/bin/sh
filename="$(cat)"
echo "$filename" | nc -q0 serverin 1338 > file
transcode.sh file > fileout
(echo "$filename"; cat fileout) | nc -q0 serverout 1338
This example assume you share your files with simple netcat commands, but you
can adapt with anything you want: wget, scp, etc.
History
=======
Normally, if you have a worker writing to standard output every file
processed, you could write something like this::
find -iname '*.ext' > queue.txt
cat queue.txt | worker > done.txt
# here you can interrupt the service at any time (Ctrl-C/SIGTERM)
comm -23 queue.txt done.txt > remaining_queue.txt
But if this is something you want to do again and again, it might be
convenient to write a little wrapper to do it for us. This is how **teaqueue**
was born.
Limitations
===========
**teaqueue** is not perfect and does not try to be. There is no security and
some rare race conditions.
Known race condition:
- If someone interrupts `teaqueue-server` just after the line has been sent
but before it gets printed out, the line will not be filtered in the
`queue.txt` file. Thus it will be send again to the next demanding client.
License
=======
**teaqueue** is shared to you under the GPL-3 license conditions. See
`LICENSE.txt`.
|
