Simple Text Queue

Why and what it is

teaqueue is a command pair able to process sequentially items such as files from a list of filenames in a text file by calling a given command for each of them. You use GNU tools like find to populate your queue, or use vim to reorder it (or any text editor you want).

It can be used to:

transcode a batch of video files such as your preferred technical presentations to a format accepted by your mobile phone,
retrieve a bunch of URLs or other remote operation,
sequential operation on anything that can be encoded on a single line (json data, base64 data, etc).

It is really useful for operations taking a lot of time when you want to be able to stop processing items in the queue and resume it. Thus, when stopped you can optionally change the order of items, add items, or remove items from the queue.

Why tea ???

Because I love tea.

Usage

Usage is linked directly to manipulation of simple text files. You use text files to make the queue, to add or remove items from it, to change priority of items. UNIX is the way.

I give some examples in this section. Take into account teaqueue-server takes queue.txt in the current directory by default as queue file modifying it when interrupted to remove processed items from it. By default, processed items goes to done.txt. This files serves to remove processed items from queue.txt when the queue is interrupted and serves as a log file for processed items.

Video batch transcoding

Firstly list filenames in the queue and serve the queue:

find -type f -iname '*.webm' > queue.txt
teaqueue-server

Then ask teaqueue-client to process each files with a transcoding command:

teaqueue-client transcode_my_file.sh

This is all if you just want to process each file and do not interrupt it. If you want to interrupt processing of the queue, modify it and resume it, first press Ctrl-C in the teaqueue-server console and modify queue.txt accordingly:

find ../another_dir -type '*.webm' >> queue.txt

Or reorder some items:

vim queue.txt

Then run again teaqueue-server, clients will continue to ask new items from the queue:

teaqueue-server

transcode_my_file.sh is a called a worker and is a simple script which calls ffmpeg or another program to convert your files. You can find examples of workers inside the worker_examples directory.

Write a worker

In order to be useful you certainly have to write a worker script. Writing a worker is simple, all you have to do is to store the standard input line and work with it.

Echo worker

For example, a useless worker, but simple to understand the principle is just a worker echo each item it receive:

#!/bin/sh
line="$(cat)"
echo "Printing line '$line' and sleeping $1 seconds"
sleep $1

You can use it with a teaqueue-client:

teaqueue-client ./echo_sleep.sh 10

Remote files

If you want to be able to use teaqueue-client on another host than the teaqueue-server you can download the files (sharing it on the server side is done through anything you want) from the host, then work on it and push back the transformed files. Just write the worker in consequence:

#!/bin/sh
filename="$(cat)"
echo "$filename" | nc -q0 serverin 1338 > file
transcode.sh file > fileout
(echo "$filename"; cat fileout) | nc -q0 serverout 1338

This example assume you share your files with simple netcat commands, but you can adapt with anything you want: wget, scp, etc.

History

Normally, if you have a worker writing to standard output every file processed, you could write something like this:

find -iname '*.ext' > queue.txt
cat queue.txt | worker > done.txt
# here you can interrupt the service at any time (Ctrl-C/SIGTERM)
comm -23 queue.txt done.txt > remaining_queue.txt

But if this is something you want to do again and again, it might be convenient to write a little wrapper to do it for us. This is how teaqueue was born.

Limitations

teaqueue is not perfect and does not try to be. There is no security and some rare race conditions.

Known race condition:

If someone interrupts teaqueue-server just after the line has been sent but before it gets printed out, the line will not be filtered in the queue.txt file. Thus it will be send again to the next demanding client.

License

Unless specified otherwise, this project is licensed under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. You should have received a copy of the GNU General Public License along with this program. If not, see <https://opensource.org/licenses/GPL-3.0> or <http://www.gnu.org/licenses/>.

SPDX-License-Identifier: GPL-3.0+

Contact

developer: vg
mail: vg@devys.org