shipper 1 shipper automatic drop-shipping of project releases shipper -h -n -N -f -v buildrpms tarball shipper is a tool for shipping project releases. Its job is to make it possible for you to run the command shipper in the top-level directory of a project and have a release be properly exported to all the places that you normally deliver it — your personal website, Linux source code archive sites, and distribution submission queues. A second goal is to arrange your shipping process in such a way that metadata like your project version only have to be kept in one place and modified once per release. The overall goal is to reduce the friction cost of shipping releases to as near zero as possible. buildrpms is a helper script that builds source and binary RPMs from a specified tarball with a BuildRoot field. shipper also calls rpm2lsm 1 to do part of its work. As much as possible, shipper tries to deduce what it should do rather than requiring you to tell it. In order to do this, it relies on your project obeying standard GNU-like naming conventions. It also relies on being able to mine project metadata out of a package specfile. (Presently the only variety of package specfile supported is an RPM spec; this may change in the future, when we fully support shipping Debian packages.) In normal use, you need set only one configuration variable, which is the list of private destinations to ship to. You may also want to add some magic Keywords comments to your project specfiles. Once you have shipper up and running, you can experiment with more advanced features such as having the program generate project web pages for you. shipper pushes deliverables out to destinations. Deliverables include: source tarballs, source zip archives, source RPMs, binary RPMs, ChangeLog files, README files, LSM files, and various other project metadata files. Destinations include both private destinations like websites, FTP archive sites and mailing lists, and public channels like ibiblio, freshmeat.net, and the submission queues for various well-known operating-system distributions. The shipper framework is extensible and it is relatively easy to add new channel types and new deliverables; in the future, we hope to support (for example) Debian packages as deliverables and SourceForge as a channel. shipper's first step is to find the project name and version, then to check that the minimum set of files that shipper requires to continue is in place. To start with, shipper needs a source tarball and a specfile. Once it knows those are in place, it can extract various pieces of information it will need to do its real work. It also reads in a handful of configuration variables. The -N (nobuild) option causes it to dump all configuration values and stop there. The first real work that gets done is finding or building local deliverables. These are either generated deliverables (like RPMs) that can be rebuilt automatically, or or stock deliverables (like a README file) that have to be changed by hand. shipper rebuilds any generated deliverable that doesn't exist when it starts up. Building local deliverables is separated from uploading because it means that you can stop and inspect what you're going to ship before committing to an upload. The -n (noupload) option stops before uploading, leaving all local deliverables in place but displaying the exact upload commands that would have been used to ship them. The -f (force) option forces a rebuild of all generated deliverables, even those that already exist. The command shipper -f -n will show you exactly what shipper would do for a real upload. Once all local deliverables have been built, shipper can begin uploading files and posting announcements. It does private destinations first, then public channels. This means, for example, that if you give shipper your personal website as a destination, the website will get updated each time before any submissions or announcements are sent to public sites like ibiblio.org or freshmeat.net. When uploads are complete, shipper cleans up after itself by deleting any deliverables it created for this run. Deliverables that were found and up to date are not removed. Finally, note that shipper makes one important assumption about the structure of your website(s). Beneath each directory in your destinations list, there will be one subdirectory for each project, with the directory leaf name being the same as the project. Thus, for example, if you have three projects named ruby, diamond and sapphire, and your personal site is at gemstones.net:/public/www/precious/, shipper will expect to be able to drop deliverables in three directories gemstones.net:/public/www/precious/ruby, gemstones.net:/public/www/precious/diamond/, and gemstones.net:/public/www/precious/sapphire/. Note that shipper will not create these project directories for you if they're missing; this is deliberate, so that uploads to sites that are not prepared for them will fail noisily. The behavior of shipper depends on a handful of internal variables. Some of these variables have defaults computed at startup time. All can be set or overridden in the per-user ~/.shipper file, and overridden in any per-project .shipper file. Both files are Python code and the syntax of variable settings is Python's. If a variable is set in a config file, that value is locked in (except for the destinations variable which can be appended to from a specfile, see below) Variables that are not set in a config file may be set by the values of fields in your project specfile. For basic use, it is only necessary to set one such variable: destinations, the list of destinations to ship to. Normally you'll set this globally, pointing all your projects at your main distribution website, in your ~/.shipper file; it is also possible to add destinations on a per-project basis by giving a comma-separated list in a #Destinations: comment in the specfile. You can set the variable in a per-project .shipper to ignore your global destination list. The first thing shipper looks for is a specfile in the current directory; there must be exactly one. It extracts the project name from the Name field. Next step is to find the project version (the variable package). This is extracted from the specfile, or by looking for a makefile macro with a name beginning with VERS; if the value of that macro is a shell command wrapped in $(shell ...), it is executed and the output is captured to yield the version. If both versions are present, they are consistency-checked. shipper gets most of the rest of the data it uses to decide what to do from headers in the specfile. The following table lists all the variables and their corresponding specfile fields. shipper uses the RPM spec file fields: the Debian entries are informational only. Variable RPM specfile field Debian specfile field Meaning destinations #Destinations: XBS-Destinations: A list of remote directories to ship to using scp 1 . Each location is a place to drop deliverables: either a [user@]site:path destination that scp 1 can use, or an FTP url that lftp 1 can use. Note that actual project directories are computed by appending the value of package to the destination you're shipping to. There is no default.. If you do not set this variable, shipper will ship only to public channels. channels - - The list of public channels to be shipped to after the private channels in the destination list. You can disable one or more of these in a config file by calling the function disable(); for example with disable('freshmeat'). whoami - - A plausible email address for the user. If not specified in the config file, it's generated from $USERNAME and $HOSTNAME. date - - The program's startup time. This can be used in the web page and email announcement templates. You can use the Python function time.strftime("...") in your ~/.shipper file to format this date to your taste. If you don't set this in the config file, the program will set it for you. indextemplate - - Template HTML from which to generate index.html for shipping. There is a default which generates a very simple page containing a title, a date, and a table listing downloadable resources. This is used when shipping to a web directory, if no index page exists when shipper is run. mailtemplate - - Template text from which to generate the file ANNOUNCE.EMAIL to be shipped to destinations that are mailto URLs. There is a default which generates a very simple email containing a subject, a pointer to the project web page, and the last entry in the project changelog. package Name: Package: Project name, used to generate the stem part of the names of RPMs and other deliverables that shipper builds. If the specfile is a Debian control file, the Debian-specific part of the version number (after the dash) is removed. version Version: Version: Project version, used in generating the names of RPMs and other deliverables that shipper builds. homepage URL: XBS-Home-Page: Project home page URL. Used when generating project announcements. arch BuildArch: Architecture: Build architecture. If this field is noarch, noarch rather than binary RPMs will be built. keywords #Keywords: XBS-Keywords: Topic keywords. Used when generating LSM files. freshmeat_name #Freshmeat-Name: XBS-Freshmeat-Name: Freshmeat shortname, used in generating freshmeat.net announcements. If this isn't present, it defaults to the project name; you only need to set it if they differ. summary Summary Description: The one-line project summary field from your specfile. description %description Description: The Description field from your specfile. changelog ChangeLog or %changelog - If a ChangeLog file exists in the project directory, its entire contents. Otherwise, if it exists, the entire changelog section from the specfile. lastchange ChangeLog or %changelog - If the source of your changlog was your specfile, this is the most recent entry from your changelog without its date/author/release header. If the source was Changelog, a line of text directing the user to see the ChangeLog file. This becomes the Changes field in your freshmeat.net announcement, and freshmeat.net doesn't like the bulleted format of GNU ChangeLog entries. resourcetable - - The HTML table of links to downloadable resources. This variable is only computed if the index page is built. Any setting of it in the startup files is ignored. All these variables are available for substitution at the time a web page or email announcement is generated. In general, any variable you set in your ~/.shipper file will be available at the time the web page or email announcement is generated. Use the Python "%(variable)s" syntax, not shell-substitution syntax. The following files are considered stock deliverables and may be shipped if they are present when shipper starts up: File Explanation README Project roadmap file. tarball The current source tarball, that is the file named ${package}-${version}.tar.gz. zipfile The current source zip archive, that is the file named ${package}-${version}.zip. NEWS Project news file. ChangeLog Project change log. HISTORY Project history file. BUGS Project bug list. TODO Current to-do list. *.html Any files with an .html extension will be shipped to all website destinations. Here are the generated deliverables that shipper will build and ship, if they don't exist when it starts up. Any of these that are created will be deleted after a successful upload. Type Explanation index.html An index web page, to be shipped to any website destination. RPMs Source and either binary or noarch RPMs. LSM If the ibiblio channel is enabled, shipper will generate a Linux Software Map file for it. CHANGES If there is no ChangeLog file but there was a %changelog in your specfile, shipper will generate a CHANGES from the changelog entries in the specfile and ship that. ANNOUNCE.FRESHMEAT If there is no ANNOUNCE.FRESHMEAT file, shipper will generate one. It will be a job card that can be fed to freshmeat.net's XML-RPC interface via freshmeat-submit1. ANNOUNCE.EMAIL If there is no ANNOUNCE.EMAIL file, shipper will generate one to be emailed to destinations that are mailto URLs. In operation, shipper walks through a list of destinations, building the required deliverables for each one and performing the required shipping actions to push them out to the destination. Here are the channel types shipper knows about: Channel Type Deliverables Specified by Explanation ibiblio tarball, RPMs, LSM file - If the ibiblio channel is enabled (it is by default), shipper will attempt to ship a source tarball, RPMs, and an an LSM file to ibiblio.org via FTP. The LSM file will be automatically generated. redhat RPMs - If the Red Hat channel is enabled (it is by default), shipper will attempt to ship source and binary RPMs to the Red Hat submission directory via FTP. freshmeat ANNOUNCE.FRESHMEAT - If the freshmeat channel is enabled (it is by default), shipper will attempt to post a release announcement on freshmeat.net using freshmeat-submit1. The announcement will include URLs for whichever of the following deliverables are shipped, using the URL field from your specfile: tarball, zipfile, RPMs, CHANGES. The user will be prompted for a Freshmeat release-focus. This announcement is generated into the local deliverable ANNOUNCE.FRESHMEAT. Generic Web site README, tarball, zipfile, RPMs, CHANGES, NEWS, HISTORY, *.html, BUGS, TODO. scp destination ([user@]host:dir) This channel type represents a website. shipper uses scp1 to put deliverables on websites. If the user part of the scp destination is absent, it will be taken from the environment variable USERNAME. No generic Web sites are shipped to by default. You must declare them by putting scp destinations in the destinations variable. Generic FTP site tarball, RPMs FTP URL Old-fashioned FTP site with no metadata. The FTP URL is parsed to get the sitename and directory where deliverables should be dropped. The FTP username to be used will be taken from the environment variable USERNAME. The FTP password will be looked up in your ~/.netrc file. No generic FTP sites are shipped to by default. You must declare them by putting FTP urls in the destinations variable. Email address ANNOUNCE.EMAIL mailto URL The contents of the generated ANNOUNCE.EMAIL file is emailed to each email address specified as a channel. No email channels are set up by default. You must declare them by putting mailto: urls in the destinations variable. rsync unit SRPM rsync address ([user@]host::unit) An SRPM is shipped to each destination that is rcognized as an rsync address (by the double colon). No rsync channels are set up by default. You must declare them by putting rsync addresses in the destinations variable. The -n option of shipper suppresses uploads, just building all deliverables locally. The -N option suppresses both uploads and builds, generating a configuration dumop instead. The -f option forces rebuilding of local deliverables even if they already exist. The -v option makes shipper chatty about what it's doing. The -h option prints a usage message and exits. The following variable definition in your makefile will ensure that the makefile version is derived from (and thus always consistent with) the specfile version. VERS=$(shell sed <*.spec -n -e '/Version: \(.*\)/s//\1/p') A makefile production like the following will allow you to type make release and be sure that all the deliverables shipper knows about will be rebuilt before being shipped. release: package-$(VERS).tar.gz package.html shipper -f You will want to change package to your project name. Note that you should not use this recipe if your project has its own (non-generated) index page, as the -f option will overwrite index.html. To make rpm1 build noarch rather than binary RPMs, insert the following header in your specfile: BuildArch: noarch Eric S. Raymond esr@thyrsus.com. The buildrpms script was originally by Sean Reifschneider. There is a project web page at http://www.catb.org/~esr/shipper/. The rsync channel type is untested. Shipping Debian packages should be supported. freshmeat-submit 1 , lftp 1 , rpm2lsm 1 , scp 1 , ssh 1 .