From 4f40dc60c125ad1e14071c234b67315fab6c6fc3 Mon Sep 17 00:00:00 2001 From: Rob Funk Date: Fri, 18 Jun 2004 04:17:07 +0000 Subject: Move shipper to dist-tools directory svn path=/trunk/; revision=3889 --- dist-tools/shipper/shipper.xml | 752 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 752 insertions(+) create mode 100644 dist-tools/shipper/shipper.xml (limited to 'dist-tools/shipper/shipper.xml') diff --git a/dist-tools/shipper/shipper.xml b/dist-tools/shipper/shipper.xml new file mode 100644 index 00000000..765f0f54 --- /dev/null +++ b/dist-tools/shipper/shipper.xml @@ -0,0 +1,752 @@ + + + +shipper +1 + + + shipper +automatic drop-shipping of project releases + + + + + shipper + -h + -n + -N + -f + -v + + + buildrpms + tarball + + + +Description + +shipper is a tool for shipping +project releases. Its job is to make it possible for you to run the +command shipper in the top-level directory of a +project and have a release be properly exported to all the places that +you normally deliver it — your personal website, Linux source +code archive sites, and distribution submission queues. A second goal +is to arrange your shipping process in such a way that metadata like +your project version only have to be kept in one place and modified +once per release. The overall goal is to reduce the friction cost +of shipping releases to as near zero as possible. + +buildrpms is a helper script that +builds source and binary RPMs from a specified tarball with a +BuildRoot field. shipper also calls + +rpm2lsm +1 to do part of its work. + +As much as possible, shipper tries to +deduce what it should do rather than requiring you to tell it. In +order to do this, it relies on your project obeying standard GNU-like +naming conventions. It also relies on being able to mine project +metadata out of a package specfile. (Presently the only variety of +package specfile supported is an RPM spec; this may change in the future, +when we fully support shipping Debian packages.) + +In normal use, you need set only one configuration variable, +which is the list of private destinations to ship to. You may also +want to add some magic Keywords comments to your +project specfiles. Once you have shipper +up and running, you can experiment with more advanced features +such as having the program generate project web pages for you. + + +Theory of Operation + +shipper pushes +deliverables out to +destinations. Deliverables include: source tarballs, +source zip archives, source RPMs, binary RPMs, ChangeLog files, README +files, LSM files, and various other project metadata files. Destinations +include both private destinations like websites, FTP +archive sites and mailing lists, and public +channels like ibiblio, freshmeat.net, and the submission +queues for various well-known operating-system distributions. The +shipper framework is extensible and it is relatively easy to add new +channel types and new deliverables; in the future, we hope to support +(for example) Debian packages as deliverables and SourceForge as a +channel. + +shipper's first step is to find the +project name and version, then to check that the minimum set of files that +shipper requires to continue is in place. +To start with, shipper needs a source +tarball and a specfile. Once it knows those are in place, it +can extract various pieces of information it will need to do its +real work. It also reads in a handful of configuration variables. +The -N (nobuild) option causes it to dump all configuration values and +stop there. + +The first real work that gets done is finding or building local +deliverables. These are either generated +deliverables (like RPMs) that can be rebuilt automatically, +or or stock deliverables (like a README file) +that have to be changed by hand. shipper +rebuilds any generated deliverable that doesn't exist when it starts +up. Building local deliverables is separated from uploading because +it means that you can stop and inspect what you're going to ship +before committing to an upload. + +The -n (noupload) option stops before uploading, leaving all +local deliverables in place but displaying the exact upload commands +that would have been used to ship them. The -f (force) option forces +a rebuild of all generated deliverables, even those that already +exist. The command shipper -f -n will show you +exactly what shipper would do for a real +upload. + +Once all local deliverables have been built, +shipper can begin uploading files and +posting announcements. It does private destinations first, then public +channels. This means, for example, that if you give +shipper your personal website as a destination, the +website will get updated each time before +any submissions or announcements are sent to public sites like +ibiblio.org or freshmeat.net. + +When uploads are complete, shipper +cleans up after itself by deleting any deliverables it created for +this run. Deliverables that were found and up to date are not +removed. + +Finally, note that shipper makes one +important assumption about the structure of your website(s). Beneath +each directory in your destinations list, there +will be one subdirectory for each project, with the directory leaf +name being the same as the project. Thus, for example, if you have +three projects named ruby, diamond and sapphire, and your personal +site is at gemstones.net:/public/www/precious/, +shipper will expect to be able to drop +deliverables in three directories +gemstones.net:/public/www/precious/ruby, +gemstones.net:/public/www/precious/diamond/, and +gemstones.net:/public/www/precious/sapphire/. +Note that shipper will not create these +project directories for you if they're missing; this is deliberate, so +that uploads to sites that are not prepared for them will fail +noisily. + + + +How Shipper Deduces What To Do + +The behavior of shipper depends on a handful of internal +variables. Some of these variables have defaults computed at startup +time. All can be set or overridden in the per-user +~/.shipper file, and overridden in any +per-project .shipper file. Both files are Python +code and the syntax of variable settings is Python's. + +If a variable is set in a config file, that value is locked in +(except for the destinations variable which can be +appended to from a specfile, see below) Variables that are +not set in a config file may be set by the values +of fields in your project specfile. + +For basic use, it is only necessary to set one such variable: +destinations, the list of destinations to ship to. +Normally you'll set this globally, pointing all your projects at your +main distribution website, in your ~/.shipper +file; it is also possible to add destinations on a per-project basis +by giving a comma-separated list in a #Destinations: comment in the +specfile. You can set the variable in a per-project +.shipper to ignore your global destination +list. + +The first thing shipper looks for is a specfile in the +current directory; there must be exactly one. It extracts the project +name from the Name field. Next step is to find the project version +(the variable package). This is extracted from the +specfile, or by looking for a makefile macro with a name +beginning with VERS; if the value of that macro is a shell command +wrapped in $(shell ...), it is executed and the output is captured to +yield the version. If both versions are present, they are +consistency-checked. + +shipper gets most of the rest of the +data it uses to decide what to do from headers in the specfile. +The following table lists all the variables and their corresponding +specfile fields. shipper uses the RPM spec +file fields: the Debian entries are informational only. + + + + + +Variable +RPM specfile field +Debian specfile field +Meaning + + + + +destinations +#Destinations: +XBS-Destinations: + +A list of remote directories to ship to using + +scp 1 +. Each location is a place to drop deliverables: +either a [user@]site:path destination that + +scp 1 + can use, or an FTP url that + +lftp 1 + +can use. Note that actual project directories are computed by +appending the value of package to +the destination you're shipping to. + +There is no default.. If you +do not set this variable, shipper will +ship only to public channels. + + + +channels +- +- + +The list of public channels to be shipped to after the private +channels in the destination list. You can disable +one or more of these in a config file by calling the function +disable(); for example with +disable('freshmeat'). + + + +whoami +- +- + +A plausible email address for the user. If not specified in the +config file, it's generated from +$USERNAME and $HOSTNAME. + + + +date +- +- + +The program's startup time. This can be used in the web page and +email announcement templates. + +You can use the Python function time.strftime("...") in your +~/.shipper file to format this date to your +taste. If you don't set this in the config file, the program will +set it for you. + + + +indextemplate +- +- + +Template HTML from which to generate index.html for shipping. There is a +default which generates a very simple page containing a title, a +date, and a table listing downloadable resources. This is used when +shipping to a web directory, if no index page exists when shipper +is run. + + + +mailtemplate +- +- + +Template text from which to generate the file ANNOUNCE.EMAIL to be +shipped to destinations that are mailto URLs. There is a default which +generates a very simple email containing a subject, a pointer to the +project web page, and the last entry in the project changelog. + + + +package +Name: +Package: + +Project name, used to generate the stem part of the names of RPMs and +other deliverables that shipper +builds. If the specfile is a Debian control file, the Debian-specific +part of the version number (after the dash) is removed. + + + +version +Version: +Version: + +Project version, used in generating the names of RPMs and +other deliverables that shipper +builds. + + + +homepage +URL: +XBS-Home-Page: + +Project home page URL. Used when generating project +announcements. + + + +arch +BuildArch: +Architecture: + +Build architecture. If this field is noarch, +noarch rather than binary RPMs will be built. + + + +keywords +#Keywords: +XBS-Keywords: + +Topic keywords. Used when generating LSM files. + + + +freshmeat_name +#Freshmeat-Name: +XBS-Freshmeat-Name: + +Freshmeat shortname, used in generating freshmeat.net +announcements. If this isn't present, it defaults to the project +name; you only need to set it if they differ. + + + +summary +Summary +Description: + +The one-line project summary field from your specfile. + + + +description +%description +Description: + +The Description field from your specfile. + + + +changelog +ChangeLog or %changelog +- + +If a ChangeLog file exists in the project +directory, its entire contents. Otherwise, if it exists, +the entire changelog section from the specfile. + + + +lastchange +ChangeLog or %changelog +- + + +If the source of your changlog was your specfile, this is the +most recent entry from your changelog without +its date/author/release header. If the source was Changelog, a +line of text directing the user to see the ChangeLog file. +This becomes the Changes field in your freshmeat.net announcement, +and freshmeat.net doesn't like the bulleted format of GNU ChangeLog +entries. + + + +resourcetable - - + +The HTML table of links to downloadable resources. This +variable is only computed if the index page is built. Any setting +of it in the startup files is ignored. + + + + + + +All these variables are available for substitution at the time a +web page or email announcement is generated. In general, any variable +you set in your ~/.shipper file will be available +at the time the web page or email announcement is generated. Use the +Python "%(variable)s" syntax, not shell-substitution syntax. + + + +Finding and Building Local Deliverables + +The following files are considered stock deliverables and may be +shipped if they are present when shipper +starts up: + + + + + +File +Explanation + + + + +README + +Project roadmap file. + + + +tarball + +The current source tarball, that is the file named ${package}-${version}.tar.gz. + + + +zipfile + +The current source zip archive, that is the file named ${package}-${version}.zip. + + + +NEWS + +Project news file. + + + +ChangeLog + +Project change log. + + + +HISTORY + +Project history file. + + + +BUGS + +Project bug list. + + + +TODO + +Current to-do list. + + + +*.html + +Any files with an .html extension will be shipped to all +website destinations. + + + + + + +Here are the generated deliverables that +shipper will build and ship, if they don't +exist when it starts up. Any of these that are created will be +deleted after a successful upload. + + + + + +Type +Explanation + + + + +index.html + +An index web page, to be shipped to any website destination. + + + +RPMs + +Source and either binary or noarch RPMs. + + + +LSM + +If the ibiblio channel is enabled, +shipper will generate a Linux Software Map +file for it. + + + +CHANGES + +If there is no ChangeLog file but there was a %changelog in your +specfile, shipper will generate a CHANGES +from the changelog entries in the specfile and ship that. + + + +ANNOUNCE.FRESHMEAT + +If there is no ANNOUNCE.FRESHMEAT file, +shipper will generate one. It will be a +job card that can be fed to freshmeat.net's XML-RPC interface via +freshmeat-submit1. + + + + +ANNOUNCE.EMAIL + +If there is no ANNOUNCE.EMAIL file, shipper +will generate one to be emailed to destinations that are mailto URLs. + + + + + + + +Shipping to Destinations +In operation, shipper walks through a +list of destinations, building the required deliverables for each one and +performing the required shipping actions to push them out to the +destination. Here are the channel types +shipper knows about: + + + + + + + + + +Channel Type +Deliverables +Specified by +Explanation + + + + +ibiblio +tarball, RPMs, LSM file +- + +If the ibiblio channel is enabled (it is by default), +shipper will attempt to ship a source +tarball, RPMs, and an an LSM file to ibiblio.org via FTP. The LSM +file will be automatically generated. + + + +redhat +RPMs +- + +If the Red Hat channel is enabled (it is by default), +shipper will attempt to ship source +and binary RPMs to the Red Hat submission directory via FTP. + + + +freshmeat +ANNOUNCE.FRESHMEAT +- + +If the freshmeat channel is enabled (it is by default), +shipper will attempt to post a release +announcement on freshmeat.net using +freshmeat-submit1. +The +announcement will include URLs for whichever of the following +deliverables are shipped, using the URL field from your specfile: tarball, +zipfile, RPMs, CHANGES. The user will be +prompted for a Freshmeat release-focus. This announcement is +generated into the local deliverable ANNOUNCE.FRESHMEAT. + + + +Generic Web site +README, tarball, zipfile, RPMs, CHANGES, NEWS, HISTORY, *.html, +BUGS, TODO. +scp destination ([user@]host:dir) + +This channel type represents a website. +shipper uses +scp1 +to put deliverables on websites. If the user part of the scp +destination is absent, it will be taken from the environment variable +USERNAME. + +No generic Web sites are shipped to by default. You must declare +them by putting scp destinations in the destinations +variable. + + + +Generic FTP site +tarball, RPMs +FTP URL + +Old-fashioned FTP site with no metadata. The FTP URL is parsed +to get the sitename and directory where deliverables should be dropped. The +FTP username to be used will be taken from the environment variable +USERNAME. The FTP password will be looked up in your +~/.netrc file. + +No generic FTP sites are shipped to by default. You must +declare them by putting FTP urls in the +destinations variable. + + + +Email address +ANNOUNCE.EMAIL +mailto URL + +The contents of the generated ANNOUNCE.EMAIL file is emailed to +each email address specified as a channel. + +No email channels are set up by default. You must +declare them by putting mailto: urls in the +destinations variable. + + + +rsync unit +SRPM +rsync address ([user@]host::unit) + +An SRPM is shipped to each destination that is rcognized as +an rsync address (by the double colon). + +No rsync channels are set up by default. You must +declare them by putting rsync addresses in the +destinations variable. + + + + + + + +Command-line Options + +The -n option of shipper suppresses +uploads, just building all deliverables locally. The -N option +suppresses both uploads and builds, generating a configuration dumop +instead. The -f option forces rebuilding of local deliverables even +if they already exist. The -v option makes +shipper chatty about what it's doing. The +-h option prints a usage message and exits. + + + +Hints and Tips +The following variable definition in your makefile will ensure +that the makefile version is derived from (and thus always consistent +with) the specfile version. + + +VERS=$(shell sed <*.spec -n -e '/Version: \(.*\)/s//\1/p') + + +A makefile production like the following will allow +you to type make release and be sure that all +the deliverables shipper knows about +will be rebuilt before being shipped. + + +release: package-$(VERS).tar.gz package.html + shipper -f + + +You will want to change package to your +project name. Note that you should not use this recipe if your +project has its own (non-generated) index page, as the -f option will +overwrite index.html. + +To make + +rpm1 + +build noarch rather than binary RPMs, insert the following header in +your specfile: + + +BuildArch: noarch + + + +Author +Eric S. Raymond esr@thyrsus.com. The buildrpms +script was originally by Sean Reifschneider. + +There is a project web page at +http://www.catb.org/~esr/shipper/. + + +Bugs +The rsync channel type is untested. Shipping Debian packages +should be supported. + + +See Also + + + +freshmeat-submit +1 +, + +lftp +1 +, + +rpm2lsm +1 +, + +scp +1 +, + +ssh +1 +. + + + + -- cgit v1.2.3