...\" $Id: pipeline.man4,v 1.3 2004/01/21 20:43:05 dgc Exp $ '\" te .TH "pipeline" "1" "27 Sep 2003" "" "dgc" .SH "NAME" pipeline \- logical data pipeliner .SH "SYNOPSIS" .PP \fBpipeline\fR \fI[-cdesvV]\fR \fI[-i file]\fR \fI[-o file]\fR \fI[--]\fR \fB[-c] cmdA [{conj} [-c] cmdB [...]]\fR .PP conjunctions are: \fI||\fR \fI&&\fR \fI|\fR \fI?\fR \fI-or\fR \fI-and\fR \fI-then\fR \fI-if\fR .SH "DESCRIPTION" .PP \fBPipeline\fR is a master control program for a data flow pipeline, in the manner of shell. .PP \fBPipeline\fR essentially combines the Bourne shell notions of the pipeline: .IP cmdA | cmdB | cmdC .PP and of the logical command list: .IP cmdA || cmdB && cmdC .PP We say that these are combined because \fBpipeline\fR \fIalways\fR passes a program's stdout to the next program's stdin, akin to the shell's pipeline. The difference is that stages of the pipeline can be excluded from execution based on the exit status of previous stages, as in the logical command list. Stages can also be excluded if no input is present from the previous stage. .PP \fBPipeline\fR reads from its command line a sequence of pipelined stages. Each stage's exit status produces a \fItruth state\fR. The \fIor\fR and \fIand\fR conjunctions decide, based on this truth state, whether the next stage of the pipeline should execute. .PP Take, for example, this simple command: .IP pipeline cmdA || cmdB && cmdC .PP or, if you want to escape shell syntax mutilation: .IP pipeline cmdA -or cmdB -and cmdC .PP \fIcmdA\fR first executes, with \fIstdin\fR open to \fI/dev/null\fR. (To force it to read \fIstdin\fR from \fBpipeline\fR's \fIstdin\fR, use the \fI-i -\fR option.) It does whatever it does; the output is accumulated inside the \fBpipeline\fR process. When \fIcmdA\fR completes, its exit status is evaluated. If it failed, \fIcmdB\fR is launched, and \fIcmdB\fR receives on stdin the accumulated stdout of \fIcmdA\fR. If \fIcmdA\fR succeeded, then \fIcmdB\fR is skipped, and stdout is passed to stdin of \fIcmdC\fR instead. If \fIcmdB\fR runs, and succeeds, that also is passed to \fIcmdC\fR. .PP The logical flow is exactly like Bourne shell's. Subject to that flow, the pipeline works similarly, with one significant difference: Bourne's approach is to immediately launch \fIcmdC\fR, \fIcmdB\fR, and \fIcmdA\fR, in that order. \fBPipeline\fR will never start \fIcmdB\fR if \fIcmdA\fR fails. This can be a useful model. .PP The \fI||\fR and \fI-or\fR conjunctions are identical in function. \fI||\fR exists to make \fBpipeline\fR's syntax read like Bourne's, but if you're running \fBpipeline\fR \fIfrom\fR Bourne, this can prove troublesome. \fI-or\fR is provided as an alternate. \fI&&\fR and \fI-and\fR are likewise identical, as are \fI|\fR and \fI-then\fR. .PP The \fI|\fR or \fI-then\fR conjuction is a simple pipe. It disregards previous truth state; it always runs; and it always receives the previous stage's output as input. .PP The \fI?\fR or \fI-if\fR conjunction disregards truth state, but considers instead whether the previous stage produced any output. If output was produced, the following stage will run. If no output was produced, the stage is ignored. (By implication, all following stages are also ignored, and it only makes sense to use one \fI-if\fR in a single invocation of \fBpipeline\fR.) .SH "OPTIONS" .PP The following options are supported: .IP "\fB-d\fR" 6 Produce \fIdebugging\fR (or tracing) output: logical flow, operation, truth state, commands run, exit statuses. .IP "\fB-c\fR" 6 Causes all stages to be run using \fIsh -c\fR instead of being run directly via \fIexecv()\fR. This allows you to use a variety of shell syntax in your pipeline stages, including input and output redirections. .IP "\fB-e\fR" 6 Merges the \fIstderr\fR stream with \fIstdout\fR for each stage. This is equivalent to running with \fI-c\fR, and adding \fB2>&1\fR to each stage. .IP "\fB-s\fR" 6 \fISuppress\fR output, if the final truth state is \fIfalse\fR. .IP "\fB-v\fR" 6 \fIInvert\fR the final truth state, producing \fItrue\fR for \fIfalse\fR and vice versa. This occurs immediately before output is produced or suppressed, so \fI-s\fR and \fI-sv\fR have precisely opposite effects. .IP "\fB-V\fR" 6 Shows \fIversion\fR information -- the Id: tag(s) from CVS. .IP "\fB-i file\fR" 6 Source initial input data from \fIfile\fR. If \fIfile\fR is "-", source from \fIstdin\fR. Without any \fI-i\fR option, initial input to the first stage is from \fI/dev/null\fR. .IP "\fB-o file\fR" 6 Sink final output data to \fIfile\fR, rather than to \fIstdout\fR. (If \fIfile\fR is "-", \fIstdout\fR is used anyway.) .PP Additionally, a \fI-c\fR can be given as the first term of any stage, and that stage will be executed via \fIsh -c\fR. Other stages will run normally. .SH "EXAMPLES" .PP Here's a simple \fIcrontab\fR entry for conditional mailing: .nf 0 * * * * /path/to/pipeline -sv somecommand -or mailx -s "oopsie" admin .fi .PP This executes \fBsomecommand\fR. If its exit status is true (success), then nothing happens, and any output produces is suppressed. But if it fails, output is released to \fBmailx\fR, which sends a nice and tidy message to \fIadmin, in place of the gruesome default \fBcron\fR mail. ]) .PP Maybe sometime I'll add more examples. .SH "EXIT STATUS" .PP The following exit values are returned: .IP "\fB0\fR" 6 By some path, the pipeline completed successfully. If all conjunctions in the pipeline were \fIand\fR conjuctions, then everything completed successfully. If some were \fIor\fR conjunctions, then some commands might have failed; but their failure was anticipated. .IP "\fB2\fR" 6 Command syntax was incorrect; refer to usage (\fBpipeline -h\fR). .IP "\fB10\fR" 6 The input file could not be opened. .IP "\fB15\fR" 6 The output file could not be opened or created. .IP "\fB20\fR" 6 Some stage of the pipeline could not be executed. .SH "EMBED" \fBPipeline\fR is compiled with \fBembed\fR. To extract the full source tree (including \fBembed\fR itself), simply run: .nf pipeline -explode # for a random subdirectory in $cwd .fi or .nf pipeline -explode=/other/path # for a specific location .fi Or, to list the files \fBembed\fRded in the program, run: .nf pipeline -list .fi .SH "ERRATA" .PP \fBPipeline\fR used to read \fIstdin\fR by default, passing the input to the first-stage process, unless the \fI-n\fR option was given. This made sense and was consistent with the model of mirroring Bourne's chaining functionality, but in practice most pipelines didn't care about existing \fIstdin\fR data: they were only concerned with whatever the first stage process produced itself. .PP The result was that if one forgot to use the \fI-n\fR option, \fBpipeline\fR would sit around waiting for \fIstdin\fR to close before ever launching the first-stage process. In almost all cases, this behavior was simply irritating. It seemed a good principle that the behavior most often desired should be the easiest to obtain, so this design was reversed. Removing the \fI-n\fR option and making its behavior the default (with \fI-i -\fR as an override) makes \fBpipeline\fR a little less logical, but more in line with expectation, and hopefully less aggravating over all. .SH "LICENSE" .PP Copyright (c) 2003, David Champion .PP This is Free Software. It is released under the terms given in the accompanying file "LICENSE", which I cribbed from the Massachusetts Institute of Technology by way of http://www.opensource.org/licenses/mit-license.php. .SH "BUGS" (\fIn\fR.) Creeping little problems that inhabit all software, no matter how conscientiously you test it. .SH "SEE ALSO" .PP \fBsh\fR(1), \fBcron\fR(1M, 8) .SH "HAIKU" Bourne shell conditions .br Are not always sufficient. .br \'Specially with \fBcron\fR. .br ...\" ...\" $Log: pipeline.man4,v $ ...\" Revision 1.3 2004/01/21 20:43:05 dgc ...\" Manual updated with a LICENSE section, since I'm "releasing" this ...\" (i.e., telling someone outside my small world about it) today. ...\" ...\" Revision 1.2 2003/12/01 04:03:47 dgc ...\" typo adj ...\" ...\" Revision 1.1 2003/11/30 07:31:14 dgc ...\" Troff form of manual now autogenerated from m4 source ...\" ...\" Revision 1.4 2003/11/29 10:11:19 dgc ...\" Removed -n; made its behavior default. See _ERRATA_ in the manual. ...\" ...\" Revision 1.3 2003/11/29 09:37:45 dgc ...\" Added -if conjunction -- scarily easy. Some miscellaneous doc fixes. ...\" ...\" Revision 1.2 2003/10/12 20:56:27 dgc ...\" Doc update ...\" ...\" Revision 1.1 2003/09/27 23:32:01 dgc ...\" + Added manual (pipeline.1) ...\" ...\" + Fixed exit-status bug ...\" ...\" + Fixed typos in usage. ...\" ...\"