off the stackhttp://off-the-stack.moorman.nu/2014-06-02T00:00:00+02:00import sh2014-06-02T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2014-06-02:2014-06-02-import-sh.html<p>A small python library which seems to work rather well when it comes to
shelling out is <a class="reference external" href="http://amoffat.github.io/sh/">sh</a>. It provides a handy and lightweight abstraction
above the means already available within python.</p>
<p>The features I found most useful:</p>
<ul class="simple">
<li>easily check for program availability: I prefer to "instantiate" the
programs I want to use within the first few moments of execution and
in case a command is not available ... the script fails fast</li>
<li>it provides a nice abstraction around program execution and lets you
chain shell commands together in a more pythonic way (could be interesting
to combine it with something like the <a class="reference external" href="https://github.com/JulienPalard/Pipe">pipe</a> module or with <a class="reference external" href="http://www.reddit.com/r/programming/comments/26upaj/firstclass_shell_scripting_support_for_python/chuvq15">hy</a>)</li>
<li>you can provide arguments to shell commands just as you would for
ordinary functions</li>
<li>you can "bake" (prepare) commands and pass them around inside your
program</li>
<li>you can take advantage of generators for stdout handling and you can
background commands easily which is quite useful for wrapping longer
running processes</li>
</ul>
<p>Take the <a class="reference external" href="http://off-the-stack.moorman.nu/2014-03-21-builder-for-docker-images.html">small script on this page</a> for example. At the top of the script
sh.py is being imported with <code>import sh</code>. Then further down, the following
lines test for command availability and make the commands available as python
objects.</p>
<div class="highlight"><pre> <span class="n">git</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">git</span>
<span class="n">ls</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">ls</span>
<span class="n">docker</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">docker</span>
<span class="n">test</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">test</span>
</pre></div>
<p>In case the command in question does not exist, the library will throw an
exception. This enables us to check the environment and bail out with a
minimal amount of code.</p>
<p>Further down I can simply use this line</p>
<div class="highlight"><pre> <span class="n">digest</span> <span class="o">=</span> <span class="n">git</span><span class="p">(</span><span class="s">'ls-remote'</span><span class="p">,</span> <span class="n">repo</span><span class="p">,</span> <span class="s">'refs/heads/master'</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
</pre></div>
<p>to read out remote git repository information and work on the command's output
right away with a small amount of code. Of course I could use the <cite>subprocess</cite>
module too to achieve similar functionality but using <a class="reference external" href="http://amoffat.github.io/sh/">sh</a> tends to result in
cleaner implementations and adds more flexibility.</p>
<p>It surely is worth a shot.</p>
software translation: real messages vs keyword messages2014-04-27T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2014-04-27:2014-04-27-software-translation-real-messages-vs-keyword-messages.html<p>Translation systems are a vital part of software as they enable us to
create programs which are accessible for a broader audience. Paired with
other tools (e.g. locale based number and date formatting), they aid
internationalization of software. In case you want to read more about
"i18n" itself, I recommend this <a class="reference external" href="http://www-archive.mozilla.org/docs/refList/i18n/">mozilla article about i18n</a>.</p>
<p>It can be quite difficult to correctly translate an application as it can
require quite some linguistic knowledge. Furthermore, assumptions about the
visible text within the UI during the interface design phase can be problematic
too (e.g. text/word length, where and in which order values are injected into the
translation).</p>
<p>While those issues are fairly high level and UI/UX related, translation systems
are normally integrated within our application's core. To use them, you often
invoke them using a "source string" as a parameter directly inside the source
code.
They often translate the source string into the configured target language using
a predefined mapping often read from a set of translation files or a database.
The translation is then returned and used within the application.</p>
<p>There are two common types of translation definitions (besides the various
translation file formats), "keyword messages" and "real messages".
They differ in what the source string looks like/is about.</p>
<p>With "real messages", the source strings contain whole sentences (and
occasionally also some markup language artifacts).
Real messages are often written in plain english and are therefore often
directly suitable for end users capable of reading the source language.
A real message could be something as simple as "Click here to login." but could
also contain several lines of copy.</p>
<p>"keyword messages" on the other hand are defined by using identifiers as source
strings. Those identifiers are short strings which are not directly suitable
for end users as they merely name what they refer to and where they belong
inside the application.
An example of a keyword message could be "login_form.login_button_label".</p>
<p>Even though real messages seem to be used more often, I would argue, that
they are a messier way to handle translations.</p>
<p>Why? Simply because they mix presentation with business logic and are the
less semantically correct way to handle translations IMHO.</p>
<p>Of course, while coding up a model class for example, or implementing some
form logic it's too easy to just type out the labels of the fields as they
should appear on the screen right there in your code.
Without adding extra translation stuff you normally have some useful texts
immediately.
Then, in case translations are needed, you just wrap those english texts
throughout your code into calls to your translation system (e.g. <cite>_()</cite>).
Furthermore, everyone else does that ... why do it differently?</p>
<p>Well, where did you put those user visible strings again? Right inside
your business logic. The text visible to the end user is clearly presentation
related, yet it is now inside the guts of your application.
For labels that may not seem such a problem, but think of descriptions,
help texts, extra validation information... For example, think of a
description for a HTML form's input field, and you are required to make some
things bold or underlined. You would have to add some (probably HTML) markup
inside the business logic too.</p>
<p>Then, after a while, a typo is found within the application's interface while
using the default interface language.
Where would you have to adjust that typo? Within the translation file?
Within the business logic? Or for cleanliness both? You want a clean solution,
don't you? But then you would have to adjust all other translation files too...?</p>
<p>That the business logic related files need adjustments when a part of the UI
has to be adjusted means that they are coupled.</p>
<p>If keyword messages were used, only the translation files would have to be
changed, as the text (which had a typo in the example) is defined within
the translation files only. The business logic only contains identifiers which
merely point to the translation. Our user visible strings are neatly separated.
In case extra markup would be needed within the translation (yuk), it would
not litter our code. The business logic stays clean and concise.</p>
<p>Besides that, things that might receive the same translation (as often
the case with various UI buttons) can be required to have distinct
translations later on. The login button on the login form just isn't the
same as the login button within the applications header. The name field
of a product model just is not the same as the name field of a category
model. They all appear in different parts of the application, so distinct
keyword message based translations are more appropriate (even though it
requires some extra work). I would rather give them distinct keyword message
based labels explicitly in the first place than touch my business logic
related files when some part of the UI needs to change.</p>
drupal experiences2014-03-24T00:00:00+01:00Rico Moormantag:off-the-stack.moorman.nu,2014-03-24:2014-03-24-drupal-experiences.html<p>During the past couple of months I learned quite a few things about <a class="reference external" href="https://drupal.org">Drupal</a>.
I built several sites in Drupal 7, "themed" them and built modules to tie
things together.</p>
<p>For now I see the following good parts:</p>
<ul class="simple">
<li>several good, working modules which form the building blocks for
your site (e.g. views)</li>
<li><a class="reference external" href="https://github.com/drush-ops/drush">drush</a> is quite useful</li>
<li>hooks and the resulting extensibility</li>
<li>a lot of aspects of the system can be adjusted by defining
appropriate hooks within your theme or module</li>
<li>a lot of things are customizable using the admin interface</li>
<li>(module) code quality seems to be OK</li>
</ul>
<p>Of course there are also a few drawbacks:</p>
<ul class="simple">
<li>the "data model" for the content of your site can be defined and
adjusted using the admin interface; you can add content types and
assign fields a.s.o. by clicking around; while this could be nice
for some use-cases, those things are not really "source-controllable"
(modules do exist to work around this but it still feels bolted on)</li>
<li>a lot of other things (while defined within code) are saved within the
database too ... increases chance that DB interaction will be a problem
for your site</li>
<li>there is a lot of indirection and magic going on (also due to hooks),
which can make "solutions" complex and difficult to build and to follow</li>
<li>misses a more intuitive/easy admin UI</li>
<li>there are a couple of <a class="reference external" href="http://hojtsy.hu/blog/2011-may-19/drupals-multilingual-problem-why-t-wrong-answer">problems with the translation system</a></li>
<li>misses a good ORM/database migration tool</li>
</ul>
<p>The above is not exhaustive of course.</p>
<p>Furthermore, Drupal seems to make more sense, the more features a site
needs or the less focus on one functionality a site has (brochure,
community, publishing ...).</p>
<p>Even though it's not all roses (there is no perfect CMS), I am glad that
I know what this Drupal thing is about.</p>
builder for docker images2014-03-21T00:00:00+01:00Rico Moormantag:off-the-stack.moorman.nu,2014-03-21:2014-03-21-builder-for-docker-images.html<p>Docker provides a facility to easily create images based on a
recipe-like "Dockerfile". After the image is built using the <code>docker build</code>
command, it can be run as a container and also be distributed
using the public or a private docker registry.</p>
<p>While using the build command is easy enough to fit into a manual
workflow, automating things can be quite handy and open up new
possibilities (such as the <a href="http://blog.docker.io/2013/11/introducing-trusted-builds/">trusted builds</a> feature
available on <a href="https://www.docker.io/">docker.io</a>).</p>
<p>Of course there are also several open source projects out there already
providing some kind of automation on top of the build command itself
(<a href="https://github.com/dotcloud/stackbrew">stackbrew</a>, <a href="https://github.com/modcloth-labs/docker-build-worker">docker-build-worker</a>
...).</p>
<p>There are also projects which use a base image ("stack") and use
it to create a new image or provide heroku-like "slugs" in order to
reduce image file size (<a href="https://github.com/progrium/buildstep">buildstep</a>,
<a href="https://github.com/flynn/slugbuilder">slugbuilder</a>). Around that abstraction, then an automated
build facility is created (e.g. <a href="https://github.com/progrium/dokku" title="Dokku on github">dokku</a>, <a href="https://flynn.io/">flynn</a>).</p>
<p>Last but not least one could also use a CI/build server such as
<a href="http://jenkins-ci.org/">jenkins</a> or <a href="http://buildbot.net/">buildbot</a> to reduce manual labor (and
obviously reap other benefits as well).</p>
<p>However, implementing a builder myself will give me more insights into docker
and will be fun I think.</p>
<h2>first shot</h2>
<p>What follows is a "small" script which reads a YAML configuration file
containing image definitions, builds and tags them, and keeps track of
the image's source files/repository in order to build a new image only
in case there have been some changes.</p>
<div class="highlight"><pre><span class="c">#!/usr/bin/env python2</span>
<span class="sd">"""</span>
<span class="sd">A simple `docker build` wrapper, reading definitions from a configuration</span>
<span class="sd">file and using file and repository information to avoid builds in case</span>
<span class="sd">nothing changed.</span>
<span class="sd">Note:</span>
<span class="sd">- implementation is quite naive</span>
<span class="sd">- no special error handling/display ... using just plain exceptions</span>
<span class="sd">- not separated enough (side-effects vs pure)</span>
<span class="sd">- more advanced implementation could use distinct services</span>
<span class="sd"> which regularly check repositories/directories and distribute</span>
<span class="sd"> work to a set of workers/builders using a queuing solution</span>
<span class="sd"> (celery e.g.)</span>
<span class="sd">- git repo check is now based on prefix check for "git://" or</span>
<span class="sd"> "github.com" of the url (as docker does too inside utils/utils.go</span>
<span class="sd"> isGIT)</span>
<span class="sd">- using a branch is currently not supported within docker</span>
<span class="sd"> (see api.go) .... so we skip it here too (local clones</span>
<span class="sd"> are out of scope here even though they provide benefits</span>
<span class="sd"> https://github.com/dotcloud/docker/issues/3556#issuecomment-32624330)</span>
<span class="sd">- more advanced implementation could also provide tracking</span>
<span class="sd"> of branches/particular revisions (see pip.vcs functionality)</span>
<span class="sd">- no optimizations have been implemented to reduce overhead in</span>
<span class="sd"> building image from git repo (transfer to docker daemon)</span>
<span class="sd">- this was using the docker-py library at first ... but that had</span>
<span class="sd"> some issues on it's own so now the docker cli is used</span>
<span class="sd">- using the docker cli on the other hand means less moving parts</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">logging</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">re</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">import</span> <span class="nn">argh</span>
<span class="kn">from</span> <span class="nn">argh.decorators</span> <span class="kn">import</span> <span class="n">arg</span>
<span class="kn">from</span> <span class="nn">sqlalchemy</span> <span class="kn">import</span> <span class="n">create_engine</span><span class="p">,</span> <span class="n">Table</span><span class="p">,</span> <span class="n">Column</span><span class="p">,</span> <span class="n">String</span><span class="p">,</span> <span class="n">Integer</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.orm</span> <span class="kn">import</span> <span class="n">sessionmaker</span>
<span class="kn">from</span> <span class="nn">sqlalchemy.ext.declarative</span> <span class="kn">import</span> <span class="n">declarative_base</span>
<span class="kn">import</span> <span class="nn">yaml</span>
<span class="kn">import</span> <span class="nn">sh</span>
<span class="n">Base</span> <span class="o">=</span> <span class="n">declarative_base</span><span class="p">()</span>
<span class="k">class</span> <span class="nc">ImageState</span><span class="p">(</span><span class="n">Base</span><span class="p">):</span>
<span class="n">__tablename__</span> <span class="o">=</span> <span class="s">'image_state'</span>
<span class="nb">id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">Integer</span><span class="p">,</span> <span class="n">primary_key</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">,</span> <span class="n">unique</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">)</span>
<span class="n">image_id</span> <span class="o">=</span> <span class="n">Column</span><span class="p">(</span><span class="n">String</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">build</span><span class="p">(</span><span class="n">config_file</span><span class="p">,</span> <span class="n">state_dir</span><span class="p">,</span> <span class="n">clean</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Build docker images based on the definitions in `config_file`, keeping</span>
<span class="sd"> track of source changes in order to avoid rebuilds if nothing has changed.</span>
<span class="sd"> Args:</span>
<span class="sd"> config_file: the path of the yaml configuration file to read</span>
<span class="sd"> state_dir: the path of the directory to use for change tracking state</span>
<span class="sd"> clean: whether the state db should be cleaned before build</span>
<span class="sd"> Returns:</span>
<span class="sd"> None</span>
<span class="sd"> """</span>
<span class="n">logger</span> <span class="o">=</span> <span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Checking command line tools'</span><span class="p">)</span>
<span class="n">git</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">git</span>
<span class="n">ls</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">ls</span>
<span class="n">docker</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">docker</span>
<span class="n">test</span> <span class="o">=</span> <span class="n">sh</span><span class="o">.</span><span class="n">test</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Load configuration file {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">config_file</span><span class="p">))</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">config_file</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">config</span> <span class="o">=</span> <span class="n">yaml</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Configuration: {!r}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">config</span><span class="p">))</span>
<span class="n">db_url</span> <span class="o">=</span> <span class="s">'sqlite:///{}/.builder.state.db'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">state_dir</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Prepare state db connection {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">db_url</span><span class="p">))</span>
<span class="n">db_engine</span> <span class="o">=</span> <span class="n">create_engine</span><span class="p">(</span><span class="n">db_url</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Create tables in state db if needed'</span><span class="p">)</span>
<span class="n">Base</span><span class="o">.</span><span class="n">metadata</span><span class="o">.</span><span class="n">create_all</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="n">db_engine</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Setup the db session'</span><span class="p">)</span>
<span class="n">db</span> <span class="o">=</span> <span class="n">sessionmaker</span><span class="p">(</span><span class="n">bind</span><span class="o">=</span><span class="n">db_engine</span><span class="p">)()</span>
<span class="k">if</span> <span class="n">clean</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Cleaning state db as requested'</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">ImageState</span><span class="p">)</span><span class="o">.</span><span class="n">delete</span><span class="p">()</span>
<span class="n">db</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Looping through image definitions'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">image</span> <span class="ow">in</span> <span class="n">config</span><span class="p">[</span><span class="s">'builder'</span><span class="p">][</span><span class="s">'images'</span><span class="p">]:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Working on {name}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="n">image</span><span class="p">))</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Definition data: {!r}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">image</span><span class="p">))</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Determine hash digest'</span><span class="p">)</span>
<span class="n">digest</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">if</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">((</span><span class="s">'git://'</span><span class="p">,</span> <span class="s">'github.com'</span><span class="p">)):</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'GIT repo ... using master head rev.'</span><span class="p">)</span>
<span class="n">repo</span> <span class="o">=</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">]</span>
<span class="k">if</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">'github.com'</span><span class="p">):</span>
<span class="n">repo</span> <span class="o">=</span> <span class="s">'https://{from}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="o">**</span><span class="n">image</span><span class="p">)</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">git</span><span class="p">(</span><span class="s">'ls-remote'</span><span class="p">,</span> <span class="n">repo</span><span class="p">,</span> <span class="s">'refs/heads/master'</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'No GIT repo ... assuming directory'</span><span class="p">)</span>
<span class="n">test</span><span class="p">(</span><span class="s">'-e'</span><span class="p">,</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">])</span>
<span class="n">digest</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">sha1</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">ls</span><span class="p">(</span><span class="s">'-lARL'</span><span class="p">,</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">],</span> <span class="n">_ok_code</span><span class="o">=</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span><span class="mi">1</span><span class="p">])))</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'digest = {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">digest</span><span class="p">))</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Determine if image needs to be build'</span><span class="p">)</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">ImageState</span><span class="p">)</span><span class="o">.</span><span class="n">filter_by</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">image</span><span class="p">[</span><span class="s">'name'</span><span class="p">])</span><span class="o">.</span><span class="n">first</span><span class="p">()</span>
<span class="n">build</span> <span class="o">=</span> <span class="ow">not</span> <span class="n">state</span> <span class="ow">or</span> <span class="n">state</span><span class="o">.</span><span class="n">digest</span> <span class="o">!=</span> <span class="n">digest</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'State {!r}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">__dict__</span> <span class="k">if</span> <span class="n">state</span> <span class="k">else</span> <span class="n">state</span><span class="p">))</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Build? {!r}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">build</span><span class="p">))</span>
<span class="n">image_id</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">if</span> <span class="n">build</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Building ...'</span><span class="p">)</span>
<span class="n">running_command</span> <span class="o">=</span> <span class="n">docker</span><span class="p">(</span><span class="s">'build'</span><span class="p">,</span> <span class="s">'-rm'</span><span class="p">,</span> <span class="s">'-t'</span><span class="p">,</span> <span class="n">image</span><span class="p">[</span><span class="s">'name'</span><span class="p">],</span> <span class="n">image</span><span class="p">[</span><span class="s">'from'</span><span class="p">],</span> <span class="n">_iter</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">build_output</span> <span class="o">=</span> <span class="s">""</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">running_command</span><span class="p">:</span>
<span class="n">build_output</span> <span class="o">+=</span> <span class="n">line</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span>
<span class="n">build_ids</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">search</span><span class="p">(</span><span class="s">'Successfully built ([a-z0-9]+)'</span><span class="p">,</span> <span class="n">build_output</span><span class="p">)</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">build_ids</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s">'Build failed: {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">build_output</span><span class="p">))</span>
<span class="n">image_id</span> <span class="o">=</span> <span class="n">build_ids</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Building done'</span><span class="p">)</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'Image id = {}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">image_id</span><span class="p">))</span>
<span class="k">if</span> <span class="n">image_id</span><span class="p">:</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Updating state with build info'</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">state</span><span class="p">:</span>
<span class="n">state</span> <span class="o">=</span> <span class="n">ImageState</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">image</span><span class="p">[</span><span class="s">'name'</span><span class="p">])</span>
<span class="n">state</span><span class="o">.</span><span class="n">digest</span> <span class="o">=</span> <span class="n">digest</span>
<span class="n">state</span><span class="o">.</span><span class="n">image_id</span> <span class="o">=</span> <span class="n">image_id</span>
<span class="n">logger</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s">'State {!r}'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">__dict__</span><span class="p">))</span>
<span class="n">db</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">state</span><span class="p">)</span>
<span class="n">db</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="n">logger</span><span class="o">.</span><span class="n">info</span><span class="p">(</span><span class="s">'Clean up stale images in state'</span><span class="p">)</span>
<span class="n">image_names</span> <span class="o">=</span> <span class="p">[</span><span class="n">x</span><span class="p">[</span><span class="s">'name'</span><span class="p">]</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">config</span><span class="p">[</span><span class="s">'builder'</span><span class="p">][</span><span class="s">'images'</span><span class="p">]]</span>
<span class="p">(</span><span class="n">db</span><span class="o">.</span><span class="n">query</span><span class="p">(</span><span class="n">ImageState</span><span class="p">)</span>
<span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="o">~</span><span class="n">ImageState</span><span class="o">.</span><span class="n">name</span><span class="o">.</span><span class="n">in_</span><span class="p">(</span><span class="n">image_names</span><span class="p">))</span>
<span class="o">.</span><span class="n">delete</span><span class="p">(</span><span class="n">synchronize_session</span><span class="o">=</span><span class="s">'fetch'</span><span class="p">))</span>
<span class="n">db</span><span class="o">.</span><span class="n">commit</span><span class="p">()</span>
<span class="nd">@arg</span><span class="p">(</span><span class="s">'config_file'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'The (YAML) configuration file to use'</span><span class="p">)</span>
<span class="nd">@arg</span><span class="p">(</span><span class="s">'-c'</span><span class="p">,</span> <span class="s">'--clean'</span><span class="p">,</span> <span class="n">dest</span><span class="o">=</span><span class="s">'clean'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'Clean the state db'</span><span class="p">)</span>
<span class="nd">@arg</span><span class="p">(</span><span class="s">'-s'</span><span class="p">,</span> <span class="s">'--state-dir'</span><span class="p">,</span> <span class="n">dest</span><span class="o">=</span><span class="s">'state_dir'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'Where the state file will go'</span><span class="p">)</span>
<span class="nd">@arg</span><span class="p">(</span><span class="s">'-v'</span><span class="p">,</span> <span class="s">'--verbose'</span><span class="p">,</span> <span class="n">dest</span><span class="o">=</span><span class="s">'verbose'</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s">'count'</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s">'How verbose the output should be; repeat for increased verbosity'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">config_file</span><span class="p">,</span> <span class="n">state_dir</span><span class="o">=</span><span class="s">'.'</span><span class="p">,</span> <span class="n">verbose</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">clean</span><span class="o">=</span><span class="bp">False</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> A simple `docker build` wrapper, reading definitions from a configuration</span>
<span class="sd"> file and using file and repository information to avoid builds in case</span>
<span class="sd"> nothing changed.</span>
<span class="sd"> """</span>
<span class="n">levels</span> <span class="o">=</span> <span class="p">{</span> <span class="mi">0</span><span class="p">:</span> <span class="n">logging</span><span class="o">.</span><span class="n">WARNING</span><span class="p">,</span> <span class="mi">1</span><span class="p">:</span> <span class="n">logging</span><span class="o">.</span><span class="n">INFO</span><span class="p">,</span> <span class="mi">2</span><span class="p">:</span> <span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span> <span class="p">}</span>
<span class="n">verbose</span> <span class="o">=</span> <span class="n">verbose</span> <span class="k">if</span> <span class="n">verbose</span> <span class="ow">in</span> <span class="n">levels</span> <span class="k">else</span> <span class="mi">0</span>
<span class="n">format</span> <span class="o">=</span> <span class="s">'</span><span class="si">%(asctime)s</span><span class="s"> </span><span class="si">%(name)s</span><span class="s"> </span><span class="si">%(levelname)6s</span><span class="s"> </span><span class="si">%(message)s</span><span class="s">'</span>
<span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">level</span><span class="o">=</span><span class="n">levels</span><span class="p">[</span><span class="n">verbose</span><span class="p">],</span> <span class="n">format</span><span class="o">=</span><span class="n">format</span><span class="p">)</span>
<span class="n">logging</span><span class="o">.</span><span class="n">getLogger</span><span class="p">(</span><span class="s">'sqlalchemy.engine'</span><span class="p">)</span><span class="o">.</span><span class="n">setLevel</span><span class="p">(</span><span class="n">levels</span><span class="p">[</span><span class="n">verbose</span> <span class="o">-</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">verbose</span> <span class="o">></span> <span class="mi">0</span> <span class="k">else</span> <span class="mi">0</span><span class="p">])</span>
<span class="n">build</span><span class="p">(</span><span class="n">config_file</span><span class="p">,</span> <span class="n">state_dir</span><span class="p">,</span> <span class="n">clean</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">argh</span><span class="o">.</span><span class="n">dispatch_command</span><span class="p">(</span><span class="n">main</span><span class="p">)</span>
</pre></div>
<p>Using the following YAML configuration:</p>
<div class="highlight"><pre><span class="nn">---</span>
<span class="l-Scalar-Plain">builder</span><span class="p-Indicator">:</span>
<span class="l-Scalar-Plain">images</span><span class="p-Indicator">:</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">test/busybox</span>
<span class="l-Scalar-Plain">from</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">github.com/dotcloud/docker-busybox</span>
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">name</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">test/tmp</span>
<span class="l-Scalar-Plain">from</span><span class="p-Indicator">:</span> <span class="l-Scalar-Plain">test</span>
</pre></div>
<p>it yields the following results:</p>
<div class="highlight"><pre><span class="nx">vagrant</span><span class="p">@</span><span class="nx">precise64</span><span class="p">:/</span><span class="nx">vagrant</span><span class="err">$</span> <span class="nx">.</span><span class="p">/</span><span class="nx">builder.py</span> <span class="nx">dingen.yml</span> <span class="na">-v</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">01</span><span class="p">,</span><span class="mi">988</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Checking</span> <span class="nb">command</span> <span class="nb">line</span> <span class="nx">tools</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">01</span><span class="p">,</span><span class="mi">990</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Load</span> <span class="nx">configuration</span> <span class="nb">file</span> <span class="nx">dingen.yml</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">01</span><span class="p">,</span><span class="mi">995</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Prepare</span> <span class="nx">state</span> <span class="nb">db</span> <span class="nb">connection</span> <span class="nx">sqlite</span><span class="p">:</span><span class="c1">///./.builder.state.db</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">002</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Create</span> <span class="n">tables</span> <span class="k">in</span> <span class="nx">state</span> <span class="nb">db</span> <span class="k">if</span> <span class="nx">needed</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">011</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Setup</span> <span class="nx">the</span> <span class="nb">db</span> <span class="nx">session</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">012</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Looping</span> <span class="nx">through</span> <span class="nb">image</span> <span class="nx">definitions</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">012</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Working</span> <span class="k">on</span> <span class="nx">test</span><span class="p">/</span><span class="nx">busybox</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">013</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Determine</span> <span class="nb">hash</span> <span class="nx">digest</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">013</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">GIT</span> <span class="nx">repo</span> <span class="nx">...</span> <span class="nx">using</span> <span class="nx">master</span> <span class="nb">head</span> <span class="nx">rev.</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">932</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Determine</span> <span class="k">if</span> <span class="nb">image</span> <span class="nx">needs</span> <span class="k">to</span> <span class="nx">be</span> <span class="nx">build</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">938</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Build</span><span class="o">?</span> <span class="kc">True</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">02</span><span class="p">,</span><span class="mi">938</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Building</span> <span class="nx">...</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">20</span><span class="p">,</span><span class="mi">757</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">0</span> <span class="p">:</span> <span class="nb">from</span> <span class="nx">scratch</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">20</span><span class="p">,</span><span class="mi">757</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">511136</span><span class="nx">ea3c5a</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">20</span><span class="p">,</span><span class="mi">758</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">1</span> <span class="p">:</span> <span class="nb">add</span> <span class="nx">busybox.tar.bz2</span> <span class="o">/</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">391</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">29903600</span><span class="nx">c7c5</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">391</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">2</span> <span class="p">:</span> <span class="nx">maintainer</span> <span class="nx">Jerome</span> <span class="nx">Petazzoni</span> <span class="o"><</span><span class="nx">jerome</span><span class="p">@</span><span class="nx">dotcloud.com</span><span class="o">></span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">409</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="n">Running</span> <span class="k">in</span> <span class="nx">a1031a9239e1</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">420</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">437</span><span class="nx">a86f8a2d9</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">421</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Successfully</span> <span class="nx">built</span> <span class="mi">437</span><span class="nx">a86f8a2d9</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">436</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Removing</span> <span class="nx">intermediate</span> <span class="nx">container</span> <span class="mi">36</span><span class="nx">f438ec0a8c</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">443</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Removing</span> <span class="nx">intermediate</span> <span class="nx">container</span> <span class="nx">a1031a9239e1</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">461</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Building</span> <span class="nb">done</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">462</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Updating</span> <span class="nx">state</span> <span class="k">with</span> <span class="nx">build</span> <span class="nx">info</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">470</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Working</span> <span class="k">on</span> <span class="nx">test</span><span class="p">/</span><span class="nx">tmp</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">470</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Determine</span> <span class="nb">hash</span> <span class="nx">digest</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">471</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">No</span> <span class="nx">GIT</span> <span class="nx">repo</span> <span class="nx">...</span> <span class="nx">assuming</span> <span class="nx">directory</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">505</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Determine</span> <span class="k">if</span> <span class="nb">image</span> <span class="nx">needs</span> <span class="k">to</span> <span class="nx">be</span> <span class="nx">build</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">514</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Build</span><span class="o">?</span> <span class="kc">True</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">514</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nx">Building</span> <span class="nx">...</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">546</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">0</span> <span class="p">:</span> <span class="nb">FROM</span> <span class="nx">ubuntu</span><span class="p">:</span><span class="mf">12.04</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">546</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">9</span><span class="nx">cd978db300e</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">547</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">1</span> <span class="p">:</span> <span class="nb">RUN</span> <span class="nx">apt</span><span class="na">-get</span> <span class="nx">update</span> <span class="o">&&</span> <span class="nx">apt</span><span class="na">-get</span> <span class="na">-q</span> <span class="na">-y</span> <span class="nb">install</span> <span class="nx">apache2</span> <span class="o">&&</span> <span class="nx">apt</span><span class="na">-get</span> <span class="nx">clean</span> <span class="o">&&</span> <span class="nx">rm</span> <span class="na">-rf</span> <span class="p">/</span><span class="nb">var</span><span class="p">/</span><span class="nx">lib</span><span class="p">/</span><span class="nx">apt</span><span class="p">/</span><span class="nx">lists</span><span class="o">/*</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">550</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="nx">Using</span> <span class="k">cache</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">551</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">5</span><span class="nx">ee6c8eb7c4c</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">551</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">2</span> <span class="p">:</span> <span class="nb">ENV</span> <span class="nx">APACHE_RUN_USER</span> <span class="nx">www</span><span class="na">-data</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">556</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="nx">Using</span> <span class="k">cache</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">557</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">1</span><span class="nx">ca385555c50</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">557</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">3</span> <span class="p">:</span> <span class="nb">ENV</span> <span class="nx">APACHE_RUN_GROUP</span> <span class="nx">www</span><span class="na">-data</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">564</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="nx">Using</span> <span class="k">cache</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">564</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">981121</span><span class="nx">ac107e</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">565</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">4</span> <span class="p">:</span> <span class="nb">ENV</span> <span class="nx">APACHE_LOG_DIR</span> <span class="p">/</span><span class="nb">var</span><span class="p">/</span><span class="k">log</span><span class="p">/</span><span class="nx">apache2</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">569</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="nx">Using</span> <span class="k">cache</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">569</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">49</span><span class="nx">a9468dbec1</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">570</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">5</span> <span class="p">:</span> <span class="nb">EXPOSE</span> <span class="mi">80</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">587</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="n">Running</span> <span class="k">in</span> <span class="nx">ca3ea1f08ac0</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">602</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="o">---></span> <span class="mi">8</span><span class="nx">e2d89709317</span>
<span class="mi">2014</span><span class="o">-</span><span class="mi">03</span><span class="o">-</span><span class="mi">21</span> <span class="mi">22</span><span class="p">:</span><span class="mi">02</span><span class="p">:</span><span class="mi">21</span><span class="p">,</span><span class="mi">602</span> <span class="nx">__main__</span> <span class="nx">INFO</span> <span class="nb">Step</span> <span class="mi">6</span> <span class="p">:</span> <span class="nx">CMD</span> <span class="err">[</span><span class="s2">"/usr/sbin/apache2"</span><span class="p">,</span> <span class="s2">"-D"</span><span class="p">,</span> <span class="s2">"FOREGROUND"</span><span class="cp">]</span>
2014-03-21 22:02:21,630 __main__ INFO ---> Running in 88ce4b522434
2014-03-21 22:02:21,648 __main__ INFO ---> 8e8691401ff3
2014-03-21 22:02:21,649 __main__ INFO Successfully built 8e8691401ff3
2014-03-21 22:02:21,660 __main__ INFO Removing intermediate container ca3ea1f08ac0
2014-03-21 22:02:21,670 __main__ INFO Removing intermediate container 88ce4b522434
2014-03-21 22:02:21,688 __main__ INFO Building done
2014-03-21 22:02:21,688 __main__ INFO Updating state with build info
2014-03-21 22:02:21,699 __main__ INFO Clean up stale images in state
vagrant@precise64:/vagrant$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
test/busybox latest 437a86f8a2d9 5 seconds ago 3.229 MB
test/tmp latest 8e8691401ff3 5 seconds ago 258.5 MB
<span class="nt"><none></span> <span class="nt"><none></span> cc2a40a7e31e 6 minutes ago 258.5 MB
<span class="nt"><none></span> <span class="nt"><none></span> f6e2afd65a0c 6 minutes ago 3.229 MB
ubuntu 12.04 9cd978db300e 6 weeks ago 204.4 MB
scratch latest 511136ea3c5a 9 months ago 0 B
vagrant@precise64:/vagrant$ ./builder.py dingen.yml -v
2014-03-21 22:02:33,577 __main__ INFO Checking command line tools
2014-03-21 22:02:33,577 __main__ INFO Load configuration file dingen.yml
2014-03-21 22:02:33,581 __main__ INFO Prepare state db connection sqlite:///./.builder.state.db
2014-03-21 22:02:33,586 __main__ INFO Create tables in state db if needed
2014-03-21 22:02:33,591 __main__ INFO Setup the db session
2014-03-21 22:02:33,592 __main__ INFO Looping through image definitions
2014-03-21 22:02:33,592 __main__ INFO Working on test/busybox
2014-03-21 22:02:33,592 __main__ INFO Determine hash digest
2014-03-21 22:02:33,592 __main__ INFO GIT repo ... using master head rev.
2014-03-21 22:02:34,408 __main__ INFO Determine if image needs to be build
2014-03-21 22:02:34,415 __main__ INFO Build? False
2014-03-21 22:02:34,415 __main__ INFO Working on test/tmp
2014-03-21 22:02:34,415 __main__ INFO Determine hash digest
2014-03-21 22:02:34,415 __main__ INFO No GIT repo ... assuming directory
2014-03-21 22:02:34,454 __main__ INFO Determine if image needs to be build
2014-03-21 22:02:34,458 __main__ INFO Build? False
2014-03-21 22:02:34,458 __main__ INFO Clean up stale images in state
</pre></div>
<p>It does quite a few things. Probably too much in retrospect. Furthermore
error handling could be improved and parallelisation could be
implemented among other things.</p>How dokku works2013-11-23T00:00:00+01:00Rico Moormantag:off-the-stack.moorman.nu,2013-11-23:2013-11-23-how-dokku-works.html<p><em>based on dokku v0.2.0-RC1-25-g698e5e5</em></p>
<h2>What dokku is</h2>
<p><a href="https://github.com/progrium/dokku" title="Dokku on github">Dokku</a> markets itself as "Docker powered mini-Heroku.
The smallest PaaS implementation you've ever seen" which pretty
much describes what it is about.</p>
<p>The project uses <a href="https://www.docker.io/" title="Docker website">Docker</a>, <a href="https://github.com/progrium/buildstep" title="Buildstep on github">Buildstep</a>,
<a href="https://github.com/progrium/sshcommand" title="sshcommand on github">ssh-command</a>, <a href="https://github.com/progrium/pluginhook" title="pluginhook on github">pluginhook</a>, <a href="http://www.openssh.com/" title="OpenSSH website">ssh</a>,
<a href="http://git-scm.com/" title="Git website">git</a>, <a href="http://nginx.org/" title="nginx website">nginx</a> and some bash-glue to provide
you with a simplified single-host version of <a href="https://www.heroku.com/" title="Heroku website">Heroku</a>.</p>
<p>Dokku is currently under quite active development and could probably
change quite a bit after this is written.</p>
<p>The project's original author also wrote a <a href="http://progrium.com/blog/2013/06/19/dokku-the-smallest-paas-implementation-youve-ever-seen/">small introduction on
dokku</a> on his website which also includes a nice
screencast.</p>
<h2>What you get</h2>
<p>As it's tagline suggests, you don't get much. At least code-wise. The
main script is about 100 lines long. The plugins that ship with dokku
contain about 470 lines of shell-code.</p>
<p>There is a git plugin for handling the interaction with the git
repositories, an nginx-vhosts plugin for making the apps available
through nginx, a config plugin which manages the settings for a
given app as well as a backup plugin which can backup and restore the
dokku and application settings.</p>
<p>Dokku and the dokku-standard plugin let you view the url and logs of
an application, run a command within the app's container, deploy and of
course delete it again.</p>
<p>Besides that, the installation procedure includes docker for managing
linux containers (which will host your applications) and a buildstep
docker image which will be used as base image for all app deployments.</p>
<p>The other dependencies are used to tie this together in a useful and
extendable way.</p>
<p>Also dokku can be easily tried out using the provided Vagrantfile.</p>
<h2>What it does</h2>
<h3>SSH command</h3>
<p>After installing dokku and configuring it as described within the
project's README it is ready to use.</p>
<p>During configuration you have to add your public key to dokku (I am
using vagrant in my example):</p>
<div class="highlight"><pre><span class="nv">$ </span>cat ~/.ssh/id_rsa.pub | vagrant ssh -- sudo sshcommand acl-add dokku rico
</pre></div>
<p>This adds a custom rule to the <code>authorized_keys</code> file for the dokku user
on your dokku server:</p>
<div class="highlight"><pre><span class="nb">command</span><span class="o">=</span><span class="s2">"FINGERPRINT=... NAME=rico `cat /home/dokku/.sshcommand` $SSH_ORIGINAL_COMMAND"</span>,no-agent-forwarding,no-user-rc,no-X11-forwarding,no-port-forwarding ssh-rsa ... ...@...
</pre></div>
<p>This line causes all commands received from a connection using the
previously added public-key, to be redirected to the dokku command.</p>
<p>To split this up:</p>
<ul>
<li>the <code>command="..."</code> part of the line specifies the command to be
executed whenever the given given key (found at the end of the
line) is used for authentication</li>
<li><code>FINGERPRINT=...</code> and <code>NAME=...</code> specify some additional environment
variables for the command to run while <code>cat /home/dokku/.sshcommand</code>
reads the actual command to be executed from a separate file</li>
<li>the file <code>/home/dokku/.sshcommand</code> contains <code>/usr/local/bin/dokku</code>; this
file is created during the installation by running
<code>sshcommand create dokku /usr/local/bin/dokku</code></li>
<li>several ssh feature exclusions are put into place; features which
won't be of much use here anyway</li>
<li>finally the line ends with the public key starting with the key
algorithm followed by the key itself and some (email-like)
identifier</li>
</ul>
<p>The <code>FINGERPRINT</code> and <code>NAME</code> variables are currently not used by dokku
itself but probably could be used by plugin-writers for additional
logging or integration purposes.</p>
<p>Normally using a "default" ssh setup you could run arbitrary commands
on the remote host, passing them as argument to the ssh-client program.
The <code>uptime</code> command for example could tell you how long the host has
been up:</p>
<div class="highlight"><pre><span class="nv">$ </span>ssh someotherhost uptime
07:51:53 up 18 days, 20:00, 0 users, load average: 0.06, 0.12, 0.17
</pre></div>
<p>But when we try this on the dokku account, we don't get that
information:</p>
<div class="highlight"><pre><span class="nv">$ </span>ssh -p2222 dokku@localhost uptime
</pre></div>
<p>It does not return anything.</p>
<p>The additional line inside the authorized_keys file causes the <code>dokku</code>
command to be invoked instead of <code>uptime</code>.</p>
<p>Dokku's default behavior for actions not defined directly within the
main script, is to call all available plugins and let them handle it.
No default plugin handles <code>uptime</code> so not much is done.</p>
<p>But using a known command provides a neat way to remotely manage your
dokku installation and deployed apps:</p>
<div class="highlight"><pre><span class="nv">$ </span>ssh -p2222 dokku@localhost <span class="nb">help</span>
<span class="nb"> </span>backup:export <span class="o">[</span>file<span class="o">]</span> Export dokku configuration files
backup:import <span class="o">[</span>file<span class="o">]</span> Import dokku configuration files
config <app> display the config vars <span class="k">for </span>an app
config:get <app> KEY display a config value <span class="k">for </span>an app
config:set <app> <span class="nv">KEY1</span><span class="o">=</span>VALUE1 <span class="o">[</span><span class="nv">KEY2</span><span class="o">=</span>VALUE2 ...<span class="o">]</span> <span class="nb">set </span>one or more config vars
config:unset <app> KEY1 <span class="o">[</span>KEY2 ...<span class="o">]</span> <span class="nb">unset </span>one or more config vars
delete <app> Delete an application
<span class="nb">help </span>Print the list of commands
logs <app> <span class="o">[</span>-t<span class="o">]</span> Show the last logs <span class="k">for </span>an application <span class="o">(</span>-t follows<span class="o">)</span>
plugins-install Install active plugins
plugins Print active plugins
run <app> <cmd> Run a <span class="nb">command </span>in the environment of an application
url <app> Show the URL <span class="k">for </span>an application
version Print dokku<span class="err">'</span>s version
</pre></div>
<h3>Git push</h3>
<p>What happens when you invoke <code>git push dokku master</code> is
defined within the <code>git</code> plugin. It contains the following commands
file:</p>
<div class="highlight"><pre><span class="nv">$ </span>cat /var/lib/dokku/plugins/git/commands
<span class="c">#!/usr/bin/env bash</span>
<span class="nb">set</span> -eo pipefail; <span class="o">[[</span> <span class="nv">$DOKKU_TRACE</span> <span class="o">]]</span> <span class="o">&&</span> <span class="nb">set</span> -x
<span class="k">case</span> <span class="s2">"$1"</span> in
git-hook<span class="o">)</span>
<span class="nv">APP</span><span class="o">=</span><span class="nv">$2</span>
<span class="k">while </span><span class="nb">read </span>oldrev newrev refname
<span class="k">do</span>
<span class="c"># Only run this script for the master branch. You can remove this</span>
<span class="c"># if block if you wish to run it for others as well.</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$refname</span> <span class="o">=</span> <span class="s2">"refs/heads/master"</span> <span class="o">]]</span> ; <span class="k">then</span>
<span class="k"> </span>git archive <span class="nv">$newrev</span> | dokku receive <span class="nv">$APP</span> | sed -u <span class="s2">"s/^/"</span><span class="s1">$'\e[1G'</span><span class="s2">"/"</span>
<span class="k">fi</span>
<span class="k"> done</span>
;;
git-*<span class="o">)</span>
<span class="nv">APP</span><span class="o">=</span><span class="s2">"$(echo $2 | perl -pe 's/(?<!\\)'\''//g' | sed 's/\\'\''/'\''/g')"</span>
<span class="nv">APP_PATH</span><span class="o">=</span><span class="nv">$DOKKU_ROOT</span>/<span class="nv">$APP</span>
<span class="k">if</span> <span class="o">[[</span> <span class="nv">$1</span> <span class="o">==</span> <span class="s2">"git-receive-pack"</span> <span class="o">&&</span> ! -d <span class="nv">$APP_PATH</span> <span class="o">]]</span>; <span class="k">then</span>
<span class="k"> </span>git init --bare <span class="nv">$APP_PATH</span> > /dev/null
<span class="nv">PRERECEIVE_HOOK</span><span class="o">=</span><span class="s2">"$APP_PATH/hooks/pre-receive"</span>
cat > <span class="nv">$PRERECEIVE_HOOK</span> <span class="s"><<EOF</span>
<span class="s">#!/usr/bin/env bash</span>
<span class="s">set -e; set -o pipefail;</span>
<span class="s">cat | DOKKU_ROOT="$DOKKU_ROOT" dokku git-hook $APP</span>
<span class="s">EOF</span>
chmod +x <span class="nv">$PRERECEIVE_HOOK</span>
<span class="k">fi</span>
<span class="k"> </span><span class="nv">args</span><span class="o">=</span><span class="nv">$@</span>
git-shell -c <span class="s2">"$args"</span>
;;
<span class="k">esac</span>
cat
</pre></div>
<p>When the <code>git</code> client program is used to push repository information
to a remote repository using the ssh transport, it actually tries to run
the <code>git-receive-pack</code> command on the remote machine (you can read more
on this <a href="http://stefan.saasen.me/articles/git-clone-in-haskell-from-the-bottom-up/#git_transport_and_pack_wire_protocol" title="interesting article about re-implementing some of git">here</a> or <a href="http://git-scm.com/book/ch9-6.html">here</a>).</p>
<p>Due to the ssh setup described earlier, this command will be fed into
dokku. Internally <code>dokku git-receive-pack 'repository-path'</code> will be
called, which in turn causes all plugin command files to be executed.
The git plugin's commands file listed above is executed too and the
<code>git-*)</code> case will match. The plugin then determines the application
repository path, creates an empty repository if none exists and places
a custom <code>pre-receive</code> hook file in there.</p>
<p>After that, it passes control to the <code>git-shell</code> command to let it
handle the rest of the push.</p>
<p>The git-shell will then invoke the actual receive-pack command which
calls the added pre-receive hook script which in turn will invoke dokku
again like <code>dokku git-hook $APP</code> and feed it information about the push.</p>
<p>The call to <code>dokku</code> with <code>git-hook</code> will lead to yet another
invocation of the git plugin commands file. This time the <code>git-hook)</code>
case will match, which will trigger <code>git archive</code> to exports the
pushed application as tar archive to stdout and feed this to
<code>dokku receive</code>.</p>
<h3>Deployment</h3>
<p>At this point, the actual build and deployment happens within the main
script:</p>
<div class="highlight"><pre><span class="k">case</span> <span class="s2">"$1"</span> in
receive<span class="o">)</span>
<span class="nv">APP</span><span class="o">=</span><span class="s2">"$2"</span>; <span class="nv">IMAGE</span><span class="o">=</span><span class="s2">"app/$APP"</span>
<span class="nb">echo</span> <span class="s2">"-----> Building $APP ..."</span>
cat | dokku build <span class="nv">$APP</span> <span class="nv">$IMAGE</span>
<span class="nb">echo</span> <span class="s2">"-----> Releasing $APP ..."</span>
dokku release <span class="nv">$APP</span> <span class="nv">$IMAGE</span>
<span class="nb">echo</span> <span class="s2">"-----> Deploying $APP ..."</span>
dokku deploy <span class="nv">$APP</span> <span class="nv">$IMAGE</span>
<span class="nb">echo</span> <span class="s2">"-----> Cleaning up ..."</span>
dokku cleanup
<span class="nb">echo</span> <span class="s2">"=====> Application deployed:"</span>
<span class="nb">echo</span> <span class="s2">" $(dokku url $APP)"</span>
<span class="nb">echo</span>
;;
</pre></div>
<p>First, the app is fed to the <code>buildstep</code> docker container by piping the
received git archive output to <code>dokku build</code>.
The buildstep container is responsible of handling heroku-style
buildpacks. It will try to detect the type of application (think ruby,
python, java, php, node ...) and run the proper buildpack.</p>
<p>After the build is finished, <code>dokku release</code> will take the application
settings defined using the <code>config</code> plugin and add them to the new
container.</p>
<p><code>dokku deploy</code> first starts the application and invokes the <code>post-deploy</code>
pluginhook. The <code>nginx</code> plugin contains a script for this hook which
creates an appropriate nginx vhost for the application and reloads nginx.</p>
<p>Finally, after some cleanup, dokku will then happily report the url of the
deployed application back to you!</p>
<h2>Wrapping it up</h2>
<p>Dokku provides a relatively simple way to deploy applications on your
own servers in a heroku-like way.</p>
<p>Using the buildstep image as base for all app deployments shows one of
docker's strengths. Due to docker's layers (union file system), disk
usage is minimized. Only the code of the application code and the
result of the buildpack run need to be saved on disk, the rest is shared.</p>
<p>Dokku can get your applications up and running. But you have to take care
of the rest yourself. Your app will probably need a database, other
services or monitoring.
Fortunately, dokku supports plugins which can let you add this extra
functionality to your dokku installation.</p>
<p>The scope of dokku is limited, but it is not intended to be much more
than it currently is. If you want more, the documentation suggests that
one could also take a look at <a href="https://flynn.io/">flynn.io</a>.</p>Gather metrics using psutil2013-09-28T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2013-09-28:2013-09-28-gather-metrics-using-psutil.html<p><a class="reference external" href="https://code.google.com/p/psutil/">Psutil</a> is a nice python module, providing an interface for gathering system and
process related metrics.</p>
<p>It can be easily installed from pypi using pip for example. After that,
a straightforward <code>import psutil</code> is enough to load the library.</p>
<p>Then one can get CPU metrics:</p>
<pre class="literal-block">
In [2]: psutil.cpu_percent()
Out[2]: 15.0
In [3]: psutil.cpu_percent(percpu=True)
Out[3]: [30.0, 10.0, 18.2, 0.0]
In [4]: psutil.cpu_percent()
Out[4]: 5.1
In [5]: psutil.cpu_percent(percpu=True)
Out[5]: [10.0, 0.0, 18.2, 0.0]
In [6]: psutil.cpu_times_percent()
Out[6]: cpupercent(user=4.9, nice=0.0, system=2.4, idle=92.7)
</pre>
<p>Or look up some disk related information:</p>
<pre class="literal-block">
In [7]: psutil.disk_partitions()
Out[7]:
[partition(device='/dev/disk1s2', mountpoint='/', fstype='hfs', opts='rw,local,rootfs,dovolfs,journaled,multilabel'),
partition(device='/dev/disk0s2', mountpoint='/Volumes/ssd', fstype='hfs', opts='rw,local,dovolfs,journaled,multilabel')]
In [8]: psutil.disk_usage('/')
Out[8]: usage(total=499248103424, used=131396407296, free=367589552128, percent=26.3)
In [9]: psutil.disk_usage('/Volumes/ssd')
Out[9]: usage(total=179701792768, used=170759176192, free=8942616576, percent=95.0)
In [10]: psutil.disk_io_counters()
Out[10]: iostat(read_count=4172348L, write_count=12281956L, read_bytes=111858322944L, write_bytes=305864595968L, read_time=41385667L, write_time=180848739L)
In [11]: psutil.disk_io_counters(perdisk=True)
Out[11]:
{'disk0': iostat(read_count=670192L, write_count=3086920L, read_bytes=15604282880L, write_bytes=107617389568L, read_time=300909L, write_time=2128735L),
'disk1': iostat(read_count=3505289L, write_count=9208877L, read_bytes=96349457920L, write_bytes=198652195328L, read_time=41111034L, write_time=178839667L)}
</pre>
<p>Or some information on memory, networking and users:</p>
<pre class="literal-block">
In [12]: psutil.virtual_memory()
Out[12]: vmem(total=8589934592L, available=3214712832L, percent=62.6, used=8093507584L, free=495013888L, active=3913555968L, inactive=2719698944L, wired=1460252672L)
In [13]: psutil.swap_memory()
Out[13]: swap(total=7516192768L, used=6119391232L, free=1396801536L, percent=81.4, sin=129702862848L, sout=11307806720L)
In [14]: psutil.net_io_counters()
Out[14]: iostat(bytes_sent=4525121720L, bytes_recv=19433505221L, packets_sent=16177042L, packets_recv=21742202L, errin=0L, errout=0L, dropin=0L, dropout=0)
In [15]: psutil.net_io_counters(pernic=True)
Out[15]:
{'gif0': iostat(bytes_sent=0L, bytes_recv=0L, packets_sent=0L, packets_recv=0L, errin=0L, errout=0L, dropin=0L, dropout=0),
'en0': iostat(bytes_sent=875356954L, bytes_recv=10604934990L, packets_sent=6271968L, packets_recv=10024643L, errin=0L, errout=0L, dropin=0L, dropout=0),
'en1': iostat(bytes_sent=1010727303L, bytes_recv=6189533434L, packets_sent=6253953L, packets_recv=8066436L, errin=0L, errout=0L, dropin=0L, dropout=0),
'lo0': iostat(bytes_sent=2639038387L, bytes_recv=2639038387L, packets_sent=3651139L, packets_recv=3651139L, errin=0L, errout=0L, dropin=0L, dropout=0),
'p2p0': iostat(bytes_sent=0L, bytes_recv=0L, packets_sent=0L, packets_recv=0L, errin=0L, errout=0L, dropin=0L, dropout=0),
'stf0': iostat(bytes_sent=0L, bytes_recv=0L, packets_sent=0L, packets_recv=0L, errin=0L, errout=0L, dropin=0L, dropout=0),
'vboxnet0': iostat(bytes_sent=0L, bytes_recv=0L, packets_sent=0L, packets_recv=0L, errin=0L, errout=0L, dropin=0L, dropout=0),
'fw0': iostat(bytes_sent=346L, bytes_recv=0L, packets_sent=0L, packets_recv=0L, errin=0L, errout=0L, dropin=0L, dropout=0)}
In [16]: psutil.get_users()
Out[16]:
[user(name='rico', terminal='console', host=None, started=1376508928.0),
user(name='rico', terminal='ttys000', host=None, started=1379359104.0),
user(name='rico', terminal='ttys001', host=None, started=1379345792.0),
user(name='rico', terminal='ttys002', host=None, started=1379362304.0),
user(name='rico', terminal='ttys003', host=None, started=1379362560.0),
user(name='rico', terminal='ttys004', host=None, started=1379310080.0),
user(name='rico', terminal='ttys005', host=None, started=1379359744.0),
user(name='rico', terminal='ttys006', host=None, started=1379363968.0),
user(name='rico', terminal='ttys007', host=None, started=1379364096.0)]
</pre>
<p>Besides those basic bits of system information, <cite>psutil</cite> also offers means to retrieve information on the currently running processes:</p>
<pre class="literal-block">
In [17]: [x.as_dict() for x in psutil.process_iter()]
Out[17]:
[
...
{'cmdline': ['postgres: checkpointer process ', '', '', '', ''],
'connections': [connection(fd=11, family=30, type=2, laddr=('::1', 63472), raddr=('::1', 63472), status=20)],
'cpu_percent': 0.0,
'cpu_times': cputimes(user=0.007971584, system=0.034409444),
'create_time': 1380061680.176461,
'cwd': '/usr/local/var/postgres',
'exe': '/usr/local/Cellar/postgresql/9.2.4/bin/postgres',
'ext_memory_info': meminfo(rss=331776L, vms=2501677056L, pfaults=757760, pageins=0),
'gids': group(real=20, effective=20, saved=20),
'memory_info': meminfo(rss=331776L, vms=2501677056L),
'memory_maps': None,
'memory_percent': 0.0038623809814453125,
'name': 'postgres: checkpointer process ',
'nice': 0,
'num_ctx_switches': amount(voluntary=415, involuntary=0),
'num_fds': 9,
'num_threads': 1,
'open_files': [openfile(path='/usr/local/var/postgres/server.log', fd=2)],
'pid': 318,
'ppid': 305,
'status': 0,
'terminal': None,
'threads': None,
'uids': user(real=501, effective=501, saved=501),
'username': 'rico'},
{'cmdline': ['postgres: writer process ', '', '', '', ''],
'connections': [connection(fd=11, family=30, type=2, laddr=('::1', 63472), raddr=('::1', 63472), status=20)],
'cpu_percent': 0.0,
'cpu_times': cputimes(user=7.840782848, system=20.813142016),
'create_time': 1380061680.176621,
'cwd': '/usr/local/var/postgres',
'exe': '/usr/local/Cellar/postgresql/9.2.4/bin/postgres',
'ext_memory_info': meminfo(rss=331776L, vms=2501677056L, pfaults=729088, pageins=8192),
'gids': group(real=20, effective=20, saved=20),
'memory_info': meminfo(rss=331776L, vms=2501677056L),
'memory_maps': None,
'memory_percent': 0.0038623809814453125,
'name': 'postgres: writer process ',
'nice': 0,
'num_ctx_switches': amount(voluntary=599872, involuntary=0),
'num_fds': 9,
'num_threads': 1,
'open_files': [openfile(path='/usr/local/var/postgres/server.log', fd=2)],
'pid': 319,
'ppid': 305,
'status': 0,
'terminal': None,
'threads': None,
'uids': user(real=501, effective=501, saved=501),
'username': 'rico'},
...
]
</pre>
<p>For each process, one can retrieve CPU-usage, memory, thread, network connections, open file descriptors information and more.</p>
<p>Last but not least, the library is multi-platform, currently supporting Linux, Windows, OSX, FreeBSD and Sun Solaris, both 32-bit and 64-bit.</p>
<p>With all this combined, <cite>psutil</cite> provides a really nice base for building system-metrics related applications.</p>
Grouped rows (old experiment)2013-09-03T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2013-09-03:2013-09-03-grouped-rows-old-experiment.html<p>While digging through a pile of files today I found something interesting back.</p>
<p>A small JS experiment with the idea of having multiple views on the same data,
grouped by some criteria. The data is defined up front and the views group
and manipulate the data. While manipulation happens, views are kept in sync
using simple callbacks and events.</p>
<p>I wrote this about 5 years ago I think, while I was working on some restaurant
review website. The content managers could - along with other information -
enter the opening times of a restaurant.
The editing functionality provided two views, one grouped by the day of the
week and one grouped by type (breakfast, lunch, brunch ...).
On the day view, one would be able to change the type, which effectively
changed the opening time row position on the type based view.</p>
<p>The original solution available inside the CMS handled most of the logic on
the server side, spitting out some HTML form with a few controls. Changing
the data and switching between the view modes happened with a form submit.
That got the job done, but felt a little clumsy.
So, within my spare time, I started to give it another try, starting from
scratch, fiddling with this concept of grouped data rows.</p>
<p>After a short while the following code was written, and has been forgotten a
few days later (being still incomplete).</p>
<ul class="simple">
<li><a class="reference external" href="http://off-the-stack.moorman.nu/static/media/2013-09-03-grouped-rows-old-experiment/grouped_row/dingen10.html">dingen10.html</a> - the main file for testing, (test data generated by
some php code)</li>
<li><a class="reference external" href="http://off-the-stack.moorman.nu/static/media/2013-09-03-grouped-rows-old-experiment/grouped_row/dingen10.js">dingen10.js</a> - the main JS code which handles the view/data bindings
and the different field types</li>
</ul>
<p>In retrospective, there are a couple of things I would do different now of
course. But this is just for reference.</p>
Two Scoops of Django: Best Practices for Django 1.52013-08-31T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2013-08-31:2013-08-31-two-scoops-of-django-best-practices-for-django-15.html<a class="reference external image-reference" href="http://www.amazon.com/gp/product/1481879707/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1481879707&linkCode=as2&tag=offthestack06-20"><img alt="Two Scoops of Django: Best Practices for Django 1.5" src="http://ecx.images-amazon.com/images/I/5145afspvuL._SX260_SH20_.jpg" /></a>
<p>Over the course of the past two months, I read the fine book <a class="reference external" href="https://django.2scoops.org/">Two
Scoops of Django: Best Practices for Django 1.5</a> cover to cover
two times.</p>
<p>This book, written and edited by Daniel Greenfield and Audrey Roy (the folks
behind <a class="reference external" href="https://www.djangopackages.com/">DjangoPackages.com</a>) as well as numerous Django experts with years
of professional experience on the platform, is currently available as Paperback
or ebook. Our shop got it from <a class="reference external" href="http://www.amazon.com/gp/product/1481879707/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=1481879707&linkCode=as2&tag=offthestack06-20">Amazon</a> as a Paperback and I couldn't wait to
read it once I heard that the book was coming.</p>
<p>The book is easy to follow and forms a rather comprehensive collection of best
practices and advice sprinkled with 3rd-party package suggestions and useful
warnings.</p>
<p>I guess this one could save the beginning and/or intermediate Django developer
quite some time, learning lessons that others have learned the hard way.
And even for the more seasoned Django developers, this book can be a good read
too. Make sure you checkout the reviews section on the <a class="reference external" href="https://django.2scoops.org/">Two Scoops website</a> too.</p>
<p>I can recommend this book to anyone being involved in Django development and I
am really looking forward to the next version.</p>
Vagrant and veewee for development sandboxing2012-11-19T00:00:00+01:00Rico Moormantag:off-the-stack.moorman.nu,2012-11-19:2012-11-19-vagrant-and-veewee-for-development-sandboxing.html<p><em>based on vagrant 1.0.5, veewee 0.3.1 and VirtualBox 4.2</em></p>
<p>As I am always looking for tools to simplify my work and play and was looking
for some virtualization helpers to easily manage virtual machines for
development I thought I should pick <a href="http://vagrantup.com/" title="Vagrant">vagrant</a> from my to-do list and
give it a try.</p>
<p>Since I looked at it for the first time quite some time has gone by and the
project grew and matured as it seems and the user-base contains some large
companies as it seems - which is nice IMHO.</p>
<p>Of course I could also stick with already packaged boxes but that would be too
easy I guess. Furthermore I am likely to need custom boxes anyway further down
the road. So I was looking for ways to build custom boxes too.</p>
<p>Quickly <a href="https://github.com/jedi4ever/veewee" title="veewee @ github">veewee</a> showed up - which is a tool for generating vagrant
boxes.</p>
<p>So here we go.</p>
<h2>Installing the software I think I need</h2>
<p>According to the <a href="http://vagrantup.com/v1/docs/getting-started/index.html">vagrant docs</a> and the <a href="https://github.com/jedi4ever/veewee" title="veewee @ github">veewee readme</a>
I will need some software installed on my (64bit debian squeeze) system for this to work.</p>
<p><strong>Mind the prompts</strong>. <code>#</code> stands for root and <code>$</code> for a normal user.</p>
<p><a href="https://www.virtualbox.org/wiki/Linux_Downloads">Installing virtualbox</a></p>
<div class="highlight"><pre>~# cat > /etc/apt/sources.list.d/oracle-virtualbox.list <span class="s"><< EOF</span>
<span class="s">deb http://download.virtualbox.org/virtualbox/debian squeeze non-free</span>
<span class="s">deb http://download.virtualbox.org/virtualbox/debian squeeze contrib</span>
<span class="s">EOF</span>
~# wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
~# apt-get update
~# apt-get install dkms
~# apt-get install virtualbox-4.2
</pre></div>
<p>Note that eventually a relogin/reboot and/or adding a user to the virtualbox
group will be needed to have correct permissions to actually use virtualbox.</p>
<p>Installing vagrant (using the appropriate download from <a href="http://downloads.vagrantup.com/">vagrant downloads</a>):</p>
<div class="highlight"><pre>~# wget -c http://files.vagrantup.com/packages/be0bc66efc0c5919e92d8b79e973d9911f2a511f/vagrant_1.0.5_x86_64.deb
~# dpkg -i vagrant_1.0.5_x86_64.deb
~# <span class="nb">echo</span> <span class="s1">'export PATH=$PATH:/opt/vagrant/bin'</span> >> /etc/profile.d/vagrant_path.sh
~# . /etc/profile.d/vagrant_path.sh
</pre></div>
<p>Maybe some other packages will be needed for installation...</p>
<p>And then veewee:</p>
<div class="highlight"><pre>~# apt-get install libxslt1-dev libxml2-dev zlib1g-dev
~<span class="nv">$ </span>bash -s stable < <<span class="o">(</span>curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer<span class="o">)</span>
~<span class="nv">$ </span>. ~/.bash_profile
~<span class="nv">$ </span>rvm install 1.9.2
~<span class="nv">$ </span>git clone https://github.com/jedi4ever/veewee.git
~<span class="nv">$ </span><span class="nb">cd </span>veewee
~/veewee<span class="nv">$ </span>gem install bundler
~/veewee<span class="nv">$ </span>bundle install
</pre></div>
<p>Note that the RVM install will probably need some extra packages. So just read
and follow the instructions when they come up.</p>
<h2>Building an image</h2>
<p>Veewee comes with quite a few "templates" to choose from in order to build your
vms. <a href="https://github.com/jedi4ever/veewee/tree/master/templates">Take a look</a>. I chose a debian image with reasonably low
memory requirements for now.</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>veewee vbox define <span class="s1">'debian-6-i386'</span> <span class="s1">'Debian-6.0.6-i386-netboot'</span>
~/veewee<span class="nv">$ </span>veewee vbox build <span class="s1">'debian-6-i386'</span>
</pre></div>
<p>That took some time (~10 minutes) ... but now I got a virtual machine up and
running based on the given veewee template.</p>
<p>Veewee tells me that I can login to the vm with:</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>ssh -o <span class="nv">UserKnownHostsFile</span><span class="o">=</span>/dev/null -o <span class="nv">StrictHostKeyChecking</span><span class="o">=</span>no -p 7222 -l vagrant 127.0.0.1
vagrant@127.0.0.1<span class="err">'</span>s password: vagrant
</pre></div>
<p>Which works. Now I verify the build.</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>veewee vbox validate <span class="s1">'debian-6-i386'</span>
</pre></div>
<p>Which runs some cucumber tests which are all green here.</p>
<p>Now I only have to create a box file from this...</p>
<h2>Setting up a box</h2>
<p>When veewee is active, vagrant provides an additional command named <code>basebox</code>
which actually does this. After that I can add it to vagrant, initialize and
start it.</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>vagrant basebox <span class="nb">export</span> <span class="s1">'debian-6-i386'</span>
~/veewee<span class="nv">$ </span>vagrant box add <span class="s1">'debian-6-i386'</span> <span class="s1">'debian-6-i386.box'</span>
~/veewee<span class="nv">$ </span>vagrant init <span class="s1">'debian-6-i386'</span>
~/veewee<span class="nv">$ </span>vagrant up
</pre></div>
<p>Which takes a few minutes ...</p>
<p>... and hangs. Looking around revealed <a href="https://github.com/mitchellh/vagrant/issues/1066">a known issue</a>.</p>
<p>So I opened up the Vagrantfile in question and added <code>config.ssh.timeout = 20</code>
to the configuration directives. Running <code>vagrant up</code> again resulted in a
properly started vm which I then can ssh into using</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>vagrant ssh
</pre></div>
<p>Nice!</p>
<div class="highlight"><pre>~/veewee<span class="nv">$ </span>vagrant down
</pre></div>Semantic versioning2012-10-30T00:00:00+01:00Rico Moormantag:off-the-stack.moorman.nu,2012-10-30:2012-10-30-semantic-versioning.html<p><a href="http://semver.org">Semantic versioning</a> seems widespread and regarded as a good way to
describe the purpose of certain library and program versions. Each release
(version) receives a version number based on it's effect on the public API.</p>
<p>A version consists of three parts X, Y and Z. The version number is formatted
like X.Y.Z. Bug fixes (not affecting the public API) increase Z. Backwards
compatible changes to the public API increase Y and backwards incompatible
changes increase X.</p>
<p>This is a good thing of course. It is clear that whenever X raises, packages
dependent on the public API could break. The main idea is that whenever the
public API of a system changes, that those changes are communicated by
increments in the specific part of the version number.</p>
<h2>Dependency hell / Diamond problem and friends</h2>
<p>Semantic versioning tries to fix the "dependency hell", a unpleasant situation
for everyone who has to deal with and manage dependencies within a project or
even operating system.</p>
<p>While one - for example - would want to provide the latest and greatest version
of software A and B those could have different requirements regarding the API
of library C (which both use). When A requires a newer version of C than B,
and both versions of C are incompatible, there is no easy and safe way to move
on. There are plenty of other examples and problems one can encounter there.</p>
<p>The diamond problem, long dependency chains, circular dependencies, arbitrary
dependencies ...</p>
<h2>Another solution</h2>
<p>While semantic versioning - as proposed - surely provides a way to clearly
state that something dramatically changed within the API of some piece of
software, it still does not allow for several major API versions to coexist
out of the box.</p>
<p>One solution which actually does allow multiple incompatible versions of the
"same" software to coexist would be to integrate the major version number
in the name. Actually this is done for a long time already.</p>
<p>Think of:</p>
<ul>
<li>kdelibs3 - kdelibs4</li>
<li>apache - apache2</li>
<li>beautifulsoup - bs4</li>
</ul>
<p>and many, many more. This naming practice is typically used with libraries
because other packages depend on them.</p>
<p>IMHO, if this practice would be more widespread, it would be less likely
to encounter dependency hell.</p>
<p>Referring back to the above example, software A and B could safely depend
on the incompatible versions of package C if both versions had another name
and thus different top level paths, namespaces a.s.o.</p>
<p>There are a few obvious drawbacks though:</p>
<ul>
<li>more work has to be done for backwards incompatible changes; package name
and all related information has to change</li>
<li>as a result the bar for backwards incompatible changes is risen</li>
<li>as a result innovation could probably be held back</li>
</ul>
<p>On the other hand, it can be refreshing and motivating too, to start with a
new package name and without the burden of thinking about incompatibilities
(yet).</p>
<p>Furthermore this could stimulate thinking about the API upfront and could
maybe lead to more overall stability/predictability of software (development).</p>
<p>Something along the lines of: Incompatible changes change the package name,
backwards compatible changes and bug fixes increase the version number. The
version number could even be reduced to one number/identifier - following
the revision numbers in your VCS-system of choice.</p>
<p>Maybe it is even more "semantically correct" to do it like this. Isn't a major
version of a piece of software something really different? Why not name it
differently?</p>Web benchmarking software2012-10-17T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-10-17:2012-10-17-web-benchmarking-software.html<p>Having an idea of how well a given (web) application performs is one of the
important things to consider while evaluating it. How would it behave on a
"normal" day, a busy day or during a Social media inflicted visitor flood?
How much system resources does the application need under all of those
circumstances? How much storage does your application need after a few hundred,
thousand or million users? And how fast will it be after that?</p>
<p>You would want to know these things <strong>before</strong> you put it online.</p>
<p>Gathering and analyzing this information can point you at bottlenecks in your
design/infrastructure and - after fixing the problems - can help you to verify
that everything performs OK (again).</p>
<p>A benchmarking tool could also help you choose a appropriate solution/framework
for the job at hand.</p>
<p>Running those "benchmarks" by hand can be cumbersome, so you could automate this
too. Adding graphs, charts and advanced report generation could be a next
interesting step.</p>
<h2>Incomplete listing</h2>
<p>Of course there is already quite a lot of good benchmarking software which could
probably just scratch your itch. Some of them support benchmarking using other
protocols than HTTP and/or have other nice features. Here are a few:</p>
<ul>
<li><a href="http://tsung.erlang-projects.org/">Tsung</a></li>
<li><a href="http://jmeter.apache.org/">JMeter</a></li>
<li><a href="http://www.hpl.hp.com/research/linux/httperf/">httperf</a></li>
<li><a href="http://www.joedog.org/JoeDog/Siege">Siege</a></li>
<li><a href="http://httpd.apache.org/docs/2.2/programs/ab.html">Apache bench</a></li>
<li><a href="http://redmine.lighttpd.net/projects/weighttp/wiki">weighttp</a></li>
<li><a href="http://multimechanize.com/">multimechanize</a></li>
<li><a href="http://funkload.nuxeo.org/">funkload</a></li>
<li><a href="http://gatling-tool.org/">gatling</a></li>
<li><a href="http://grinder.sourceforge.net/">grinder</a></li>
</ul>
<p>For ad-hoc benchmarking and integration within scripts, quite a few people
seem to prefer httperf. Tsung for example seems to be able to generate a
realistic load on the target machines using various protocols.</p>
<h2>Things to consider</h2>
<p>Before conducting some more serious benchmarks there are a few special things
to consider too:</p>
<ul>
<li><a href="http://gwan.com/en_apachebench_httperf.html" title="Testing Performance > How-To">take care of the limits</a></li>
<li>synchronize the time between the metric providing machines</li>
<li>try this on spare machines, not in production</li>
<li>be aware of, handle or even measure outbound traffic/requests/emails
in a sensible way</li>
<li>a user session is more than just a bunch of requests on the app's HTML
think of static files, cookies, sessions, uploads, websockets ...
a realistic traffic scenario can lead to other insights than just hitting
a bunch of URLs with HTTP GET requests</li>
<li>trying to break your application, taking it for a rough ride and probing
for the limits should be high priority</li>
<li>investigate what mean people could use for their attacks (DDos, slow loris)
and benchmark for those vulnerabilities</li>
<li>benchmarking tools differ in how they <em>scale</em> and how they implement
<em>concurrency</em></li>
<li>establish a baseline or known reference measurement (e.g. using a minimal
webserver to define a optimal case)</li>
<li>running benchmarking tools on the same machine as the target application
will influence the benchmark results</li>
<li>using VMs might reveal different results than running the application
on bare metal</li>
<li>maybe we should benchmark the benchmark tools</li>
</ul>User services with runit2012-10-16T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-10-16:2012-10-16-user-services-with-runit.html<p><a href="http://smarden.org/runit/" title="Runit process manager">Runit</a> is a pretty lightweight process supervisor (tool-set). It can be
used alongside your distro's own init scripts to build flexible service
structures.</p>
<p>To get a short overview/introduction one could read <a href="http://www.sanityinc.com/articles/init-scripts-considered-harmful" title="Init scripts considered harmful">"Init scripts considered
harmful"</a>, read <a href="http://two-pi-r.livejournal.com/623733.html">a blog post</a> or
take a look on at the <a href="https://wiki.archlinux.org/index.php/Runit">Arch Linux wiki</a> for some examples.</p>
<p>This post aims to describe the minimal steps to get a user service tree up and
running. Furthermore it documents the solution for a small problem when using
the <code>runsvdir</code> command this way.</p>
<h2>Why</h2>
<p>Because it can be quite convenient to have facilities to let users of a system
define services on their own - and let them run as their own user - with
convenient access to all process related information such as pids or log-files.</p>
<p>And while there is some information out there regarding user-level services
using runit, I ran into an issue while trying to take down a whole user service
tree. Therefore it seems useful for me to document the problem and the solution.</p>
<h2>Installation</h2>
<p>The installation of runit should be quite easy on most Linux distributions.</p>
<p>Using Debian (squeeze) it is a matter of running the following command as root:</p>
<div class="highlight"><pre>apt-get install runit
</pre></div>
<p>This retrieves an archive of about 130kB in size. Extracted this take less
than 400kB disk space. Nice.</p>
<h2>The user services service</h2>
<p>From here we just need to create a service which runs the <code>runsvdir</code> program
as the desired user. This is done by creating a service directory and a
corresponding <code>run</code> file (still being root). Suppose we have a user named
<code>rico</code>:</p>
<div class="highlight"><pre>mkdir -p /etc/sv/user/rico
cat > /etc/sv/user/rico/run <span class="s"><< EOF</span>
<span class="s">#!/bin/sh</span>
<span class="s">exec 2>&1</span>
<span class="s">exec chpst -urico runsvdir -P /home/rico/service 'log:....................'</span>
<span class="s">EOF</span>
chmod u+x /etc/sv/user/rico/run
</pre></div>
<p>And in case you are using NIS or other services to provide the system with
the user in question you may want to use sudo within your <code>run</code> file:</p>
<div class="highlight"><pre><span class="c">#!/bin/sh</span>
<span class="nb">exec </span>2>&1
<span class="nb">exec </span>sudo -H -u rico runsvdir -P /home/rico/service <span class="s1">'log:....................'</span>
</pre></div>
<p>To activate the service tree we then just have to add a symlink:</p>
<div class="highlight"><pre>ln -s /etc/sv/user/rico /etc/service/user-rico
</pre></div>
<p>Now the user <code>rico</code> may add services by creating a <code>~/service</code> directory and
adding service directories/symlinks therein.</p>
<p>Take the following (useless) service for example (added as regular user <code>rico</code>):</p>
<div class="highlight"><pre>mkdir -p /home/rico/service/catrandom
cat > /home/rico/service/catrandom/run <span class="s"><<EOF</span>
<span class="s">#!/bin/sh</span>
<span class="s">exec cat /dev/random > /dev/null</span>
<span class="s">EOF</span>
chmod +x /home/rico/service/catrandom/run
</pre></div>
<p>Adding this directory and the run file and making it executable should fire up
the service. This can be verified using ps or htop for example.</p>
<p>Taking down this (useless) service can be done by simply using
<code>sv down ~/service/catrandom</code>. Refer to the documentation for more examples ...</p>
<h2>A small gotcha</h2>
<p>There is still one thing left to do for our user service tree. We should create
a <code>finish</code> script within the same directory where we put the <code>run</code> script for
<code>runsvdir</code>. This script will contain logic which takes down all user services
in case we decide to stop the user services service.</p>
<p>As root:</p>
<div class="highlight"><pre>cat > /etc/sv/user/rico/finish <span class="s"><<EOF</span>
<span class="s">#!/bin/sh</span>
<span class="s">sv -w600 force-stop /home/rico/service/*</span>
<span class="s">sv exit /home/rico/service/*</span>
<span class="s">EOF</span>
chmod u+x /etc/sv/user/rico/finish
</pre></div>
<p>After which we can safely issue <code>sv down user-rico</code> as root to stop all
user services for rico.</p>
<p>This has been suggested by the author of runit himself <a href="http://article.gmane.org/gmane.comp.sysutils.supervision.general/581/match=runsvdir">on the</a> <a href="http://permalink.gmane.org/gmane.comp.sysutils.supervision.general/997">mailing-list</a> and
because <a href="http://www.mail-archive.com/ports@openbsd.org/msg08708.html">svwaitdown has been merged into sv</a> we use sv
there.</p>Keep in touch with your servers using XMPP2012-10-10T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-10-10:2012-10-10-keep-in-touch-with-your-servers-using-xmpp.html<h2>Why</h2>
<p>My first contact with XMPP was some years ago. Back then I had to glue together
the <a href="http://www.facebook.com/">facebook.com</a> and <a href="http://www.hyves.nl/">hyves.nl</a> instant messaging features in one
application providing an improved social experience while shopping over the
internet.</p>
<p>Time has gone by and I still have <a href="http://psi-im.org/" title="Psi - The cross-platform XMPP client for power users">Psi</a> installed on my machines to stay in
touch with a few people. The services I am using don't give me any trouble, it
tends to just work.</p>
<p>Lately my interest in server monitoring grew and while browsing through some old
bookmarks I came across <a href="http://www.darkcoding.net/software/jimbo/">Jimbo</a> and <a href="http://www.schrankmonster.de/2009/01/22/using-jabber-to-monitor-windows-eventlogs/">some</a>
<a href="http://www.schrankmonster.de/2009/01/23/jabber-instant-messaging-eventlog-service-with-presence-information/">posts</a> <a href="http://www.schrankmonster.de/2009/01/27/jabber-logging-windows-service-sourcecode/">from</a>
<a href="http://www.schrankmonster.de/">schrankmonster.de</a> again. This got me thinking and I went on to create a
little prototype of my own.</p>
<h2>The plan</h2>
<p>Sometimes a prototype can help to explore an idea. A prototype can give you an
idea what works and can point you at flaws in your reasoning/design.</p>
<p>Giving the idea of (ad-hoc) server monitoring using a XMPP-client installed on
the server a try, I started with the following requirements my client should
meet:</p>
<ul>
<li>should be able to connect to a remote XMPP server with given credentials</li>
<li>should automatically try to reconnect upon disconnection</li>
<li>should update it's "presence" in order to reflect load (e.g. "do not disturb"
on high load)</li>
<li>should display load statistics within the status message on high load</li>
<li>should respond to commands given to gather system information</li>
<li>should only respond to commands received from a certain user</li>
</ul>
<p>Furthermore I wanted to keep it simple and small.</p>
<p>Shopping around for libraries - and looking for a target language too - I
noticed that most major programming languages do have some client libraries for
dealing with XMPP. I chose Python and after trying SleekXMPP first (which
resulted in a "hanging" client somehow) I finally settled with xmpppy.</p>
<p>For package installation, I will use virtualenv along with virtualenvwrapper in
order to get things working without screwing up my system's python
installation.</p>
<p>Before I can start coding, I will need a XMPP server to connect to though.</p>
<h2>Setup a server</h2>
<p>Of course I could use a public server for this, but it seems more appropriate
for testing purposes to be in charge of the server myself. There are several
good XMPP servers out there - <a href="http://www.ejabberd.im/" title="Erlang xmpp server">ejabberd</a>, <a href="http://www.igniterealtime.org/projects/openfire/" title="Java xmpp server">openfire</a> and
<a href="http://prosody.im/" title="Lua xmpp server">prosody</a> just to name a few. Just head over to the <a href="http://xmpp.org/" title="XMPP Standards Foundation Website">XMPP standards
foundation</a> to get a more comprehensive list of servers.</p>
<p>I will stick with prosody for now as it is advertised as "easy to set up and
configure, and light on resources" which seems just right.</p>
<p>For this experiment I will need at least one "monitor" account - which I will
connect to and which is allowed to communicate with the client) and one "test"
account which is used by the client to log in to the server.</p>
<p>So here we go (as root on my debian squeeze installation):</p>
<div class="highlight"><pre>apt-get install prosody
prosodyctl register monitor localhost password
prosodyctl register <span class="nb">test </span>localhost password
</pre></div>
<h2>Connect your client of choice</h2>
<p>For this experiment I will use Psi, but it should not matter which of the
available XMPP clients is used. For a list of clients head over to the
<a href="http://xmpp.org/xmpp-software/clients/" title="XMPP Clients">xmpp.org website</a>.</p>
<p>For Psi I had to go to "Account Setup", click on "Add", give a sensible name
such as "monitor@localhost" leaving the "Register new account" checkbox
unchecked. The Jabber ID for the next screen would be "monitor@localhost" and
the password (as provided while setting up the server) should be entered
here too. I then just hit "Save" and connected to the account using the main
Psi window.</p>
<h2>Setting up the client</h2>
<p>First of all, I will have to install the xmpppy library which provides the xmpp
module. For safety I will do this using virtualenv(wrapper):</p>
<div class="highlight"><pre>mkvirtualenv <span class="nb">test</span>
pip install xmpppy
</pre></div>
<p>And finally, the client program:</p>
<div class="highlight"><pre><span class="c">#!/usr/bin/env python</span>
<span class="sd">"""</span>
<span class="sd">Prototype client for simple monitoring purposes.</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">traceback</span>
<span class="kn">import</span> <span class="nn">xmpp</span>
<span class="k">def</span> <span class="nf">handle_messages</span><span class="p">(</span><span class="n">jids</span><span class="p">,</span> <span class="n">commands</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a stanza handler function which executes given commands</span>
<span class="sd"> from the provided jids.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">handler</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">stanza</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Handler which executes given commands from authorized jids.</span>
<span class="sd"> """</span>
<span class="n">sender</span> <span class="o">=</span> <span class="n">stanza</span><span class="o">.</span><span class="n">getFrom</span><span class="p">()</span>
<span class="n">message_type</span> <span class="o">=</span> <span class="n">stanza</span><span class="o">.</span><span class="n">getType</span><span class="p">()</span>
<span class="k">if</span> <span class="nb">any</span><span class="p">([</span><span class="n">sender</span><span class="o">.</span><span class="n">bareMatch</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">jids</span><span class="p">]):</span>
<span class="n">command</span> <span class="o">=</span> <span class="n">stanza</span><span class="o">.</span><span class="n">getBody</span><span class="p">()</span>
<span class="k">if</span> <span class="n">commands</span> <span class="ow">and</span> <span class="n">command</span> <span class="ow">in</span> <span class="n">commands</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">commands</span><span class="p">[</span><span class="n">command</span><span class="p">]()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="s">"Unknown command."</span>
<span class="n">client</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">xmpp</span><span class="o">.</span><span class="n">Message</span><span class="p">(</span><span class="n">sender</span><span class="p">,</span> <span class="n">message</span><span class="p">,</span> <span class="n">typ</span><span class="o">=</span><span class="n">message_type</span><span class="p">))</span>
<span class="k">return</span> <span class="n">handler</span>
<span class="k">def</span> <span class="nf">handle_presences</span><span class="p">(</span><span class="n">jids</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Returns a stanza handler function which automatically authorizes</span>
<span class="sd"> incoming presence requests from the provided jids.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">handler</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">stanza</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Handler which automatically authorizes subscription requests</span>
<span class="sd"> from authorized jids.</span>
<span class="sd"> """</span>
<span class="n">sender</span> <span class="o">=</span> <span class="n">stanza</span><span class="o">.</span><span class="n">getFrom</span><span class="p">()</span>
<span class="n">presence_type</span> <span class="o">=</span> <span class="n">stanza</span><span class="o">.</span><span class="n">getType</span><span class="p">()</span>
<span class="k">if</span> <span class="n">presence_type</span> <span class="o">==</span> <span class="s">"subscribe"</span><span class="p">:</span>
<span class="k">if</span> <span class="nb">any</span><span class="p">([</span><span class="n">sender</span><span class="o">.</span><span class="n">bareMatch</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">jids</span><span class="p">]):</span>
<span class="n">client</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">xmpp</span><span class="o">.</span><span class="n">Presence</span><span class="p">(</span><span class="n">to</span><span class="o">=</span><span class="n">sender</span><span class="p">,</span> <span class="n">typ</span><span class="o">=</span><span class="s">"subscribed"</span><span class="p">))</span>
<span class="k">return</span> <span class="n">handler</span>
<span class="k">def</span> <span class="nf">run_client</span><span class="p">(</span><span class="n">client_jid</span><span class="p">,</span> <span class="n">client_password</span><span class="p">,</span> <span class="n">authorized_jids</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Initializes and runs a fresh xmpp client (blocking).</span>
<span class="sd"> """</span>
<span class="n">high_load</span> <span class="o">=</span> <span class="mi">5</span>
<span class="n">commands</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"stats"</span><span class="p">:</span> <span class="k">lambda</span> <span class="o">*</span><span class="n">args</span><span class="p">:</span> <span class="n">os</span><span class="o">.</span><span class="n">popen</span><span class="p">(</span><span class="s">"uptime"</span><span class="p">)</span><span class="o">.</span><span class="n">read</span><span class="p">()</span><span class="o">.</span><span class="n">strip</span><span class="p">(),</span>
<span class="p">}</span>
<span class="c"># Initialize and connect the client.</span>
<span class="n">jid</span> <span class="o">=</span> <span class="n">xmpp</span><span class="o">.</span><span class="n">JID</span><span class="p">(</span><span class="n">client_jid</span><span class="p">)</span>
<span class="n">client</span> <span class="o">=</span> <span class="n">xmpp</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="n">jid</span><span class="o">.</span><span class="n">domain</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">client</span><span class="o">.</span><span class="n">connect</span><span class="p">():</span>
<span class="k">return</span>
<span class="c"># Authenticate.</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">client</span><span class="o">.</span><span class="n">auth</span><span class="p">(</span><span class="n">jid</span><span class="o">.</span><span class="n">node</span><span class="p">,</span> <span class="n">client_password</span><span class="p">,</span> <span class="n">jid</span><span class="o">.</span><span class="n">resource</span><span class="p">):</span>
<span class="k">return</span>
<span class="c"># Register the message handler which is used to respond</span>
<span class="c"># to messages from the authorized contacts.</span>
<span class="n">message_callback</span> <span class="o">=</span> <span class="n">handle_messages</span><span class="p">(</span><span class="n">authorized_jids</span><span class="p">,</span> <span class="n">commands</span><span class="p">)</span>
<span class="n">client</span><span class="o">.</span><span class="n">RegisterHandler</span><span class="p">(</span><span class="s">"message"</span><span class="p">,</span> <span class="n">message_callback</span><span class="p">)</span>
<span class="c"># Register the presence handler which is used to automatically</span>
<span class="c"># authorize presence-subscription requests from authorized contacts.</span>
<span class="n">presence_callback</span> <span class="o">=</span> <span class="n">handle_presences</span><span class="p">(</span><span class="n">authorized_jids</span><span class="p">)</span>
<span class="n">client</span><span class="o">.</span><span class="n">RegisterHandler</span><span class="p">(</span><span class="s">"presence"</span><span class="p">,</span> <span class="n">presence_callback</span><span class="p">)</span>
<span class="c"># Go "online".</span>
<span class="n">client</span><span class="o">.</span><span class="n">sendInitPresence</span><span class="p">()</span>
<span class="n">client</span><span class="o">.</span><span class="n">Process</span><span class="p">()</span>
<span class="c"># Finally, loop, update the presence according to the load and</span>
<span class="c"># "sleep" for a while.</span>
<span class="n">previous_load_maximum</span> <span class="o">=</span> <span class="mi">0</span>
<span class="k">while</span> <span class="n">client</span><span class="o">.</span><span class="n">isConnected</span><span class="p">():</span>
<span class="c"># Determine the load averages and the maximum of the 1, 10 and 15 minute</span>
<span class="c"># averages.</span>
<span class="c"># In case the maximum is above the "high_load" value, the status will be</span>
<span class="c"># set to "do not disturb" with the load averages as status message.</span>
<span class="c"># The status will be updated continuously once we reached high load.</span>
<span class="c"># When the load decreases and the maximum is below the "high_load" value</span>
<span class="c"># again, the status is reset to "available".</span>
<span class="n">load</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getloadavg</span><span class="p">()</span>
<span class="n">load_string</span> <span class="o">=</span> <span class="s">"load: </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s"> </span><span class="si">%s</span><span class="s">"</span> <span class="o">%</span> <span class="n">load</span>
<span class="n">load_maximum</span> <span class="o">=</span> <span class="nb">max</span><span class="p">(</span><span class="nb">list</span><span class="p">(</span><span class="n">load</span><span class="p">))</span>
<span class="k">if</span> <span class="n">load_maximum</span> <span class="o">>=</span> <span class="n">high_load</span><span class="p">:</span>
<span class="n">client</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">xmpp</span><span class="o">.</span><span class="n">Presence</span><span class="p">(</span><span class="n">show</span><span class="o">=</span><span class="s">"dnd"</span><span class="p">,</span> <span class="n">status</span><span class="o">=</span><span class="n">load_string</span><span class="p">))</span>
<span class="k">elif</span> <span class="n">load_maximum</span> <span class="o"><</span> <span class="n">high_load</span> <span class="ow">and</span> <span class="n">previous_load_maximum</span> <span class="o">>=</span> <span class="n">high_load</span><span class="p">:</span>
<span class="n">client</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">xmpp</span><span class="o">.</span><span class="n">Presence</span><span class="p">())</span>
<span class="n">previous_load_maximum</span> <span class="o">=</span> <span class="n">load_maximum</span>
<span class="c"># Handle stanzas and sleep.</span>
<span class="n">client</span><span class="o">.</span><span class="n">Process</span><span class="p">()</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">CLIENT_JID</span> <span class="o">=</span> <span class="s">"test@localhost"</span>
<span class="n">CLIENT_PASSWORD</span> <span class="o">=</span> <span class="s">"password"</span>
<span class="n">AUTHORIZED_JIDS</span> <span class="o">=</span> <span class="p">[</span><span class="s">"monitor@localhost"</span><span class="p">]</span>
<span class="c"># Loop until the program is terminated.</span>
<span class="c"># In case of connection problems and/or exceptions, the client is</span>
<span class="c"># run again after a delay.</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Trying to connect ..."</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">run_client</span><span class="p">(</span><span class="n">CLIENT_JID</span><span class="p">,</span> <span class="n">CLIENT_PASSWORD</span><span class="p">,</span> <span class="n">AUTHORIZED_JIDS</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">,</span> <span class="n">e</span><span class="p">:</span>
<span class="k">print</span> <span class="s">"Caught exception!"</span>
<span class="n">traceback</span><span class="o">.</span><span class="n">print_exc</span><span class="p">()</span>
<span class="k">print</span> <span class="s">"Not connected - attempting reconnect in a moment."</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">10</span><span class="p">)</span>
</pre></div>
<p>It includes three functions, <code>run_client</code> for actually running the client,
<code>handle_messages</code> which returns a handler function responding to received
messages and <code>handle_presences</code> which returns a handler for presence
information/requests.</p>
<p>The program starts at the end of the file. Here the login credentials for
our XMPP server are defined as well as the authorized "jids" which is a list
of accounts for which the client may execute commands and which may receive
presence updates.</p>
<p>Then we enter a simple while loop which basically just starts and restarts
the client over and over again in case of disconnection or other errors.</p>
<p>Within the <code>run_client</code> function some example command is defined together
with the load average which the client should perceive as "high". Then client
is initialized and started providing the presence and message handlers.</p>
<p>Well, starting the client (which I saved in client.py) with:</p>
<div class="highlight"><pre>python client.py
</pre></div>
<p>Results in a working prototype which meets the criteria mentioned above.
Sending "stats" to "test@localhost" is answered with statistics and I
can see the client online (after adding the contact to my roster) with a
status reflecting the load.</p>
<h2>Further thoughts</h2>
<p>The "responsiveness" of the client is sub-optimal; messages are processed with
a noticeable delay because all processing happens synchronously/blocking and the
interpreter is sleeping too within the main-loop.
This could be helped with restructuring and using another (asynchronous)
approach (multiprocessing/threading, twisted, node-xmpp a.s.o).</p>
<p>Furthermore the capabilities are limited (due to the scope of this test) and
the communication could be improved too. Iq stanzas would be more appropriate
for querying the client I suppose. Retrieving information from the client is
in fact a request and each request should have a distinct response. Otherwise
there would be no proper way to determine if we lost messages for example.</p>
<p>We also could use a custom XMPP component to further improve the communication
between the monitor user and the client.</p>
<p>There is a lot left to do ...</p>
<h2>Books that could be of interest</h2>
<p><a href="http://www.amazon.com/gp/product/059652126X/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=059652126X&linkCode=as2&tag=offthestack06-20">XMPP: The Definitive Guide: Building Real-Time Applications with Jabber Technologies</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=offthestack06-20&l=as2&o=1&a=059652126X" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
<p><a href="http://www.amazon.com/gp/product/0470540710/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0470540710&linkCode=as2&tag=offthestack06-20">Professional XMPP Programming with JavaScript and jQuery</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=offthestack06-20&l=as2&o=1&a=0470540710" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
<p><a href="http://www.amazon.com/gp/product/0672325365/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0672325365&linkCode=as2&tag=offthestack06-20">Jabber Developer's Handbook</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=offthestack06-20&l=as2&o=1&a=0672325365" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
<p><a href="http://www.amazon.com/gp/product/0596806159/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0596806159&linkCode=as2&tag=offthestack06-20">Building the Realtime User Experience: Creating Immersive and Interactive Websites</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=offthestack06-20&l=as2&o=1&a=0596806159" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
<p><a href="http://commons.oreilly.com/wiki/index.php/Programming_Jabber">Programming Jabber: Online book from o'reilly</a></p>A python can bite you too2012-09-15T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-09-15:2012-09-15-a-python-can-bite-you-too.html<p>While doing maintenance/bug fixing work on a website built with Django, a came
across an issue with the commenting system. The problem was, that the order of
the comments was not right sometimes. Comments which were placed as a comment
on another comment were occasionally even displayed before their subject.
Another strange thing about it was that it mostly happened at times with higher
traffic.</p>
<p>The application which contained the commenting functionality contained the
following model definition:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">Comment</span><span class="p">(</span><span class="n">models</span><span class="o">.</span><span class="n">Model</span><span class="p">):</span>
<span class="n">comment</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">TextField</span><span class="p">()</span>
<span class="n">published</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">User</span><span class="p">,</span> <span class="n">related_name</span><span class="o">=</span><span class="s">"user_comment"</span><span class="p">)</span>
<span class="n">content_type</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">ForeignKey</span><span class="p">(</span><span class="n">ContentType</span><span class="p">)</span>
<span class="n">object_id</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">PositiveIntegerField</span><span class="p">()</span>
<span class="n">content_object</span> <span class="o">=</span> <span class="n">generic</span><span class="o">.</span><span class="n">GenericForeignKey</span><span class="p">(</span><span class="s">'content_type'</span><span class="p">,</span> <span class="s">'object_id'</span><span class="p">)</span>
<span class="n">client_ip</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">CharField</span><span class="p">(</span><span class="n">max_length</span><span class="o">=</span><span class="mi">15</span><span class="p">)</span>
</pre></div>
<p>Can you spot the bug?</p>
<p>It is right there on line 3 (where the <code>published</code> field is defined).</p>
<p>The problem here was the call to <code>datetime.datetime.now()</code> right there in the
class definition.</p>
<h2>Whats happening</h2>
<p>When the models.py of the application in question is loaded, the class
definition of Comment is evaluated too. Each field of the Comment class receives
an instance of a Field class.</p>
<p>The <code>published</code> field receives an instance of the <code>DateTimeField</code>. No problem so
far. The instance is created using the result of <code>datetime.datetime.now()</code> as
default. And right there things are wrong.</p>
<p>Because we actually call a function here, the result is used. This means the
default value of the field is the "datetime" of the moment when the class
definition is being evaluated.
The python interpreter remembers the resulting class as long as it is running.</p>
<p>In case of the website, the python interpreter was invoked using Apache
with mod_wsgi. This means that a python instance will be reused for several
requests. And all of those requests will use the same default time for
published...</p>
<h2>The fix</h2>
<p>The fix for the bug was the removal of two characters. I had to change the line
from this:</p>
<div class="highlight"><pre> <span class="n">published</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">())</span>
</pre></div>
<p>to this:</p>
<div class="highlight"><pre> <span class="n">published</span> <span class="o">=</span> <span class="n">models</span><span class="o">.</span><span class="n">DateTimeField</span><span class="p">(</span><span class="n">default</span><span class="o">=</span><span class="n">datetime</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">)</span>
</pre></div>
<p>Now the parentheses <code>()</code> after <code>now</code> are missing from the definition, a
reference to the <code>now</code> function is being passed as the default.</p>
<p>When Django tries to retrieve the default value of a field (for instance when
the model is being saved) it now calls the <code>datetime.datetime.now()</code> function
instead of using the result of it when the models were loaded the first time.</p>
<h2>Similar issues</h2>
<p>Other places which are only evaluated once during an interpreter session will
probably expose similar issues.</p>
<p>Take a basic function definition for example:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">items</span><span class="o">=</span><span class="p">[]):</span>
<span class="n">items</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">"test"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">items</span>
</pre></div>
<p>What will a call to <code>example()</code> return? What will be returned a second or
third time?</p>
<div class="highlight"><pre><span class="o">>>></span> <span class="k">def</span> <span class="nf">example</span><span class="p">(</span><span class="n">items</span><span class="o">=</span><span class="p">[]):</span>
<span class="o">...</span> <span class="n">items</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s">"test"</span><span class="p">)</span>
<span class="o">...</span> <span class="k">return</span> <span class="n">items</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">example</span><span class="p">()</span>
<span class="p">[</span><span class="s">'test'</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">example</span><span class="p">()</span>
<span class="p">[</span><span class="s">'test'</span><span class="p">,</span> <span class="s">'test'</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">example</span><span class="p">()</span>
<span class="p">[</span><span class="s">'test'</span><span class="p">,</span> <span class="s">'test'</span><span class="p">,</span> <span class="s">'test'</span><span class="p">]</span>
<span class="o">>>></span>
</pre></div>
<p>You can even get more confusing issues when you combine this behavior with
more side-effects such as database queries.</p>
<p>Once, I saw something like this within a Django application:</p>
<div class="highlight"><pre><span class="k">class</span> <span class="nc">ExampleForm</span><span class="p">(</span><span class="n">forms</span><span class="o">.</span><span class="n">Form</span><span class="p">):</span>
<span class="n">ages</span> <span class="o">=</span> <span class="p">[(</span><span class="s">u''</span><span class="p">,</span> <span class="s">u'Age group'</span><span class="p">)]</span>
<span class="n">ages</span><span class="o">.</span><span class="n">extend</span><span class="p">([([</span><span class="n">c</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">label</span><span class="p">])</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">Age</span><span class="o">.</span><span class="n">objects</span><span class="o">.</span><span class="n">all</span><span class="p">()])</span>
<span class="n">age</span> <span class="o">=</span> <span class="n">forms</span><span class="o">.</span><span class="n">ChoiceField</span><span class="p">(</span><span class="n">choices</span><span class="o">=</span><span class="n">ages</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="bp">False</span><span class="p">)</span>
</pre></div>
<p>This will probably work. The models.py will validate and the application will
start just fine. But what will happen when an age group is added within the
Django admin interface while the front-end application is running?</p>
<p><a href="https://github.com/rmoorman/ots-post-3-example">Try it</a>. You will see that the list of age groups will not
change as long as Django is not restarted/reloaded...</p>
<h2>Conclusion</h2>
<p>Just take care where you declare something and think about when your declaration
will be evaluated, how it will change within the application's lifetime and if
this is really what you want.</p>
<p><strong>So just think about it.</strong></p>Piping hosts to Fabric2012-09-11T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-09-11:2012-09-11-piping-hosts-to-fabric.html<p><em>based on fabric version 0.9.1 and python 2.6</em></p>
<p><a href="http://fabfile.org/" title="Fabric website">Fabric</a> is a command-line tool and library which can be used to
streamline various system administration and application deployment tasks
providing facilities to run commands locally and remotely (through SSH).</p>
<h2>Why</h2>
<p>For remote commands, you can specify the hosts to run the commands on in
various ways. They may be added to the <code>env.hosts</code> right within the "fabfile"
or may be given using the <code>-H HOSTS, --hosts=HOSTS</code> command-line options.
You can even use tasks to populate your <code>env.hosts</code> in the way you want it.
Refer to the documentation for some examples.</p>
<p>I am currently using fabric mostly for ad-hoc tasks on remote systems, such as
placing files, adjusting configuration and restarting services.
Most of the time I am using it like this:</p>
<div class="highlight"><pre>fab -H host1,host2,host3 example_task
</pre></div>
<p>Recently, I needed to execute some task on several hosts given as a file though.
The file consisted of several lines, with a hostname on each line
I was used to using the <code>-H</code> option and came up with something like this:</p>
<div class="highlight"><pre>fab -H <span class="sb">`</span>cat hosts | perl -pe <span class="s1">'s/\n/,/'</span> | perl -pe <span class="s1">'s/(.*),$/$1/'</span><span class="sb">`</span> example_task
</pre></div>
<p>This reads the file "hosts", joins the lines using a comma (as required by <code>-H</code>)
and removes the eventually trailing comma. The output is then used to call fab
with a list of hosts.</p>
<p>While this works, it looks ugly and could become unwieldy in case the list of
hosts needs extra preprocessing/filtering IMHO. I would rather rewrite it to
something like this:</p>
<div class="highlight"><pre>cat hosts | fab read_hosts example_task
</pre></div>
<h2>How</h2>
<p>What basically needs to be done is read the lines from STDIN and use them
to populate the "env.hosts" list.
Furthermore I want this logic to reside within a fabric task in order to
keep the possibility to use other ways of populating the hosts-list too.</p>
<p>The corresponding code would be something like this:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">fabric.api</span> <span class="kn">import</span> <span class="n">env</span><span class="p">,</span> <span class="n">run</span>
<span class="k">def</span> <span class="nf">read_hosts</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Reads hosts from sys.stdin line by line, expecting one host per line.</span>
<span class="sd"> """</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[</span><span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">readlines</span><span class="p">()]</span>
<span class="k">def</span> <span class="nf">example_task</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Example task which executes the uptime command remotely.</span>
<span class="sd"> """</span>
<span class="n">run</span><span class="p">(</span><span class="s">"uptime"</span><span class="p">)</span>
</pre></div>
<p>Looking at the <code>read_hosts()</code> function, it is actually quite simple to add
support for reading from STDIN to a fabfile. From here it should be easy to add
more processing and filtering if appropriate such as ignoring empty and
commented lines:</p>
<div class="highlight"><pre><span class="k">def</span> <span class="nf">read_hosts_filtered</span><span class="p">():</span>
<span class="sd">"""</span>
<span class="sd"> Reads hosts from sys.stdin line by line, expecting one host per line and</span>
<span class="sd"> ignoring empty or commented lines.</span>
<span class="sd"> """</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="o">.</span><span class="n">readlines</span><span class="p">():</span>
<span class="n">host</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
<span class="k">if</span> <span class="n">host</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">host</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s">"#"</span><span class="p">):</span>
<span class="n">env</span><span class="o">.</span><span class="n">hosts</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">host</span><span class="p">)</span>
</pre></div>
<p>Another example reading CSV files with the fields host and role:</p>
<div class="highlight"><pre><span class="kn">import</span> <span class="nn">csv</span>
<span class="k">def</span> <span class="nf">read_hosts_csv</span><span class="p">(</span><span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="sd">"""</span>
<span class="sd"> Reads hosts from sys.stdin as CSV, passing any additional parameters to the</span>
<span class="sd"> csv.reader() function.</span>
<span class="sd"> The CSV is expected to have two columns, one for the hostname and one for</span>
<span class="sd"> the role which are used to populate the fabric roledefs.</span>
<span class="sd"> """</span>
<span class="n">reader</span> <span class="o">=</span> <span class="n">csv</span><span class="o">.</span><span class="n">reader</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">stdin</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="k">for</span> <span class="n">row</span> <span class="ow">in</span> <span class="n">reader</span><span class="p">:</span>
<span class="n">host</span><span class="p">,</span> <span class="n">role</span> <span class="o">=</span> <span class="n">row</span>
<span class="n">env</span><span class="o">.</span><span class="n">roledefs</span><span class="p">[</span><span class="n">role</span><span class="p">]</span> <span class="o">=</span> <span class="n">env</span><span class="o">.</span><span class="n">roledefs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">role</span><span class="p">,</span> <span class="p">[])</span> <span class="o">+</span> <span class="p">[</span><span class="n">host</span><span class="p">]</span>
</pre></div>
<p>Which then can be used to read a standard CSV file like this:</p>
<div class="highlight"><pre>cat hosts_csv | fab read_hosts_csv example_task:role<span class="o">=</span>web
</pre></div>
<p>The <code>read_hosts_csv()</code> task supports different CSV dialects too due to the
<code>**kwargs</code> being passed to the <code>csv.reader()</code> function:</p>
<div class="highlight"><pre><span class="nv">$ </span>cat hosts_csv_colon
localhost:db
localhost:web
localhost:cache
127.0.0.1:web
<span class="nv">$ </span>cat hosts_csv_colon | fab read_hosts_csv:delimiter<span class="o">=</span><span class="s2">":"</span> example_task:role<span class="o">=</span>web
</pre></div>
<p>Further error checking and validation is left as an exercise for the reader.</p>
<p>Similarly it would be easy to support different formats too, such as JSON or
XML. Just implement a matching reader function which translates the input to
hosts or other fabric settings. Then you could for example read the
configuration from a remote feed using curl and pipe it to fabric. Or you
could use a command-line database client to retrieve host information.</p>Using TypoScript outside of Typo32012-07-02T00:00:00+02:00Rico Moormantag:off-the-stack.moorman.nu,2012-07-02:2012-07-02-using-typoscript-outside-of-typo3.html<p><a href="http://en.wikipedia.org/wiki/TYPO3#TypoScript" title="Description of TypoScript on wikipedia">TypoScript</a> is the main configuration language for the
<a href="http://www.typo3.org/" title="Typo3 website">Typo3</a> Content Management System. Within Typo3 it is used for various
things such as declaring what should be used to build the HTML of a page or
declaring which "content columns" should be available to the CMS users for
content entry.</p>
<p>For more information about the syntax and how TypoScript is integrated within
Typo3, here some resources:</p>
<ul>
<li><a href="http://typo3.org/documentation/document-library/core-documentation/doc_core_ts/" title="TypoScript Syntax and In-depth Study">TypoScript Syntax and In-depth Study</a></li>
<li><a href="http://typo3.org/documentation/document-library/core-documentation/doc_core_tsref/" title="TSref">TSref - reference for "TypoScript templates"</a></li>
</ul>
<p>This document is about using TypoScript outside of Typo3 though.</p>
<h2>Why</h2>
<p>While I have been quite positive about TypoScript in general since my first
steps in Typo3 (and think of it as one of the best parts of it actually),
quite a few people I worked with were not.</p>
<p>Thinking about this little configuration language and its place within Typo3,
as well as my own and other's experiences, it seems to me that it could be more
an issue of the surrounding (eco-) system though.</p>
<p>Therefore I would like to show the basic steps that would be required to use
it in a different setting.</p>
<h2>How</h2>
<h3>Get the source</h3>
<p>In order to get it working we need - of course - the sources of the parser.
Unfortunately, the parser - even though it is quite small on itself - requires
the "t3lib_div" class in order to work which on itself depends on quite a few
other parts of typo3.</p>
<p>We have two options here. <a href="http://typo3.org/download/" title="Typo3 Download page">Download the whole source of Typo3</a>
or just get the parser class and stub the rest.</p>
<p>Due to the fact that the download is quite large and we actually just need
one file I will go for option 2 here.</p>
<div class="highlight"><pre>wget -c -O class.t3lib_tsparser.php <span class="s2">"http://git.typo3.org/TYPO3v4/Core.git?a=blob_plain;f=t3lib/class.t3lib_tsparser.php;hb=HEAD"</span>
</pre></div>
<h3>Stub</h3>
<p>From here it is actually quite simple. First we define a few constants needed
by the parser for line endings and tabulator characters. Then we define a stub
<code>t3lib_div</code> class with a few functions used by the parser.</p>
<div class="highlight"><pre><span class="cp"><?php</span>
<span class="c1"># Define constants used by the parser.</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'TAB'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">9</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'LF'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">10</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'CR'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">13</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'CRLF'</span><span class="p">,</span> <span class="nx">CR</span> <span class="o">.</span> <span class="nx">LF</span><span class="p">);</span>
<span class="c1"># Define a t3lib_div class with the functions used by the parser.</span>
<span class="k">class</span> <span class="nc">t3lib_div</span> <span class="p">{</span>
<span class="c1"># Used by the built in functions which can be applied to values.</span>
<span class="c1"># The parser doesn't need all the functionality of this function</span>
<span class="c1"># and therefore the filtering and limiting is left out.</span>
<span class="k">static</span> <span class="k">function</span> <span class="nf">trimExplode</span><span class="p">(</span><span class="nv">$sep</span><span class="p">,</span> <span class="nv">$string</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">array_map</span><span class="p">(</span><span class="s2">"trim"</span><span class="p">,</span> <span class="nb">explode</span><span class="p">(</span><span class="nv">$sep</span><span class="p">,</span> <span class="nv">$string</span><span class="p">));</span>
<span class="p">}</span>
<span class="c1"># Used by the parser to report parse errors/warnings</span>
<span class="k">const</span> <span class="no">SYSLOG_SEVERITY_WARNING</span> <span class="o">=</span> <span class="nx">E_USER_WARNING</span><span class="p">;</span>
<span class="k">static</span> <span class="k">function</span> <span class="nf">sysLog</span><span class="p">(</span><span class="nv">$message</span><span class="p">,</span> <span class="nv">$key</span><span class="p">,</span> <span class="nv">$severity</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">trigger_error</span><span class="p">(</span><span class="s2">"[</span><span class="si">$key</span><span class="s2">] </span><span class="si">$message</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$severity</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<p>The defined constants and functions are the bare minimum to parse TypoScript,
handle errors while parsing and use the few built-in "TSparser functions" to
manipulate values. However, it is not enough to provide own TSparser functions
or to support TypoScript includes.</p>
<p>In order to support includes one would have to call
<code>t3lib_tsparser::checkIncludeLines($typoscript)</code> before parsing which
will return a new TypoScript string with includes added. This, however, requires
a few more t3lib_div functions.
Supporting extra TSparser functions would require messing with Typo3-specific
globals and adding another stub function to our t3lib_div.</p>
<h3>Add conditions</h3>
<p>The TypoScript parser supports <em>conditions</em> which are evaluated at parse-time
and can be used to adjust the results according to your rules.</p>
<p>In order to use them, we have to supply a condition object which has a <code>match</code>
method. This match method will receive the whole condition string (such as
<code>[dayofweek = 0] && [hour <= 12]</code>). The match method should return true if
the condition matches and false in case it does not.</p>
<p>We are not required to supply a match object. In case we don't, the parser
acts as if all conditions are false.</p>
<div class="highlight"><pre><span class="cp"><?php</span>
<span class="c1"># ...</span>
<span class="k">class</span> <span class="nc">MyConditions</span> <span class="p">{</span>
<span class="k">function</span> <span class="nf">match</span><span class="p">(</span><span class="nv">$condition</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$condition</span> <span class="o">==</span> <span class="s2">"[RANDOM]"</span> <span class="o">&&</span> <span class="nx">rand</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></div>
<h3>Try</h3>
<p>Now we are ready to try out the parser.</p>
<div class="highlight"><pre><span class="cp"><?php</span>
<span class="c1"># Load the parser class</span>
<span class="k">require_once</span> <span class="s2">"class.t3lib_tsparser.php"</span><span class="p">;</span>
<span class="c1"># Define constants used by the parser.</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'TAB'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">9</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'LF'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">10</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'CR'</span><span class="p">,</span> <span class="nb">chr</span><span class="p">(</span><span class="mi">13</span><span class="p">));</span>
<span class="nb">define</span><span class="p">(</span><span class="s1">'CRLF'</span><span class="p">,</span> <span class="nx">CR</span> <span class="o">.</span> <span class="nx">LF</span><span class="p">);</span>
<span class="c1"># Define a t3lib_div class with the functions used by the parser.</span>
<span class="k">class</span> <span class="nc">t3lib_div</span> <span class="p">{</span>
<span class="c1"># Used by the built in functions which can be applied to values.</span>
<span class="c1"># The parser doesn't need all the functionality of this function</span>
<span class="c1"># and therefore the filtering and limiting is left out.</span>
<span class="k">static</span> <span class="k">function</span> <span class="nf">trimExplode</span><span class="p">(</span><span class="nv">$sep</span><span class="p">,</span> <span class="nv">$string</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">array_map</span><span class="p">(</span><span class="s2">"trim"</span><span class="p">,</span> <span class="nb">explode</span><span class="p">(</span><span class="nv">$sep</span><span class="p">,</span> <span class="nv">$string</span><span class="p">));</span>
<span class="p">}</span>
<span class="c1"># Used by the parser to report parse errors/warnings</span>
<span class="k">const</span> <span class="no">SYSLOG_SEVERITY_WARNING</span> <span class="o">=</span> <span class="nx">E_USER_WARNING</span><span class="p">;</span>
<span class="k">static</span> <span class="k">function</span> <span class="nf">sysLog</span><span class="p">(</span><span class="nv">$message</span><span class="p">,</span> <span class="nv">$key</span><span class="p">,</span> <span class="nv">$severity</span> <span class="o">=</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="nb">trigger_error</span><span class="p">(</span><span class="s2">"[</span><span class="si">$key</span><span class="s2">] </span><span class="si">$message</span><span class="s2">"</span><span class="p">,</span> <span class="nv">$severity</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">class</span> <span class="nc">MyConditions</span> <span class="p">{</span>
<span class="k">function</span> <span class="nf">match</span><span class="p">(</span><span class="nv">$condition</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$condition</span> <span class="o">==</span> <span class="s2">"[RANDOM]"</span> <span class="o">&&</span> <span class="nx">rand</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$typoscript</span> <span class="o">=</span> <span class="s2">"</span>
<span class="s2">foo = BAR</span>
<span class="s2">foo.bar {</span>
<span class="s2"> baz = FOO</span>
<span class="s2"> list = 0,1,2,3</span>
<span class="s2">}</span>
<span class="s2">[RANDOM]</span>
<span class="s2"> foo.baz = BAR</span>
<span class="s2">[ELSE]</span>
<span class="s2"> foo.baz = BAZ</span>
<span class="s2">[END]</span>
<span class="s2">foo.bar.list := removeFromList(0)</span>
<span class="s2">foo.bar.list := addToList(4)</span>
<span class="s2">foo.bar.list := prependString(0,)</span>
<span class="s2">foo.bar.list := appendString(,5-)</span>
<span class="s2">foo.bar.list := removeString(-)</span>
<span class="s2">foo.bar.list := replaceString(,|-)</span>
<span class="s2">"</span><span class="p">;</span>
<span class="nv">$match_obj</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">MyConditions</span><span class="p">();</span>
<span class="nv">$parser</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">t3lib_tsparser</span><span class="p">();</span>
<span class="nv">$parser</span><span class="o">-></span><span class="na">parse</span><span class="p">(</span><span class="nv">$typoscript</span><span class="p">,</span> <span class="nv">$match_obj</span><span class="p">);</span>
<span class="nb">print_r</span><span class="p">(</span><span class="nv">$parser</span><span class="o">-></span><span class="na">setup</span><span class="p">);</span>
<span class="nb">print_r</span><span class="p">(</span><span class="nv">$parser</span><span class="o">-></span><span class="na">errors</span><span class="p">);</span>
</pre></div>
<p>Running this should result in something like this:</p>
<div class="highlight"><pre><span class="nv">$ </span>php test.php
Array
<span class="o">(</span>
<span class="o">[</span>foo<span class="o">]</span> <span class="o">=</span>> BAR
<span class="o">[</span>foo.<span class="o">]</span> <span class="o">=</span>> Array
<span class="o">(</span>
<span class="o">[</span>bar.<span class="o">]</span> <span class="o">=</span>> Array
<span class="o">(</span>
<span class="o">[</span>baz<span class="o">]</span> <span class="o">=</span>> FOO
<span class="o">[</span>list<span class="o">]</span> <span class="o">=</span>> 0-1-2-3-4-5
<span class="o">)</span>
<span class="o">[</span>baz<span class="o">]</span> <span class="o">=</span>> BAR
<span class="o">)</span>
<span class="o">)</span>
Array
<span class="o">(</span>
<span class="o">)</span>
</pre></div>
<p>And occasionally <code>foo.baz</code> set to <code>BAZ</code>.</p>