A utility for automating multi-host, multi-environment software builds and deployments.
Project description
batou
=====
batou is a multi-(component|host|environment|...) deployment utility.
Deployments are described in a model using "components" and can be deployed
locally and remotely. Models are written as Python code.
See the `batou documentation <http://batou.readthedocs.org>`_ for details
on how to use it.
=========
Changelog
=========
1.0b27 (2014-09-15)
===================
- Do not configure environments locally when running remote deployments. Use
order as determined by one reference remote host. #13530
- Nagios configuration now supports contact groups.
1.0b26 (2014-08-05)
===================
- Enhance git Clone component to support the full suite of functionality the
mercurial one has (incoming/outgoing/local changes etc.).
NOTE: This changes ``batou.lib.git.Clone`` in backward-incompatible ways:
1. The namevar is now only the ``url``. The previously also accepted usage
of passing the ``target`` as the namevar no longer works.
2. ``update_unpinned`` is gone, use the ``branch``/``revision``/``vcs_update``
mechanism instead.
1.0b25 (2014-07-31)
===================
- Improve gpg handling: don't do --no-tty or --batch any longer as we actually
only run the gpg code in (local) terminal environments usually.
1.0b24 (2014-07-04)
===================
- Actually fix unzipping.
- Remove environment option ``connect_method``, and use autodetection instead:
If we're not the configured ``service_user`` after the plain ssh connect,
run sudo.
- Refactor attributes and overrides: attributes allow template expansion now
and also converting default values. The Attribute class has moved to the
batou component module.
- Yield nonzero returncode on remote exceptions.
- Add convenience converter for attributes "list" to handle comma separated
lists of strings.
- Force safety of Nagios service description field: prohibit "," as its not
safe for Nagios business processes.
- Allow specifying custom rsync options for SyncDir: the '--inplace' option
is a good idea in most cases, but not all. Fixes #11846.
- Make supervisor restart/reload more robust towards non-running supervisors
with lying PID files. Fixes #12744.
- Remove harmful decoding step in File verify that caused wrong results.
1.0b23 (2014-06-30)
===================
- Allow unzip to overwrite existing files if the archive has newer copies.
1.0b22 (2014-06-30)
===================
- Fix uncaught import-error on mysql module.
1.0b21 (2014-06-30)
===================
- Fix bug in bundle shipping that made batou depend on the locale (#13465).
- Add MySQL components.
- Introduce batou.utils.Attribute() to avoid imperative type conversions in
configure().
1.0b20 (2014-05-13)
===================
- Start using python-requests for HTTP/S downloads to support authentication
(and other goodies).
- Improve error logging.
1.0b19 (2014-04-07)
===================
- Allow specifying alternate find-links in requirements.txt
- Remove remnants of the buildout component's ``extends`` option.
1.0b18 (2014-03-31)
===================
- Introduce environment option ``connect_method``, either 'ssh+sudo' (default)
or plain 'ssh'.
- Improve update code: existing installations would not update cleanly to a new
version due to wrong use of pkg_resources API.
Existing installations need to be cleaned manually. :/
- Archive: recognize ``.tar.xz`` files and extract them with ``tar``.
- Fix bug in CMMI verify that would always raise UpdateNeeded due to a path
mismatch (#13145).
- Component.cmd() now performs expand() on the given command by
default. Saves typing:
self.cmd('rm -rf {{component.workdir}}')
instead of
self.cmd(self.expand('rm -rf {{component.workdir}}'))
- File component: apply character encoding when reading and writing files.
(#13165)
- Mercurial clone: changed update behaviour yet again: unless configured
differently, always achieve target state, even by losing uncommitted changes.
- Allow unnamed access to the last added/ored component using the
'self._' attribute:
def configure(self):
self += Download(...)
self += Extract(self._.target)
1.0b17 (2013-12-06)
===================
- Mercurial clone: refuse to update if there are uncommitted changes or
outgoing changesets.
- Mercurial clone: added an option to avoid updating existing clones, which is
often desirable during development.
1.0b16 (2013-11-29)
===================
- Improve the bootstrap/upgrade fix for #13049: the bootstrap template needed a
second improvement.
1.0b15 (2013-11-29)
===================
- Try to fix #13049 - will only work once the dev version is already at least
at b15. Not sure whether it really does the job, though. Hard to test.
- Make environment and host arguments for running ``batou local`` optional.
``dev`` and ``localhost`` are the defaults now. Fixes #13034
- Allow specifying secrets with [component:xy] in addition to [xy] to make it
easier to discover when used to the environment overrides. Both options are
supported now. Fixes #13003
- Refactor Python virtualenv and package management. Provide explicit knowledge
about Python versions 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, and 3.3 to use compatible
virtualenv/pip/setuptools versions and commands. Fixes #12941
NOTE: The API for batou.lib.python.Package has changed in
backwards-incompatible ways:
Package() needs to be added to a VirtualEnv component. Before::
self += VirtualEnv(...)
self += Package(...)
Now::
venv = VirtualEnv(...)
self += venv
venv += Package(...)
Package parameter ``pip_install_options`` has been renamed to
``install_options``.
- Use ``urllib`` module instead of ``wget`` command for cross platform
compatibility.
1.0b14 (2013-11-26)
===================
- Make Buildout component install zc.buildout without dependencies (which is
just setuptools), since that can break the version pinning of setuptools.
- Check whether packages required via a ``-e`` expression in the local
``requirements.txt`` are already installed.
- #12946: Make full path to python executable configurable for virtualenv
(and buildout) component.
- Remove cleanup code that removes batou-0.2-style virtualenv directories (bin,
lib, include etc.) from remote deployments.
- Add mailto setting to crontab component.
- Added some documentation.
1.0b13 (2013-11-19)
===================
- Introduce a plugin mechanism: if a local ``requirements.txt`` file is
present, install its contents into the ``.batou`` virtualenv.
- Load components in alphabetical order. This enables importing from other
components (when naming the directories appropriately).
- #12986: Only try to unbundle if there actually are changes to unbundle.
- Raise an error if a cron job is configured without a timing value.
- Add platform support for Debian (crontab, logrotate, supervisor).
- Added some documentation.
1.0b12 (2013-11-04)
===================
- Added ``branch`` argument to ``mercurial.Clone``. Setting a branch
automatically updates to the branch head on deploy. This is mostly useful for
development environments.
- Create the 'secrets' directory if it doesn't exist, yet. Also, disallow
editing secret files for non-existing environments.
- Support continuing remote bootstrapping if we failed after creating the
initial remote directory but were unable to use Mercurial.
- #12898: build.Configure component was broken when using the default
prefix.
1.0b11 (2013-10-17)
===================
- #12897: Use non-SSL pypi mirror for downloading virtualenv to fix tests
failing randomly on machines that (for some reason) can't validate PyPI's
certificate.
- #12911: Ensure that we can configure file owners when they don't
exist during configure phase yet.
- #12912: Fix untested and broken file ownership management.
- #12847: Clean up unicode handling for File and Content components and
templating.
- #12910: Remote deployments failed when using bundles for transfers if no
changes needed bundling.
- #12766: Allow bootstrapping a batou project in an existing directory to
support migration from 0.2.
- #12283: Recognize files as 'is_template' by default. Auto-detect source files
in the definition directory if they have the same basename. This is what you
want in 99% of all cases. Explicitly stating either the 'content' or 'source'
parameter disables auto-detection.
Now you can write this:
File('foo')
and have `components/x/foo` recognized as the source file and handled as a
template.
- Use ConfigParser instead of configobj which is effectively unmaintained
(see https://twitter.com/voidspace/status/380699699042213889) and support
lists separated by newlines in addition to commas.
1.0b10 (2013-09-27)
===================
- Package our own virtualenv instead of depending on the system-installed one.
This should alleviate troubles due to old virtualenv versions that package
``distribute``, which causes conflicts with recent ``setuptools`` versions
(#12874).
- Update supervisor version to 3.0.
1.0b9 (2013-08-22)
==================
- Update Package component so it ignores installed packages when installing.
This way, we actually install setuptools even when distribute is installed.
(Otherwise it's a no-op since distribute tells pip that setuptools is
already satisfied).
- Fix update process: wrong call to old '.batou/bin/batou' failed and early
bootstrapping would downgrade temporarily which is confusing and superfluous.
Fixes #12739.
1.0b8 (2013-08-17)
==================
- Remove superfluous mkdir call during remote bootstrap.
- Make ``batou init`` print that it's working. Bootstrapping can take a while,
so at least signal that something's going on.
1.0b7 (2013-08-17)
==================
- Depend on Python2.7 to be available on the PATH during early bootstrap.
Otherwise our chances to get a 2.7 virtualenv are pretty small, too.
- Improve project template: ignore the work/ directory by default.
1.0b6 (2013-08-17)
==================
- More MANIFEST inclusions: bootstrap-template.
1.0b5 (2013-08-17)
==================
- Improve MANIFEST so we actually package the init template and other generated
files, like version.txt and requirements.txt.
1.0b4 (2013-08-17)
==================
- Provide a simple project-creation command, both for pip-installed batou's as
well as spawning new projects from existing ones. Fixes #12730
- Fix #12679: make timeouts configurable.
- Removed re-imports from batou main module to support light-weight
self-installation and bootstrapping. I.e. 'from batou import Component' no
longer works.
- Provide a single main command together with a 'bootstrap' wrapper that you
can check into your project and that is maintained during updates
automatically. It also provides fully automatic bootstrapping, installation,
upgrading and other maintenance.
- Fix Python package installation version check.
- Don't use `bin/buildout bootstrap` command anymore. PIP installs a sufficient
`bin/buildout` so buildout can do the rest internally.
- Install zc.buildout during bootstrapping phase using PIP to avoid
bootstrap.py problems.
- Shorten URLs in the Build component to their basename.
- Add 'assert_cmd' API to support simpler assertions for verify when needing to
check the result of an external command.
- Switch to asking pip installing eggs instead of flat installations as
namespaces seem to collide otherwise.
- Remove non-functional deprecated 'md5sum' attribute.
- Components are context managers now. If you provide __enter__ it will be
called before verify() and if you provide __exit__ this will be called after
update (always - even if update isn't actually called). This allows you to
manage temporary state on the target system more gracefully. See the
DMGExtractor for an example.
- Major refactoring of internal data structures to simplify and improve test
coverage. Some breakage to be expected:
* Components do not have a 'service' attribute any longer. If you need access
to the service base directory use 'environment.base_dir'.
- Another attempt at getting convergency for package installation: actually
leverage pkg_resources to check what version of a package is installed. PIP
does not necessarily list or freeze setuptools, even if forced to reinstall.
- Remove superfluous '--update' from pip calls that would temporarily install
the incorrect package version.
- Improve convenience of the 'secrets-edit' wrapper script to allow
re-encrypting without re-entering the editor.
- Consistently switch to using setuptools.
- Fix #12399: incorrect stat attributes for Owner and Group
- Add ``exclude`` parameter to Directory component.
- Add ``env`` parameter to ``Component.cmd()`` (and corresponding
``build_environment`` parameter to the Build component) to allow
adding/overriding environment variables.
1.0b3 (2013-07-09)
==================
- Enable logging in the remote core to see what's going on on the remote side.
- Try to better format exceptions from the remote side.
- Try harder to get virtualenv back into a working state.
- Allow remote deployments from root of repository.
- Make PIP management more robust.
1.0b2 (2013-07-09)
==================
- Add component to manage PIP within a virtual env.
- Add component to manage packages with PIP within a virtual env.
- Restructure buildout component to make it more robust regarding
setuptools/distribute preparation. Also remove usage of bootstrap completely
as we rely on virtualenv anyway.
1.0b1 (2013-07-09)
==================
- Apply semantic versioning: initial development is over, so this is 1.0 now.
- Major revamp of secrets management:
- switch to GPG (instead of aespipe)
- turn secrets into a core feature, removing the need for a special component
- Add '--single' to suppress parallel bootstrapping.
================
Developing batou
================
:Issue tracker:
https://projects.gocept.com/projects/batou/
:Source code:
https://bitbucket.org/gocept/batou/
=====
batou is a multi-(component|host|environment|...) deployment utility.
Deployments are described in a model using "components" and can be deployed
locally and remotely. Models are written as Python code.
See the `batou documentation <http://batou.readthedocs.org>`_ for details
on how to use it.
=========
Changelog
=========
1.0b27 (2014-09-15)
===================
- Do not configure environments locally when running remote deployments. Use
order as determined by one reference remote host. #13530
- Nagios configuration now supports contact groups.
1.0b26 (2014-08-05)
===================
- Enhance git Clone component to support the full suite of functionality the
mercurial one has (incoming/outgoing/local changes etc.).
NOTE: This changes ``batou.lib.git.Clone`` in backward-incompatible ways:
1. The namevar is now only the ``url``. The previously also accepted usage
of passing the ``target`` as the namevar no longer works.
2. ``update_unpinned`` is gone, use the ``branch``/``revision``/``vcs_update``
mechanism instead.
1.0b25 (2014-07-31)
===================
- Improve gpg handling: don't do --no-tty or --batch any longer as we actually
only run the gpg code in (local) terminal environments usually.
1.0b24 (2014-07-04)
===================
- Actually fix unzipping.
- Remove environment option ``connect_method``, and use autodetection instead:
If we're not the configured ``service_user`` after the plain ssh connect,
run sudo.
- Refactor attributes and overrides: attributes allow template expansion now
and also converting default values. The Attribute class has moved to the
batou component module.
- Yield nonzero returncode on remote exceptions.
- Add convenience converter for attributes "list" to handle comma separated
lists of strings.
- Force safety of Nagios service description field: prohibit "," as its not
safe for Nagios business processes.
- Allow specifying custom rsync options for SyncDir: the '--inplace' option
is a good idea in most cases, but not all. Fixes #11846.
- Make supervisor restart/reload more robust towards non-running supervisors
with lying PID files. Fixes #12744.
- Remove harmful decoding step in File verify that caused wrong results.
1.0b23 (2014-06-30)
===================
- Allow unzip to overwrite existing files if the archive has newer copies.
1.0b22 (2014-06-30)
===================
- Fix uncaught import-error on mysql module.
1.0b21 (2014-06-30)
===================
- Fix bug in bundle shipping that made batou depend on the locale (#13465).
- Add MySQL components.
- Introduce batou.utils.Attribute() to avoid imperative type conversions in
configure().
1.0b20 (2014-05-13)
===================
- Start using python-requests for HTTP/S downloads to support authentication
(and other goodies).
- Improve error logging.
1.0b19 (2014-04-07)
===================
- Allow specifying alternate find-links in requirements.txt
- Remove remnants of the buildout component's ``extends`` option.
1.0b18 (2014-03-31)
===================
- Introduce environment option ``connect_method``, either 'ssh+sudo' (default)
or plain 'ssh'.
- Improve update code: existing installations would not update cleanly to a new
version due to wrong use of pkg_resources API.
Existing installations need to be cleaned manually. :/
- Archive: recognize ``.tar.xz`` files and extract them with ``tar``.
- Fix bug in CMMI verify that would always raise UpdateNeeded due to a path
mismatch (#13145).
- Component.cmd() now performs expand() on the given command by
default. Saves typing:
self.cmd('rm -rf {{component.workdir}}')
instead of
self.cmd(self.expand('rm -rf {{component.workdir}}'))
- File component: apply character encoding when reading and writing files.
(#13165)
- Mercurial clone: changed update behaviour yet again: unless configured
differently, always achieve target state, even by losing uncommitted changes.
- Allow unnamed access to the last added/ored component using the
'self._' attribute:
def configure(self):
self += Download(...)
self += Extract(self._.target)
1.0b17 (2013-12-06)
===================
- Mercurial clone: refuse to update if there are uncommitted changes or
outgoing changesets.
- Mercurial clone: added an option to avoid updating existing clones, which is
often desirable during development.
1.0b16 (2013-11-29)
===================
- Improve the bootstrap/upgrade fix for #13049: the bootstrap template needed a
second improvement.
1.0b15 (2013-11-29)
===================
- Try to fix #13049 - will only work once the dev version is already at least
at b15. Not sure whether it really does the job, though. Hard to test.
- Make environment and host arguments for running ``batou local`` optional.
``dev`` and ``localhost`` are the defaults now. Fixes #13034
- Allow specifying secrets with [component:xy] in addition to [xy] to make it
easier to discover when used to the environment overrides. Both options are
supported now. Fixes #13003
- Refactor Python virtualenv and package management. Provide explicit knowledge
about Python versions 2.4, 2.5, 2.6, 2.7, 3.1, 3.2, and 3.3 to use compatible
virtualenv/pip/setuptools versions and commands. Fixes #12941
NOTE: The API for batou.lib.python.Package has changed in
backwards-incompatible ways:
Package() needs to be added to a VirtualEnv component. Before::
self += VirtualEnv(...)
self += Package(...)
Now::
venv = VirtualEnv(...)
self += venv
venv += Package(...)
Package parameter ``pip_install_options`` has been renamed to
``install_options``.
- Use ``urllib`` module instead of ``wget`` command for cross platform
compatibility.
1.0b14 (2013-11-26)
===================
- Make Buildout component install zc.buildout without dependencies (which is
just setuptools), since that can break the version pinning of setuptools.
- Check whether packages required via a ``-e`` expression in the local
``requirements.txt`` are already installed.
- #12946: Make full path to python executable configurable for virtualenv
(and buildout) component.
- Remove cleanup code that removes batou-0.2-style virtualenv directories (bin,
lib, include etc.) from remote deployments.
- Add mailto setting to crontab component.
- Added some documentation.
1.0b13 (2013-11-19)
===================
- Introduce a plugin mechanism: if a local ``requirements.txt`` file is
present, install its contents into the ``.batou`` virtualenv.
- Load components in alphabetical order. This enables importing from other
components (when naming the directories appropriately).
- #12986: Only try to unbundle if there actually are changes to unbundle.
- Raise an error if a cron job is configured without a timing value.
- Add platform support for Debian (crontab, logrotate, supervisor).
- Added some documentation.
1.0b12 (2013-11-04)
===================
- Added ``branch`` argument to ``mercurial.Clone``. Setting a branch
automatically updates to the branch head on deploy. This is mostly useful for
development environments.
- Create the 'secrets' directory if it doesn't exist, yet. Also, disallow
editing secret files for non-existing environments.
- Support continuing remote bootstrapping if we failed after creating the
initial remote directory but were unable to use Mercurial.
- #12898: build.Configure component was broken when using the default
prefix.
1.0b11 (2013-10-17)
===================
- #12897: Use non-SSL pypi mirror for downloading virtualenv to fix tests
failing randomly on machines that (for some reason) can't validate PyPI's
certificate.
- #12911: Ensure that we can configure file owners when they don't
exist during configure phase yet.
- #12912: Fix untested and broken file ownership management.
- #12847: Clean up unicode handling for File and Content components and
templating.
- #12910: Remote deployments failed when using bundles for transfers if no
changes needed bundling.
- #12766: Allow bootstrapping a batou project in an existing directory to
support migration from 0.2.
- #12283: Recognize files as 'is_template' by default. Auto-detect source files
in the definition directory if they have the same basename. This is what you
want in 99% of all cases. Explicitly stating either the 'content' or 'source'
parameter disables auto-detection.
Now you can write this:
File('foo')
and have `components/x/foo` recognized as the source file and handled as a
template.
- Use ConfigParser instead of configobj which is effectively unmaintained
(see https://twitter.com/voidspace/status/380699699042213889) and support
lists separated by newlines in addition to commas.
1.0b10 (2013-09-27)
===================
- Package our own virtualenv instead of depending on the system-installed one.
This should alleviate troubles due to old virtualenv versions that package
``distribute``, which causes conflicts with recent ``setuptools`` versions
(#12874).
- Update supervisor version to 3.0.
1.0b9 (2013-08-22)
==================
- Update Package component so it ignores installed packages when installing.
This way, we actually install setuptools even when distribute is installed.
(Otherwise it's a no-op since distribute tells pip that setuptools is
already satisfied).
- Fix update process: wrong call to old '.batou/bin/batou' failed and early
bootstrapping would downgrade temporarily which is confusing and superfluous.
Fixes #12739.
1.0b8 (2013-08-17)
==================
- Remove superfluous mkdir call during remote bootstrap.
- Make ``batou init`` print that it's working. Bootstrapping can take a while,
so at least signal that something's going on.
1.0b7 (2013-08-17)
==================
- Depend on Python2.7 to be available on the PATH during early bootstrap.
Otherwise our chances to get a 2.7 virtualenv are pretty small, too.
- Improve project template: ignore the work/ directory by default.
1.0b6 (2013-08-17)
==================
- More MANIFEST inclusions: bootstrap-template.
1.0b5 (2013-08-17)
==================
- Improve MANIFEST so we actually package the init template and other generated
files, like version.txt and requirements.txt.
1.0b4 (2013-08-17)
==================
- Provide a simple project-creation command, both for pip-installed batou's as
well as spawning new projects from existing ones. Fixes #12730
- Fix #12679: make timeouts configurable.
- Removed re-imports from batou main module to support light-weight
self-installation and bootstrapping. I.e. 'from batou import Component' no
longer works.
- Provide a single main command together with a 'bootstrap' wrapper that you
can check into your project and that is maintained during updates
automatically. It also provides fully automatic bootstrapping, installation,
upgrading and other maintenance.
- Fix Python package installation version check.
- Don't use `bin/buildout bootstrap` command anymore. PIP installs a sufficient
`bin/buildout` so buildout can do the rest internally.
- Install zc.buildout during bootstrapping phase using PIP to avoid
bootstrap.py problems.
- Shorten URLs in the Build component to their basename.
- Add 'assert_cmd' API to support simpler assertions for verify when needing to
check the result of an external command.
- Switch to asking pip installing eggs instead of flat installations as
namespaces seem to collide otherwise.
- Remove non-functional deprecated 'md5sum' attribute.
- Components are context managers now. If you provide __enter__ it will be
called before verify() and if you provide __exit__ this will be called after
update (always - even if update isn't actually called). This allows you to
manage temporary state on the target system more gracefully. See the
DMGExtractor for an example.
- Major refactoring of internal data structures to simplify and improve test
coverage. Some breakage to be expected:
* Components do not have a 'service' attribute any longer. If you need access
to the service base directory use 'environment.base_dir'.
- Another attempt at getting convergency for package installation: actually
leverage pkg_resources to check what version of a package is installed. PIP
does not necessarily list or freeze setuptools, even if forced to reinstall.
- Remove superfluous '--update' from pip calls that would temporarily install
the incorrect package version.
- Improve convenience of the 'secrets-edit' wrapper script to allow
re-encrypting without re-entering the editor.
- Consistently switch to using setuptools.
- Fix #12399: incorrect stat attributes for Owner and Group
- Add ``exclude`` parameter to Directory component.
- Add ``env`` parameter to ``Component.cmd()`` (and corresponding
``build_environment`` parameter to the Build component) to allow
adding/overriding environment variables.
1.0b3 (2013-07-09)
==================
- Enable logging in the remote core to see what's going on on the remote side.
- Try to better format exceptions from the remote side.
- Try harder to get virtualenv back into a working state.
- Allow remote deployments from root of repository.
- Make PIP management more robust.
1.0b2 (2013-07-09)
==================
- Add component to manage PIP within a virtual env.
- Add component to manage packages with PIP within a virtual env.
- Restructure buildout component to make it more robust regarding
setuptools/distribute preparation. Also remove usage of bootstrap completely
as we rely on virtualenv anyway.
1.0b1 (2013-07-09)
==================
- Apply semantic versioning: initial development is over, so this is 1.0 now.
- Major revamp of secrets management:
- switch to GPG (instead of aespipe)
- turn secrets into a core feature, removing the need for a special component
- Add '--single' to suppress parallel bootstrapping.
================
Developing batou
================
:Issue tracker:
https://projects.gocept.com/projects/batou/
:Source code:
https://bitbucket.org/gocept/batou/
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
batou-1.0b27.zip
(115.0 kB
view hashes)