Python wrapper for the C-Blosc2 library
Reason this release was yanked:
Introduced a dependency on blosc2-private.h
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the full C-Blosc2 functionality to support super-chunks (SChunk), multi-dimensional arrays (NDArray), metadata, serialization and other bells and whistles introduced in C-Blosc2.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
A SChunk is a simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.5.0-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76fb57f887ee448b2360239da7e7480f3adb4e32297ab63e5f7dc1b3f236f629 |
|
MD5 | 43c52ff4fcf11dc6465176c9c0b32352 |
|
BLAKE2b-256 | b03bc5c44dcc724ea8f061d797ea57e2328e0375e6b0cb9d4c0ccab29a5ceb66 |
Hashes for blosc2-2.5.0-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7eab81b1796bb7d0b356217bbf75b528e9bfd28564cbcd10756117f2411b1094 |
|
MD5 | 11edc75e0f861fde9a2c3166eca18370 |
|
BLAKE2b-256 | ade64eed7be2b5b647d49bd11cfeaf35a3136afc6e78e6aec9b6f7daee683787 |
Hashes for blosc2-2.5.0-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3ec31a66bcfd75a179a2b6ae489ee06fe68ca02a8e4f4e817f567b4e9182ea17 |
|
MD5 | 47b920bc33216581e5f0a01f611d0e7c |
|
BLAKE2b-256 | 587eef4c349894d209ed79ac43488fc4a5eae6eca6a139c35974a23b2a29f111 |
Hashes for blosc2-2.5.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 467dd2a24559cab24266f6d97cc22600489d32c6759e57cca772f4c2c13075fc |
|
MD5 | 280cc9dae3f35f74c6a96f06f5b3424f |
|
BLAKE2b-256 | 7d22342b5b576eecf7d988fb0ab20c1e3441224fd6ad7cadb7b6ff8c3c600571 |
Hashes for blosc2-2.5.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 754a0163d57fe7f4a723bf857d5cd38765e2cd7679c7c7787431207cacf45f55 |
|
MD5 | e9fb11391b5ecd5662f51aaf056d00fc |
|
BLAKE2b-256 | 9437c76c8ec792af0b8b92d6672c0767ed833ddb13e01fe713da984e73d26cd5 |
Hashes for blosc2-2.5.0-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 81c20f8f30017d5bcf3f0abd460edc823505d3b3431f5536bcd52eb5a59a3cd6 |
|
MD5 | e32ce0d6202d6be55fb6d544f05226c1 |
|
BLAKE2b-256 | 9cd89c2f91b2b130650803752cf4d2eea4c0b1d0703ab41555c9667ecf067c74 |
Hashes for blosc2-2.5.0-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b57782551062a4315e6b36fdd383550bac83f6c5e14a6a2497144fdd38ce1f8 |
|
MD5 | 9b18074e56c706207be2fc297fe65d57 |
|
BLAKE2b-256 | 8b6176783f195631a64152682c6eb94b33c47130874c307ea443744d433889e1 |
Hashes for blosc2-2.5.0-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac1a777846c154d03442ff38cee655fc8e961c23756bd5f245436bb611844121 |
|
MD5 | 60ebaf09e0a2933b96151b2c38909038 |
|
BLAKE2b-256 | 2c30a3078339465ca6b8af3dd1c76121bfe39af328e8d763da751871eea573a3 |
Hashes for blosc2-2.5.0-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ae2ac317ec951a81ad85da6160332f4937671009b2e30737c255337ade03ae1 |
|
MD5 | 4bc61c9ef5fc589baaf761e6f1a6bd2c |
|
BLAKE2b-256 | b8b12c037a903af77f9df565034c37bd3987f28eb7a1b398a11e7354d5584fc7 |
Hashes for blosc2-2.5.0-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1636b57b08acf5246a218e2b8158fe36a127f0f80ffdd59cbe24c3ad98fd9724 |
|
MD5 | ead68792f8cf482a575ba6abd9aafc4e |
|
BLAKE2b-256 | 1b6ff369f00ac4339a7734d7a8b4ab4c2b671394a26a55fafcd5b97a4b71daa7 |
Hashes for blosc2-2.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22fb9b973ee072f93847a2d9c0ebce01af138b0f8287e19d5b5c263292fedfaa |
|
MD5 | ea1c0420e99b5506bf89b73cf6a6058d |
|
BLAKE2b-256 | fe9e02a269de3259bd0dfad1ca505d6557a9a4d034e3f7f6ad303e87fc8432c3 |
Hashes for blosc2-2.5.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7e4291db52ff464bee24e3265b7a7617327375116cc328258398bd5874eb5e4 |
|
MD5 | a02a4d7bf04ada8c384795b15b33106c |
|
BLAKE2b-256 | a31a8a8e99850b93c7749a731ae74134a1e9096e467b7fe98b68421f5ee0f74e |
Hashes for blosc2-2.5.0-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a517fad26c0aad121751b51649bfc389b4967417df9df975020cac988bad721f |
|
MD5 | 32a25fff932c2d6335e21e5a86062caf |
|
BLAKE2b-256 | b95abe1dab83a4abad00f9c4ca05c90a6e98b4dcdf70849f6d99fb725940b168 |
Hashes for blosc2-2.5.0-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67e668665e14808458491bf488a81d99a0920b316328269279f6fb774b7cb1a2 |
|
MD5 | 9103477c09331c396510039d37a19f50 |
|
BLAKE2b-256 | fd344004381c3124dc8ce827a1825d5173ddb54c6e456859f7bf7c2323fafde5 |
Hashes for blosc2-2.5.0-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0157caebef7a4939f05a572e10cf32415ad73514eb0fc80f22dbb8feafc34e5 |
|
MD5 | 5e7b04050395e973c933cc8ae25af829 |
|
BLAKE2b-256 | d68087b489eb327c89c14f79848fe71fe43b7f5e50adb9bb6ed310dafb683fad |
Hashes for blosc2-2.5.0-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0c0dd86c94e4a015af3639b6078dd050a2cee1d6a3222686f6750c05de01f2e |
|
MD5 | 64b3d5af6c9b3c030fcd4811ebacbeb3 |
|
BLAKE2b-256 | 3fa9fda8cc8f8c9bf2747e5d4c5bfc70eeee476660087501a9f07adba5e78e8d |
Hashes for blosc2-2.5.0-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f246a860ffea3107c6c189aa61e068906349a3e1fcf8713e5136f467d2c2b082 |
|
MD5 | 4b574dc77e137e773db33d2b4e3cf5c0 |
|
BLAKE2b-256 | f99bffb5aa4ec6ce2b60edecfeb8c35132b9a65c8dcb9083368d74095bb5fef7 |
Hashes for blosc2-2.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5aa12e52b50a86a0b6dc01d1f55029c251a1939b7691e1b9b487449148545a4a |
|
MD5 | c726b33ae0c6ce9d52e160f78cbe5948 |
|
BLAKE2b-256 | 9a51f758835851acfe6c80adfdba929607c99c2b8621e70fb22664ffb2fa058f |
Hashes for blosc2-2.5.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89d187fa27dc4b708b704bfd0c8f2cd71112ea145f4c481dfefbb31c645d2523 |
|
MD5 | f863d32e88b63583888fdb091ba3a8d7 |
|
BLAKE2b-256 | 8cfbd228e507e4f3e7ded30024ced972a3f0ce2126c2588b3996d0741b36bca7 |
Hashes for blosc2-2.5.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab062bdd120cfcb47f9fe9bc94c3a5d621b260118f63ac1b6d326f75da065e88 |
|
MD5 | dae9990b981b14ff4486fae310b55b08 |
|
BLAKE2b-256 | 0f0b1c6b06cc38073e570b43cddec2e12cba2b6eee5081acd3d7393a27b1a722 |
Hashes for blosc2-2.5.0-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d73af9af7b0841ce3d797c00240079ed7ce7831e20233c5bc3e3f06ebe2d029 |
|
MD5 | 9216853ffa3fbdd086b74d687c454b7e |
|
BLAKE2b-256 | 328be7652d5c10bc0f645a975e7e1c9e6aa4d28c46e48e630f26324c207aec4d |
Hashes for blosc2-2.5.0-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ce472e1289742adc06d1e3cf21ed5caddbdc6ed590d3601ce51ac3d7e39ab00 |
|
MD5 | aa0849940e19859ab74e5884626ed3b5 |
|
BLAKE2b-256 | 8269e4cdccc1e05e899515a7a63b076065c077b5ed035be5e051a63c32d7a8f3 |
Hashes for blosc2-2.5.0-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1761a9f7ed4fc5286d1298b45541b330396472d6490f39cd4e3a0de0e29efba3 |
|
MD5 | e172a2068007e944a4ecefa44784aa85 |
|
BLAKE2b-256 | 5e4f2cab68004bab21b552f2c77ce021f5a50d99691e1055d3baa9b02aba0fd3 |
Hashes for blosc2-2.5.0-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fbcdfa3df90619723b30ee89f3bb1c42ef12fe774ce886a86e3d67a5c7d2c2a |
|
MD5 | 567198c8cd441de3aa945dafacc6f35c |
|
BLAKE2b-256 | d01200574566c7de8408d83903feae80e31d3ce3811a56e49ce0e3f4221adb5d |
Hashes for blosc2-2.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45b93b0ace9ae8f0a9c4efc4d6a18da364baa520ad3eea2569a1751e748e530a |
|
MD5 | b5dcc6d7032f5dadbbda95ea7d35fe4c |
|
BLAKE2b-256 | 75472f3e75edb2f687bbb9213ae34f1eccc67486996d9ec9574275610ce1b224 |
Hashes for blosc2-2.5.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afdb13f551121828511e9c88503b97ad5a5b651fbb9239546878b2ae7d052478 |
|
MD5 | c61c37ea810566fc08e5f3da3f296a0c |
|
BLAKE2b-256 | f629a5363b201fdc5818fe27d8ad4037617e0ad2bf27f58eded88f7e9e1aabb5 |
Hashes for blosc2-2.5.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d748054ab78356598853cf3398821d16696c7e776ecdeb6a28d56ea35064bd92 |
|
MD5 | a1ec7455d300a9942f72b58b64973608 |
|
BLAKE2b-256 | e73e4bad7e89e593b4389f4c4f244158fcfe14ac3f92548f6f8eb7273ab9580c |
Hashes for blosc2-2.5.0-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20b1d0c3e3da41991f9f9189ecdad6a42523c7e9420125e02aa4b87f007a0d00 |
|
MD5 | c93e5f34bb5eb1fb2fbde420da1bd9de |
|
BLAKE2b-256 | 588b7f649a3fef97414fe5de6d43b816fce16e58193c4b56685200bcab804451 |