Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.3.1-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a973bb5f4ba8f17e094b2aec92679b5d0ce9ef01ca2e4c6373771297f342b62 |
|
MD5 | fa16af50885ea07e0e1f9395647a9890 |
|
BLAKE2b-256 | d2f9d79b35304ac811e2509b9942bc884f4ff7d8cb573157ceac006383053400 |
Hashes for blosc2-2.3.1-cp312-cp312-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66abff3be98d9d1399c91a1c6b9aff5bf1c3dc98197d34c9e67d098f575ec194 |
|
MD5 | 760d145a3355bd043171d2272e2fddd1 |
|
BLAKE2b-256 | 98de7e997bc7b4dc2d85a69650244d03d1f3e13d80d1d7b8d95844a26a968ba2 |
Hashes for blosc2-2.3.1-cp312-cp312-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1a0c3e0c95eb88b86fb06b59baeaeff1baf3f6c30da6b7627fc9a7520b4c010 |
|
MD5 | 95fc8419aca5f29f206bc53d34763c3f |
|
BLAKE2b-256 | eedd92fd778aa6f9b166bc479a105d8a8e96303fe9084b846fc293867f754300 |
Hashes for blosc2-2.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4dbaa098b3c10a161280498a063ea7d381a3b7dbdb17a6c66810372640db9075 |
|
MD5 | 4b9efbcbb7de8bf35871a79a8d91f33e |
|
BLAKE2b-256 | a56aa98811c80f0be26b445d822f74b4f7d84a8b8c212119aa5c1b5be1718daf |
Hashes for blosc2-2.3.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b0dfd563a2bbcbc3c5f2f10fec53862a4bc02d0e7057dfb1bec45bb7a26122d |
|
MD5 | 8f50d7a08c8c1d42b04c2a1fa315a64d |
|
BLAKE2b-256 | 545cf539ac7fd1abfcfbee9fb658b0734eb954efc1420db7a6dee7e8b14e2498 |
Hashes for blosc2-2.3.1-cp312-cp312-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b392772e333a890bd42abad30ef1ba0dfdd03b734fed001c55382019d8b1c1aa |
|
MD5 | cc27102574ab4b987db507373ffa1e4a |
|
BLAKE2b-256 | 568df60c17106cdef080793897f44605d2f53f325623771237a26893db4c0091 |
Hashes for blosc2-2.3.1-cp312-cp312-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | afb4b46ea0597cc90ae40f51d7bdb9460d91dcfd45f88f89dce89b808b209102 |
|
MD5 | 6e6106552a2e2b6190fd331aaaf71470 |
|
BLAKE2b-256 | 505971ac5b067b27b2d94d315098c0e8a8dec579b4c4416f7d3f809d206c43d8 |
Hashes for blosc2-2.3.1-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 545c580935b568c8e1b476b75902b27e886d7f7975e1f6e6518b69b063150910 |
|
MD5 | e424c1618d3c45f31423099ebd7243a4 |
|
BLAKE2b-256 | c13e067fb60fc571c0894c30371b8fa910e75cf43e82a3e7a6d5356e42910960 |
Hashes for blosc2-2.3.1-cp311-cp311-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be5daf95854fa14eb6c0f1de86f1afe388c9a5cb6dd2be05c31163796bddfef8 |
|
MD5 | 96781d871d7e74b4ee290c5b9c8ec6df |
|
BLAKE2b-256 | 5d88591fc07f4df495b87e82c776c9ceff21dec669e08c0d7ff7b3e48ee69582 |
Hashes for blosc2-2.3.1-cp311-cp311-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4f479cd4e7d95eedb69ff72d803a2e8b1eabdc5724b2da7bd5a6f4d73c96833 |
|
MD5 | 79c9b4b21587ec4ac1e71f88b694c802 |
|
BLAKE2b-256 | 2d59093e25b5ab56234e9c238ab03d908ba39238518f1b62e21e75f1ee6fef54 |
Hashes for blosc2-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6bd982b437be5a3d21dc7f6dfa1e9febe056be090c21208c8422bb2013d36046 |
|
MD5 | e669af58901fbb10b44a70d972552feb |
|
BLAKE2b-256 | 054e4aa4e1cdbf290fcda499951f41f0a68564949f8a462e516778ce8280b679 |
Hashes for blosc2-2.3.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00d000a6b42044e970de18495007b0a3de2d13385fad3b3bc48c9376b1531a3b |
|
MD5 | 8830afbc55fbf987c5efdfdf3bf1524a |
|
BLAKE2b-256 | 1aef6c31d0cf4cf419dcf391020789e507c312fc909d4c179eec529763917a48 |
Hashes for blosc2-2.3.1-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62354c09aa8ba12a63e7dfb23659f40866752eb2f2d3f55ebc65485f9f7cdefc |
|
MD5 | c7fac90643fb358d757486ce96430c11 |
|
BLAKE2b-256 | 7108ce7233ed4496209920f6e7953ea23dd6a069936db9b29dcbe1441204168e |
Hashes for blosc2-2.3.1-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de07c31c65b3abcd62338dcf929f67aa81e9eb8dbaa6b6d52e4dd6a18962a781 |
|
MD5 | 16bef9e0ddc2118ef0a2ac10aff7a4d1 |
|
BLAKE2b-256 | 8e40778c577c914ebc6e6b49d612a281cd281bdd44220bb7dc53e853244663dd |
Hashes for blosc2-2.3.1-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4ba2f1c7af60efb414783fd13b61491e3c542dfe1415c8416210d6759873e7c |
|
MD5 | f78ed96f6003cb30fb04f3d907fa8a03 |
|
BLAKE2b-256 | c6431f17c87af8684829b089a2f4c4bd46010b8229654e79e2de4843931f97fe |
Hashes for blosc2-2.3.1-cp310-cp310-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 946c755b48cc088e02ddc327407cd9d06f8afd20f315eb4dccdab116b3a1add6 |
|
MD5 | 758085a2d7a9f6f0a191fd44b2e5f74f |
|
BLAKE2b-256 | 082b455474a0b102a8e146d3b5a58efb52ec0339ff0f12cc6649890127c5d0ee |
Hashes for blosc2-2.3.1-cp310-cp310-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2acd72742a21fe2ad8f77e92bc493af5aecbe62e08bbb7324c5ae1f61d53c3a |
|
MD5 | 09e555f1220beb02b32f3ab47b496c78 |
|
BLAKE2b-256 | 9ca0864f0e33b1e9586a16766b073020779497bc6b31cafb1e9d2b5d18892d16 |
Hashes for blosc2-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a7e9e71daaaa5ab12c9766801a3884c21e8829c07131416faa6bd9c2af5f797 |
|
MD5 | 0d0a0ad5f5a577f6a4919b74a2277626 |
|
BLAKE2b-256 | de69ff8fc14ed656f49176cff7bfb03454631079f596cfa48647f8766b9361dc |
Hashes for blosc2-2.3.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a690d3ee5d68296874e5069865118f577a0d1dc5697edb19394e174ab88506fc |
|
MD5 | f50823faa4fbae7d55aae359190fec36 |
|
BLAKE2b-256 | ed70896395d9b787ce12b64383f9ca7ec592041e78343248da5cc462207d0b5f |
Hashes for blosc2-2.3.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1f0e3f15f7755637a6681e746b760e9d46ed61d8b53fa5e59a1b22b73f67a96 |
|
MD5 | a65a5f5ef1d7ac0a0a3cae3f7137b465 |
|
BLAKE2b-256 | 7ca6e621bd797e7874d0997ca6bac4dd3b527c1ed0c67fbb43e0fa1f72a57a90 |
Hashes for blosc2-2.3.1-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 70d7b5993ed6af56d2d76a82edd03d06fdd7eb6f735f1daeca63ddbd99bde16c |
|
MD5 | 6c065d513c5fde1a7c65d92656e8d273 |
|
BLAKE2b-256 | 5f43bddbdc61bce8e7277a3d52af0f904da19878075ca8900e3387d81beab981 |
Hashes for blosc2-2.3.1-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 042164f59f5949745438edcf52a1720e6a142f5494ddba2bdd6646acf88d35d3 |
|
MD5 | 746e777e3526561dda4ff0d1e62dee07 |
|
BLAKE2b-256 | 28634faf0c2af99b6bf6b5430d0c86b4ed209c46651d4e3f1606b8596f077d13 |
Hashes for blosc2-2.3.1-cp39-cp39-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a2ea0b358e60a23138556fdc24abdbf557247280b35445634178f2f0f1241a17 |
|
MD5 | 311168f5ec104e0df11c0aa588490820 |
|
BLAKE2b-256 | ddad20a374e7943be19352eb68cd82e770486ddd1fffba65c239c0f0e7902310 |
Hashes for blosc2-2.3.1-cp39-cp39-musllinux_1_1_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14d70bf7c7a819b286b0cceca778ecd4f5dc0ca31ee288496e838de6f3d1fd46 |
|
MD5 | 274b40d1e9385f742bf8f4cc3199b9c4 |
|
BLAKE2b-256 | b909594740c070249756135c87e7720c74cdff64bff53dc940e3d509715e0c0d |
Hashes for blosc2-2.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec79f835cb5bbb6ccd45261de73451976487c9d34a3be39b2b2b2582873c237d |
|
MD5 | 0e82c69e1ec1a91f940de94e03fd420c |
|
BLAKE2b-256 | 069f6a08a792ad4633cdedd17adb0c83cba4e3692ad4c2de11bee729d3313be6 |
Hashes for blosc2-2.3.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba13b4491680d65c4caccffd83d54dcc17e8236b9241710a0cc3cded56a5d9bf |
|
MD5 | 9154e03e29e870ef3b021fb9697e8d2e |
|
BLAKE2b-256 | 2effb52c3536b9e1eecd137e0ed9fd86cc5f89aafbb6613f7a538ca380001700 |
Hashes for blosc2-2.3.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 976d4d4525ab1475414e2dd44375d88da4a5c395b2334c3d5ede0b7fd836f8b5 |
|
MD5 | 6e6b6649e45b6b49760c429ca6887285 |
|
BLAKE2b-256 | 247462f7cc30ab2d82fb66f79e2b7f39f62c357045198a11a66aac5cbea12f4d |
Hashes for blosc2-2.3.1-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff61c594d96c6830dd5b7928a12d7562330b7e31d89cd4794aea7354c89ae71a |
|
MD5 | f93fd72a2a1255499a7de52647ddbe60 |
|
BLAKE2b-256 | 7f2670072c453cdd6675a426a019d6e192ce3acca7d95320cf4bd01343b61cbd |