Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Citing Blosc
You can cite our work on the different libraries under the Blosc umbrella as:
@ONLINE{blosc,
author = {{Blosc Development Team}},
title = "{A fast, compressed and persistent data store library}",
year = {2009-2023},
note = {https://blosc.org}
}
Enjoy!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.6-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf4bb1b89a73b1fbaf65d0606f2e47be04c7e32eb3f5d6038c00d3032c4a2de3 |
|
MD5 | 69da01be5ec894e3ff96a4157c443746 |
|
BLAKE2b-256 | e3fb03891d997e080d5be2330c91c8c47be60bf859b1ddf71ac818d9e431dcfa |
Hashes for blosc2-2.2.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c75f8acf6a4ea6034d642d5235cc99fa60bd5ea7117a0fef444c4a915ecfe7a |
|
MD5 | 44a8f583dc8a3c6623af431a24c61691 |
|
BLAKE2b-256 | e3786606511b4a580277185dd4cd5d052f63fd98c25ad0f7ab414705c28650d6 |
Hashes for blosc2-2.2.6-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a71163a61350c75b7543c0a4ed84b5d506f1df03f75fac1a8f74b13161698f65 |
|
MD5 | d2b30b168c0c53209c99c88bd2dce9a2 |
|
BLAKE2b-256 | af3108853865a27a0b2ee46cf1ec2c6eb89043a06f17c5b7dcd9c3d00c847d19 |
Hashes for blosc2-2.2.6-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c611cfdf0fb38cacaaf21cb7f207d7f01fa806f8ae822c923afbb49897d4124 |
|
MD5 | 9edf245362974fc0470ac4e9391cde91 |
|
BLAKE2b-256 | 2bac3e24f39e7714dbcdc0e7d2a4e5573ce0962c2af66a584cd69417bd729394 |
Hashes for blosc2-2.2.6-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c45291007cacbcb5c46ba22476e85fe6b64efbb2e796079bd619e891e00fff1 |
|
MD5 | f53523a5d5608e17d085bb9978994ea1 |
|
BLAKE2b-256 | 2159dfdbea93bebc1dc52fa234774423d9cf83d30ce306d7afe6f08613153fa3 |
Hashes for blosc2-2.2.6-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cb8714ac35f7a52860dde1c767930e568d4a8f6faf503763a06970ff59853259 |
|
MD5 | 1ccdbd840335715465a7d23c803e5fa1 |
|
BLAKE2b-256 | 4c672303069e8e93978e0b9c6c6381a53d33d61b4cdfde577e6e19829112bdd7 |
Hashes for blosc2-2.2.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c22009aab27f5e78c39b2281a7a6add3a26dd02fb71e183d7f1a2997a7ff458 |
|
MD5 | ae269e7d460e0fe446e5393c2bdd0697 |
|
BLAKE2b-256 | 93e4e741888c629c48688825bd13c7c952cb3039ab18d6c01686b19103551bcb |
Hashes for blosc2-2.2.6-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a85bcb247fba4d833aeb2a82b5fe47f4c5545c2cb61dc37618d0739b4563cd56 |
|
MD5 | 49d1019f5baeadb570dbba579d8e9e56 |
|
BLAKE2b-256 | 0234948498cafb3c521fc21cbb8a6a98f82dc6bef5edb2d14cf52af47bc2df0b |
Hashes for blosc2-2.2.6-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3a6bbdcddb99506de0df08ea21b0e98202534f08454d645a644719d6b283a2a |
|
MD5 | dcde4e6ea807c5f1df5fa6bce95b74a1 |
|
BLAKE2b-256 | 570d9d1799da7456616961f80a135f6656bc515180a3d303bfb44dc4ad15c782 |
Hashes for blosc2-2.2.6-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2855a6c95d4ca919e80ced065b8c193569d43ab4e199a662a4cc1e213492b1d6 |
|
MD5 | d44380de324ed547f8133b140c6671dd |
|
BLAKE2b-256 | 2ae8bbd6b1cbd3fbc50f6b5d6e3c52b3fceb6ce9a20d7360bbcf8247e91abf77 |
Hashes for blosc2-2.2.6-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74425f674446304270b38e326a1ce72d2ae0e80f142dacd46ce2872a99fad671 |
|
MD5 | a42a600f6d52a3664155f4fb4352e149 |
|
BLAKE2b-256 | e6f2c619ac4c85ed6c84976295ba9519c248f8f66d7b52e815bf90826a10b772 |
Hashes for blosc2-2.2.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95ce6b09399c15138e19e00689a65ba2b9e96a92212d100b7bd20ba04658f2d8 |
|
MD5 | a057148c1a251ec15b1f129b0056ce16 |
|
BLAKE2b-256 | b72947e22c89a4a005f19261975c45eec8793cf0b2c514bcab2f445b1868d019 |
Hashes for blosc2-2.2.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b107ea9ed29e7a66908264fd6501371b592afb7dd381f51165cb9bedd932d435 |
|
MD5 | 9cc3bc616d49ae421b09f2f7c9897dfa |
|
BLAKE2b-256 | a1c6e5653c9fde4bb53d286dc2d523acb30a33ecd3435090090f7083276f194b |
Hashes for blosc2-2.2.6-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 690d6f8bd31270aad7cf019a6296bd8da773f0bf3c7b783e443f7177aef74d98 |
|
MD5 | 30b25f91a784ac3868b57ab2f77d57e4 |
|
BLAKE2b-256 | 4941a570926399a519e668b9df0704340b1b6598fc3547daee56922430e0cd03 |
Hashes for blosc2-2.2.6-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cebcdf8388dd3173edf0a59b452402ee1c1102fc21b03215eb926ca02e89ef87 |
|
MD5 | 167ba4d8590ee71464959c1e7074a0f9 |
|
BLAKE2b-256 | ac3e47bbde8b023be5ab5c50a18e71b987e3a8d98d70ad2915cac0073193af92 |
Hashes for blosc2-2.2.6-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b416be54f5770fdf812b463617134a4637ed3d995659b86970a91228150692d1 |
|
MD5 | 400042bf67881b7c94ba4329d3f204bb |
|
BLAKE2b-256 | 67c8ffb061c87697acba4413907e7f7f24026fa6fa50d37d1ff1189e5aae25d1 |
Hashes for blosc2-2.2.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f855dc9ada046170f96ddaf63d13ad67f5f97f93c78e4363dd15f6ca25b1f30 |
|
MD5 | a6b60539fa83c805fa69d064e13c09c9 |
|
BLAKE2b-256 | 235fc8df9f0d3873dd294c66ef4aff371c997bd26be4e31236e3f837358ee72f |
Hashes for blosc2-2.2.6-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e1b00222457516b2ba9a7d205e7be2680fd10caf5242a48fe701ba9beb851d0 |
|
MD5 | 2766f602d8bdcf06ef67240015044e56 |
|
BLAKE2b-256 | 275766a3b47e692365f99bc4cd11997f741115a3585ec46f4f96c3418485efa8 |
Hashes for blosc2-2.2.6-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b6d006b53872d25de4f2a84ddec6f0ad9df7ece77f130cd724e32c9913dfed28 |
|
MD5 | 0454c5a5846718ad02b0a1f0ce503285 |
|
BLAKE2b-256 | 9c4f1cd889475810d82033a620683478e5e3a091a2c4887bede125bafe42b1c0 |
Hashes for blosc2-2.2.6-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 391322885b9b27e20372b955dc99a443acc0a585857ea6ecf00909b8a54325e3 |
|
MD5 | afa4eac9d72175828fda1796736be904 |
|
BLAKE2b-256 | f1d4b79081b639c81472c6a3cbd551cd3d906c16fa625278205015aef43460ad |