Python wrapper for the C-Blosc2 library
Project description
A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2
Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplace
That’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)
Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.py
License
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Enjoy data!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for blosc2-2.2.3-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eeb840983f1ed7bcb23526379e618ccc5a7f7c9a62ae58ef532ecd52d95e42ab |
|
MD5 | 24f2b214001511d71710dc5ae17108e7 |
|
BLAKE2b-256 | cfc286f307f490a153ad01edf96deba4f21264f36b3883e2ec853112869a7ab2 |
Hashes for blosc2-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a74531d2907664120561325267124ff89a374377223170b3cf29439c93c06de |
|
MD5 | f75a9f9765d4f9ea4a9ebb1416a9056b |
|
BLAKE2b-256 | b31cec35f535207f4a0d14b181b90d63506b9126ac2a3af99b986efb69f531ba |
Hashes for blosc2-2.2.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77ed06591577dcf6746064b94155886be6d871fb7009e6a9ca50832a84e1f9b0 |
|
MD5 | f5b55c45b8a4a23df5bf9b0484aed438 |
|
BLAKE2b-256 | cc25d110e1887473ed1c5403df06bc0c68d4a44eb12dd6053994d0e32c8f5e34 |
Hashes for blosc2-2.2.3-cp311-cp311-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7bd0a91ac72004e655c67e1a42f1a214fe447d77bdf9ab381c81b35294b5752 |
|
MD5 | 719861fec401db2ec470ea140b595e28 |
|
BLAKE2b-256 | e6e3edebb9dd03bf26fc900241b0b79185924dda858504cca2a2bb9850a410c2 |
Hashes for blosc2-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a1a798267e6e5c63b2f2c853789b1faf875fe07d8a0fbc50c1cf1e0150dace4 |
|
MD5 | 3c11920897d1c53c44f5e501d0581900 |
|
BLAKE2b-256 | 5bcde5ea9f4167e06e0e4116b9aa76a4717e2e68d453b18d18576ef4f636e587 |
Hashes for blosc2-2.2.3-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e63942ab4fd831e37755df55cd665b6acc9b04ba5388eb4922fe34488a1ff883 |
|
MD5 | e31c3a2211d39641fe3e3b5ba6313c9c |
|
BLAKE2b-256 | 0ad0d95357893f4cef8409ca040deafa069bc6af87a9cceffecfe8ff1808be3b |
Hashes for blosc2-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a418c1a72b9a71ccca9b71c44eebb491b3287b7ed38a82a24684b7f2d38a2c7 |
|
MD5 | 5a9db46fbd30580c763173827f6ed5da |
|
BLAKE2b-256 | 0cadb8ba993078fdc83119081b4290bda160a5b074048169f19785d4091a961a |
Hashes for blosc2-2.2.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c33066f1f70431e252c0e0f8c7743112c73d2442e78b63abcf68de779c60f9f9 |
|
MD5 | 3ec7f9b5b2ee8983e6dfbfc66aadcda6 |
|
BLAKE2b-256 | fcf5d2ee6bcfda5627ca82e2cc6db33127f73fe8fa8817ae5251ff3a9956cc64 |
Hashes for blosc2-2.2.3-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4828c6886cff6dc51e3cc3e8e21f24ccbc58ff03b481c534a51461e380207105 |
|
MD5 | efe4626ce58a9af2a1c29b6e46c3e1f5 |
|
BLAKE2b-256 | 9654f570d33ca4ec145ef13d84f16305767471b2a5995d0608fc8087cec4146d |
Hashes for blosc2-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b9bc5dfa19081b2f4ec856181af99fc9c438509e4102370ae0ec9756630fd5f |
|
MD5 | f036eea5c2f467afa92bb6ac7029a604 |
|
BLAKE2b-256 | f0c891b6a42200441e0c632d43b49a5be6e9e3b3715cb84bbf18c7a90d18bd21 |
Hashes for blosc2-2.2.3-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf97e16ba60ba0fcee2aff7314888ee7c03d451cafe3f799d322b0e6261dc8ba |
|
MD5 | 848459efe385b98f90b71dfa2a435df6 |
|
BLAKE2b-256 | c4642377162ea17dfe04a277215878e507e6ee158fc74aecb17eb20f0534135c |
Hashes for blosc2-2.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e24bddd44402ba91035db594aba0e03d53e65db418f959549ca6dbb5b48200f1 |
|
MD5 | b3db8273865de82605e4ca6bd7366610 |
|
BLAKE2b-256 | c5267572cb5b5fabb137c9b769a73a338892b1df8d0c2060542c56b4090405b9 |
Hashes for blosc2-2.2.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a394c6ff4d6e7b10c60a50e1ce94e2fcf8be9b8caa6124db5983bd107e94ae4 |
|
MD5 | 94b8c23dc0ab1647ce948f426e03dd48 |
|
BLAKE2b-256 | cb677f7231497f81db93d610c9c1da3060fdfe9dcedbccea601c5a1b27a0074c |
Hashes for blosc2-2.2.3-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f4902026d44826c0b725963098e1a3c29abfb679c776a0f2f875488316715f8a |
|
MD5 | e6ec321819ef1d451f3fc9ebb9db0476 |
|
BLAKE2b-256 | 39e666e40e0b7cb2bb66b3f09e72a10af3f2a9ce2b045063e4c0977e4f9d81b2 |
Hashes for blosc2-2.2.3-cp39-cp39-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6078dc80cac9aa59e3d8aad2b4ef88b8e798c4f5531fa11d613aaa7945920821 |
|
MD5 | 98d4963d7948400ac88ba377c3eb628b |
|
BLAKE2b-256 | 894bf7c8ca526fe10b54384007a612138e52e26b49d27ef53684baf3bcc05300 |
Hashes for blosc2-2.2.3-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ddacf096140e515032cb97a0512189183a57a564acc356a8299669bbb2905619 |
|
MD5 | 3b8793ee9b282492e43a67d94fc58d32 |
|
BLAKE2b-256 | 5c9e04ff2aa8f1e38f5fd7fc6b33eeafc6be2f435863b6f1af7f5383cb4b4e02 |
Hashes for blosc2-2.2.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b470af7e42e5fe1b122a4859086a7447c49808c36c355dc2ed4121b8be511f6d |
|
MD5 | 8631967482636bd26e40aae37face1a3 |
|
BLAKE2b-256 | c1390e95cff858b2c97540c7d0f159da6550e1503cca0dbe0178f5076d104d12 |
Hashes for blosc2-2.2.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30c32a36ce83a37d6e2c14c1eda4ea0382f4358bcacfc243a7fa39a54661ade8 |
|
MD5 | 26db2f67b3bc811960b7b93f1f140a4d |
|
BLAKE2b-256 | 0500f29ebb5f376f305e063f0357240043c80696860fd88317658133f5b6cf0e |
Hashes for blosc2-2.2.3-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5e5cde5b5f2a4bd92e0ff6d381e89b367e3cb61c702e01ab264536339592d6f7 |
|
MD5 | b18c0c712fa7070eb3810b483330d23e |
|
BLAKE2b-256 | 8f9817306c38676b4d8a6a789fced9a4765e24c8ef3e546a5a9c4a0fc8c07f74 |
Hashes for blosc2-2.2.3-cp38-cp38-macosx_10_9_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0020580e4899d7c7573acb92a06d895f783e9033ebc53991d458e7ac7515ed04 |
|
MD5 | be5d880113da41f57a8bddab833d241b |
|
BLAKE2b-256 | 77d2ded29ede9927e3371a5dfbc002a2a9fe6c604b47b654b11debd2e0acf490 |