Inspect HEAD/LIST/GET requests withing Rasterio
Project description
tilebench
Inspect HEAD/LIST/GET requests withing Rasterio
Source Code: https://github.com/developmentseed/tilebench
Inspect HEAD/LIST/GET requests withing Rasterio.
Note: In GDAL 3.2, logging capabilities for /vsicurl, /vsis3 and the like was added (ref: https://github.com/OSGeo/gdal/pull/2742).
Install
You can install tilebench
using pip
$ pip install -U pip
$ pip install -U tilebench
or install from source:
$ git clone https://github.com/developmentseed/tilebench.git
$ cd tilebench
$ pip install -U pip
$ pip install -e .
API
from tilebench import profile
import rasterio
[@profile](https://github.com/profile)()
def info(src_path: str):
with rasterio.open(src_path) as src_dst:
return src_dst.meta
meta = info("https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif")
> 2022-10-25T00:20:24.215385+0200 | TILEBENCH | {"LIST": {"count": 0}, "HEAD": {"count": 1}, "GET": {"count": 1, "bytes": 32768, "ranges": ["0-32767"]}, "Timing": 0.8705799579620361}
from tilebench import profile
from rio_tiler.io import Reader
[@profile](https://github.com/profile)()
def _read_tile(src_path: str, x: int, y: int, z: int, tilesize: int = 256):
with Reader(src_path) as cog:
return cog.tile(x, y, z, tilesize=tilesize)
img = _read_tile(
"https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif",
9114,
13216,
15,
)
> 2022-10-25T00:21:32.895752+0200 | TILEBENCH | {"LIST": {"count": 0}, "HEAD": {"count": 1}, "GET": {"count": 2, "bytes": 409600, "ranges": ["0-32767", "32768-409599"]}, "Timing": 1.2970409393310547}
Command Line Interface (CLI)
$ tilebench --help
Usage: tilebench [OPTIONS] COMMAND [ARGS]...
Command line interface for the tilebench Python package.
Options:
--help Show this message and exit.
Commands:
get-zooms Get Mercator Zoom levels.
profile Profile COGReader Mercator Tile read.
random Get random tile.
viz WEB UI to visualize VSI statistics for a web mercator tile request
Examples
$ tilebench get-zooms https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif | jq
{
"minzoom": 14,
"maxzoom": 19
}
$ tilebench random https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --zoom 15
15-9114-13215
$ tilebench profile https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --tile 15-9114-13215 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 2,
"bytes": 409600,
"ranges": [
"0-32767",
"32768-409599"
]
},
"Timing": 1.2364399433135986
}
$ tilebench profile https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --tile 15-9114-13215 --config GDAL_DISABLE_READDIR_ON_OPEN=FALSE | jq
{
"LIST": {
"count": 1
},
"HEAD": {
"count": 8
},
"GET": {
"count": 3,
"bytes": 409600,
"ranges": [
"0-32767",
"32768-409599"
]
},
"Timing": 2.2018940448760986
}
Starlette Middleware
Warning: This is highly experimental and should not be used in production (https://github.com/developmentseed/tilebench/issues/6)
In addition of the viz
CLI we added a starlette middleware to easily integrate VSI statistics in your web services.
from fastapi import FastAPI
from tilebench.middleware import VSIStatsMiddleware
app = FastAPI()
app.add_middleware(VSIStatsMiddleware)
The middleware will add a vsi-stats
entry in the response headers
in form of:
vsi-stats: list;count=1, head;count=1, get;count=2;size=196608, ranges; values=0-65535|65536-196607
Some paths may be excluded from being handeld by the middleware by the exclude_paths
argument:
app.add_middleware(VSIStatsMiddleware, exclude_paths=["/foo", "/bar"])
GDAL config options
- GDAL_DISABLE_READDIR_ON_OPEN: Allow or Disable listing of files in the directory (e.g external overview)
- GDAL_INGESTED_BYTES_AT_OPEN: Control how many bytes GDAL will ingest when opening a dataset (useful when a file has a big header)
- CPL_VSIL_CURL_ALLOWED_EXTENSIONS: Limit valid external files
- GDAL_CACHEMAX: Cache size
- GDAL_HTTP_MERGE_CONSECUTIVE_RANGES
- VSI_CACHE
- VSI_CACHE_SIZE
See the full list at https://gdal.org/user/configoptions.html
Internal tiles Vs Mercator grid
$ tilebench viz https://noaa-eri-pds.s3.amazonaws.com/2022_Hurricane_Ian/20221002a_RGB/20221002aC0795145w325100n.tif --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
Blue lines represent the mercator grid for a specific zoom level and the red lines represent the internal tiles bounds
We can then click on a mercator tile and see how much requests GDAL/RASTERIO does.
Docker
Ready to use docker image can be found on Github registry.
docker run \
--volume "$PWD":/data \
--platform linux/amd64 \
--rm -it -p 8080:8080 ghcr.io/developmentseed/tilebench:latest \
tilebench viz --host 0.0.0.0 https://noaa-eri-pds.s3.us-east-1.amazonaws.com/2020_Nashville_Tornado/20200307a_RGB/20200307aC0865700w360900n.tif
Contribution & Development
Issues and pull requests are more than welcome.
dev install
$ git clone https://github.com/developmentseed/tilebench.git
$ cd tilebench
$ pip install -e .[dev]
pre-commit
This repo is set to use pre-commit
to run isort, flake8, pydocstring, black ("uncompromising Python code formatter") and mypy when committing new code.
$ pre-commit install
$ git add .
$ git commit -m'my change'
isort....................................................................Passed
black....................................................................Passed
Flake8...................................................................Passed
Verifying PEP257 Compliance..............................................Passed
mypy.....................................................................Passed
$ git push origin
License
See LICENSE
Authors
See contributors for a listing of individual contributors.
Changes
See CHANGES.md.
Changelog
0.8.2 (2022-11-21)
- update hatch config
0.8.1 (2022-10-31)
- fix issue with min/max zoom when there is no overviews
- calculate windows from block_shapes
0.8.0 (2022-10-25)
- update rio-tiler/rasterio dependencies
- remove python 3.7 support
- add python 3.10 support
- add image endpoint to show the data footprint
- switch from mapbox to maplibre
0.7.0 (2022-06-14)
- add
cProfile
stats
0.6.1 (2022-04-19)
- Remove usage of
VSIStatsMiddleware
intilebench viz
0.6.0 (2022-04-19)
- switch to pyproject.toml
0.5.1 (2022-03-04)
- make sure we don't cache previous request when using
tilebench profile
without--tile
option
0.5.0 (2022-02-28)
- update rio-tiler requirement
- add
reader
option
0.4.1 (2022-02-14)
- update Fastapi requirement
- use WarpedVRT to get dataset bounds in epsg:4326
0.4.0 (2021-12-13)
- update rio-tiler's version requirement
- add more information about the raster in the Viz web page (author @drnextgis, https://github.com/developmentseed/tilebench/pull/14)
- fix bug for latest GDAL/rasterio version
- add default STAMEN basemap in viz and remove mapbox token/style options.
- update fastapi requirement
0.3.0 (2021-03-05)
- add
exclude_paths
options inVSIStatsMiddleware
to exclude some endpoints (author @drnextgis, https://github.com/developmentseed/tilebench/pull/10) - renamed
ressources
toresources
0.2.1 (2021-02-19)
- fix typo in UI
0.2.0 (2021-01-28)
- add warp-kernels in output in
profile
CLI - add rasterio/curl stdout in output
- add dataread time in Viz
0.1.1 (2021-01-27)
- update requirements
0.1.0 (2021-01-04)
- add web UI for VSI stats visualization
- add starlette middleware
0.0.2 (2020-12-15)
- Update for rio-tiler==2.0.0rc3
0.1.0 (2020-07-13)
- Initial release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tilebench-0.8.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c66b47d01f8b57b42b5bdc906247487f42a5cc780b484aa46610e953c1397737 |
|
MD5 | 8415d9e32feaddf368edb0280b7f14b9 |
|
BLAKE2b-256 | 9d3850f042be9accc753646bbbbb02e8490e3c13d6917a098fe461080317a1a7 |