0

What I need:

I need to speed up queries as much as possible to a series of large VRT datasets (total is around 1TB of data, although each VRT is much smaller than that).

I am open to ANY suggestions, including other programming languages, and/or other libraries, or else just improvements to my code below. Possibly what I need is to do some transformations on the VRTs themselves?

Context:

Previously posted this on GIS stack exchange but for some reason they didn't think it fit there, even though it is GIS specific.

Currently I'm using Python with gdal to query these VRTs, here is the key piece of code in question:

# Retrieve a sub-image as a matrix of numbers
# 'boundingBox' is a simple list, coordinates of the corners of the sub-image I would like

def queryVRT(boundingBox, pathToVRT):
        res = 250
        igh = "+proj=igh +lat_0=0 +lon_0=0 +datum=WGS84 +units=m +no_defs"
        kwargs = {'format': 'MEM', 'projWin': boundingBox,
                  'projWinSRS': igh, 'xRes': res, 'yRes': res}
        memory_ds = gdal.Translate('', pathToVRT, **kwargs)
        my_array = memory_ds.ReadAsArray()
        del memory_ds
        return my_array

The VRTs in question are actually publicly available here: https://www.isric.org/explore/soilgrids

But here is a small sample of one of the .vrt files, incase it helps (I'm not great at reading them):

<VRTDataset rasterXSize="159243" rasterYSize="58034">
  <SRS>PROJCS["Interrupted_Goode_Homolosine",GEOGCS["GCS_WGS_1984 ellipse",DATUM["D_WGS_1984",SPHEROID["WGS_1984",6378137,298.257223563]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Interrupted_Goode_Homolosine"],UNIT["Meter",1]]</SRS>
  <GeoTransform> -1.9949000000000000e+07,  2.5000000000000000e+02,  0.0000000000000000e+00,  8.3610000000000000e+06,  0.0000000000000000e+00, -2.5000000000000000e+02</GeoTransform>
  <Metadata>
    <MDI key="Code_version">v2.0.0</MDI>
    <MDI key="Covariates">depth,clm_wcl_s07rad,clm_wcl_srdyrsum,clm_wcl_s08rad,luc_lds_mng,mor_mrg_vdp,clm_wcl_s04rad,clm_wcl_bio16,luc_gfc_trely10,clm_wcl_bio07,mor_env_demm,mor_mrg_twi,clm_esa_s09avg,clm_wcl_srdyrstd,luc_esa_l170,clm_mod_lstd01std,clm_esa_s05avg,clm_wcl_p05tot,mor_mrg_vbf,clm_wcl_p04tot,clm_wcl_bio18,clm_wcl_p10tot,clm_wcl_p09tot,clm_mod_lstd03std,veg_mod_evirng,veg_mod_eviyravg,clm_wcl_bio19,clm_mod_lstd09std,clm_wcl_p12tot,clm_wcl_bio08,clm_wcl_bio17,clm_mod_lstd11std,clm_mod_lstd05std,clm_mod_lstd02std,clm_esa_s08avg,clm_wcl_bio15,clm_mod_lstd10std,veg_mod_nppy15,clm_mod_lstd04std,clm_mod_lstd12std</MDI>
    <MDI key="Depth_interval"></MDI>
    <MDI key="Litter_layers">FALSE</MDI>
    <MDI key="Mapped_units"></MDI>
    <MDI key="Model">Quantile Regression Forests</MDI>
    <MDI key="Model_type">ranger</MDI>
    <MDI key="Mtry">12</MDI>
    <MDI key="Number_trees">200</MDI>
    <MDI key="Other_synth_profiles"></MDI>
    <MDI key="Outputs_abstract">no WISE, March 2020 snapshot, NZ and Siberia gaps</MDI>
    <MDI key="Outputs_version">RUN03</MDI>
    <MDI key="Quantile"></MDI>
    <MDI key="Transformation"></MDI>
    <MDI key="WoSIS_version">Data stream 7</MDI>
  </Metadata>
  <VRTRasterBand dataType="Int16" band="1">
    <NoDataValue>-32768</NoDataValue>
    <ColorInterp>Gray</ColorInterp>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">./bdod_0-5cm_mean/tileSG-018-027/tileSG-018-027_1-1.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="600" RasterYSize="600" DataType="Int16" BlockXSize="600" BlockYSize="6" />
      <SrcRect xOff="0" yOff="0" xSize="600" ySize="600" />
      <DstRect xOff="48246" yOff="31441" xSize="600" ySize="600" />
      <NODATA>-32768</NODATA>
    </ComplexSource>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">./bdod_0-5cm_mean/tileSG-018-027/tileSG-018-027_3-3.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="600" RasterYSize="600" DataType="Int16" BlockXSize="600" BlockYSize="6" />
      <SrcRect xOff="0" yOff="0" xSize="600" ySize="600" />
      <DstRect xOff="49446" yOff="32641" xSize="600" ySize="600" />
      <NODATA>-32768</NODATA>
    </ComplexSource>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">./bdod_0-5cm_mean/tileSG-018-027/tileSG-018-027_3-1.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="600" RasterYSize="600" DataType="Int16" BlockXSize="600" BlockYSize="6" />
      <SrcRect xOff="0" yOff="0" xSize="600" ySize="600" />
      <DstRect xOff="48246" yOff="32641" xSize="600" ySize="600" />
      <NODATA>-32768</NODATA>
    </ComplexSource>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">./bdod_0-5cm_mean/tileSG-018-027/tileSG-018-027_3-2.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="600" RasterYSize="600" DataType="Int16" BlockXSize="600" BlockYSize="6" />
      <SrcRect xOff="0" yOff="0" xSize="600" ySize="600" />
      <DstRect xOff="48846" yOff="32641" xSize="600" ySize="600" />
      <NODATA>-32768</NODATA>
    </ComplexSource>
    <ComplexSource>
      <SourceFilename relativeToVRT="1">./bdod_0-5cm_mean/tileSG-018-027/tileSG-018-027_2-3.tif</SourceFilename>
      <SourceBand>1</SourceBand>
      <SourceProperties RasterXSize="600" RasterYSize="600" DataType="Int16" BlockXSize="600" BlockYSize="6" />
      <SrcRect xOff="0" yOff="0" xSize="600" ySize="600" />
      <DstRect xOff="49446" yOff="32041" xSize="600" ySize="600" />
      <NODATA>-32768</NODATA>
    </ComplexSource>

.... etc

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.