Showing posts with label GDAL. Show all posts
Showing posts with label GDAL. Show all posts

Friday, October 19, 2018

raster_chunk_processing.py Examples

This is part 3 of a three-part post. Read part 1 (Introduction) here and part 2 (Installation and Usage) here.

Ok, enough dry words. Let's see the goods!

These are all clips of the Grand Canyon of the Yellowstone River/Dunraven pass area from the USGS 1/3rd arc-second DEMs. For the smoothed DEMs, I created a hillshade with the gdaldem hilllshade -combined command. The elevation shade is the North_America_State_Utah_2 ramp from the ESRI Terrain Tools download.

Original Hillshade

Mean Radius 10 Hillshade
The simplest smoothing is averaging the value of the nearby pixels. This gets rid of noise, but also gets rid of the ridges and valleys. The sharp brown line is an error from importing the Terrain Tool's ArcMap styles into ArcGIS Pro.

Gauss Radius 10 Hillshade
A Gaussian kernel does a better job at smoothing away the high-frequency noise (ie, small sudden changes in elevation) while preserving the broader changes. Note the smoothing in the sides of the canyon, and the dampening of the contour-like artifacts in the gulleys.

mdenoise Hillshade
This is an aggressive smoothing setting; a similar effect might be possible with a higher-radius Gaussian blur.

Skymodel Hillshade
Behold the bold shades of the skymodel (using the original, unsmoothed DEM). With such tweakability available, this is but one possible output from this DEM. Much experimentation is needed to see if it's worth using for your data. Note that this particular output can be approximated by a simple slope calculation, with a white-to-black, shallow-to-steep-slope color ramp

CLAHE
This is a CLAHE stretch on the unsmoothed DEM's hillshade with a radius of 30 and a clip of 0.01. CLAHE gives a slightly different look than what's possible with the contrast stretcher tools in ArcGIS. It also permanently alters the file, which can be useful for bump-mapping imagery with a hillshade.

TPI 10
TPI gives different looks depending on the radius you use. It's usually symbolized with a green-yellow-red color ramp, but I like to use it as a layer in a terrain stack (set partially transparent, it can help highlight some of the more exciting terrain areas) either as is or hillshaded. This is the raw output from a kernel radius of 10.

TPI 10 Hillshade
Here's the raw hillshade from the TPI 10 dataset. Definitely not something useful on it's own, what it's really showing is channels and ridglines (which is what the raw TPI is often used to identify).

TPI 10 Combined
And here's the TPI 10 hillshade at 75% transprency layered on top of a smoothed skymodel hillshade. Look at how well-defined the draws and ridge lines are; this could make for a great basemap for a stream map.

TPI 30 Hillshade
We can get a slightly different look just by increasing the kernel radius to 30. We lose the sense of height in Mount Washburn, but the ridge lines really stand out.

Default Hillshade
Here's the default hillshade (without the -combined option) just to remind us of where we came from. Quite the snoozer, if terrain is your thing. Also note how the massive amounts of grey midtones muddle the colors of the elevation ramp. This is particularly abhorrent when laying it over imagery.



raster_chunk_processing.py Installation and Usage

This is part 2 of a three-part post. Read part 1 (Introduction) here and part 3 (Example Output) here.

Installation

Git Repo
RCP lives in the https://github.com/jacobdadams/rcp git repo, along with some other playground scripts and the environment.yml file for easily recreating the conda environment.

Prerequisites
RCP is written in Python 3 and relies on the numpy, astropy, scikit-image, numba, and gdal libraries. The recommended way of installing these packages is from the conda-forge channel using the conda package management system.

After you've installed conda, use the environment.yml file in the git repo to automatically create the rcp conda environment:

(NOTE: The environment.yml file currently does not include the numba packages. It will be updated later)
conda env create -f environment.yml

Or, just install the needed packages thus:
conda install numpy numba astropy scikit-image gdal

The package solver will take a while to get this set up. This environment is configured to avoid problems with the default channel's GDAL not supporting bigtiffs, an issue with the solver not handling the conda-forge channel properly (see https://github.com/conda-forge/gdal-feedstock/issues/219), and changes to numpy that are throwing warnings in astropy (see https://github.com/astropy/astropy/issues/7816). Just using the normal gdal installation from conda-forge seems to work now.

See my earlier post on conda for a brief intro if you're new to this world.

It may be possible to run RCP in Python 2.7, but this has not been tested. Various small tweaks would probably be required to get it running.

The mdenoise option requires Sun et al's mdenoise executable from the author's webpage. Download the appropriate version, and note it's location—you'll need to set a variable to this path.

The skymodel option requires an illumination file, which can be generated by the skyLum.exe program written by Dr. James Stewart. This can be found in the skyModels.zip archive on the repo listed below. Thank you to Dr. Stewart for allowing me to redistribute this program. skyLum.exe is only compiled for Windows.

Installing

  1. Clone the git repo: https://github.com/jacobdadams/rcp
  2. Change the mdenoise_path variable to point to your downloaded mdenoise executable (it's towards the end of the script).
  3. Create the conda environment (if you haven't done so already): conda env create -f environment.yml
    1. Or, install the needed packages in an existing environment: conda install numpy numba astropy scikit-image gdal

Usage
python raster_chunk_processing.py -m [method] {general options} {method-specific options} input_file output_file

RCP is run by your python interpreter and takes two types of options: general and method-specific. General options control the chunk size and processing, and they are required for most methods (they usually won't choose defaults if you don't specify anything). The method-specific options control the parameters for each

For a brief overview of all the options, run python raster_chunk_processing.py -h.

General Options
-m The processing method (blur_mean, blur_gauss, blur_toews, mdenoise, hillshade, skymodel, clahe, TPI).
-s The size of the chunks that the raster will be divided into for processing. mdenoise seems to fail on chunks larger than 1500, but for other methods it depends on how much memory you have and (for small rasters) how many cores you want to use at a single time.
-o The number of pixels to be read beyond each edge of the calculated chunk to avoid edge artifacts. I find something like 20 or 25 is sufficient. Some methods overwrite this setting to ensure they have good data.
-p Number of concurrent processes to run. Values larger than 1 will allow parallel processing if you have the cores available. I usually set this to n-1, where n is the number of cores I have. This is the one option with a default (1).
--verbose Toggles verbosity on. By default, RCP reports when each chunk was started, it's index (ie, 1 of 220), and the percentage of chunks already started/finished (ie, 25%; this gets a little gray because when running parallel a previously-ordered chunk in a separate stream may not be completed before this message is printed). With verbosity enabled, it reports the value of all the options passed to it, and for each chunk it adds the pixel indices of the chunk (including overlaps) and the process ID. The output from the mdenoise executable will only be shown if the verbose flag is set.

Kernel Size Option
-r Radius (in pixels) to use for blur_mean, blur_toews, blur_gauss, and TPI. This is the main parameter to control the blurring effect. Play around with this on a subset of your data to get a feel for the best look.

blur_gauss Option
-d Standard deviation of the Gaussian distribution (sigma). Start with 1, and experiment from there. A large sigma for a given radius (say, greater than 1/2 the radius value) effectively turns it into a mean filter.

mdenoise Options
-n Iterations for normal updating. Start with something around 10 and play around from there. Large values of -n and -v will increase the runtime for each chunk.
-t Threshold value, between 0 and 1. Try starting with 0.6.
-v Iterations for vertex updating. Start with 20 and play around.

CLAHE Options
-c Clip limit, between 0 and 1. I usually use really small limits, like 0.01 or 0.05; 0.1 is usually as high as I'll go.
-k Kernel size for clahe. Start with 30.

Hillshading Options
-az Azimuth of the light source (defaults to the standard of 315 degrees).
-alt Altitude of the light source (defaults to the standard of 45 degrees).

Skymodel Options
-l Path to the luminance file generated by SkyLum.exe. The header lines must be deleted so that every line in the file is a az/elevation/weight tuple.

File Options
input_file The input DEM. This must currently be a single-band file readable and createable by GDAL.
output_file The output file that will be created. A full-size but blank file is created at the beginning of the program and is then opened and updated each time a chunk is finished. The source file is used to determine the file type, projection, extent, and cell size of the output file (unless it's a VRT, in which case it defaults to a GeoTiff). If a GeoTiff is used, the output file is compressed with LZW, tiled, and bigtiff-enabled (-co compress=lzw, -co tiled=yes, -co bigtiff=yes). If you are using another format, you can edit the program to support different options (search for lzw_opts in the ParallelRCP() function).

Usage Examples
Ok, that was a lot of explanation. Here are some examples in action:

Gaussian Blur
python raster_chunk_processing.py -m blur_gauss -s 2000 -o 50 -p 3 -r 10 -d 1 dem_state.tif dem_state_gauss10.tif
(Took about a minute to process a 10k pixel by 10k pixel DEM).

mdenoise
python raster_chunk_processing.py -m mdenoise -s 1500 -o 50 -p 3 -n 20 -t 0.6 -v 30 dem_state.tif dem_state_md206030.tif
(Took about 8 minutes on the 10k by 10k DEM).

skymodel
python raster_chunk_processing.py -m skymodel -s 2000 -o 600 -p 3 -l 1_35_215_150.csv dem_state.tif dem_state_skymodel.tif



Introducing raster_chunk_processing.py (aka, RCP)

This is part 1 of a three-part post. Read part 2 (Installation and Usage) here and part 3 (Example Output) here.

After over a year of off-and-on development, I've finally got my second-favorite project ready for the masses (ie, I finally added command-line argument parsing). I present to you the new, the shiny, raster_chunk_processing.py!

Edit, May 2019:
That feeling when you think you've got something working, only to discover that its not doing what you think it's doing. rcp's skymodeling algorithm has been updated to properly compute shadows. Updated examples will be forthcoming.


Ok, so I need to work on my naming skills.



What Makes RCP Special?
RCP applies any one of several raster processing methods to arbitrarily-large rasters by automatically breaking up the raster into smaller chunks. This a) avoids out-of-memory errors with large rasters, 32-bit programs, and/or 8 GB or 16 GB memory loadouts in normal desktop computers and b) facilitates parallel processing when feasible.

What makes RCP better than re-tiling a large raster and processing the chunks individually? Many of these algorithms modify or create a new value for each pixel based on the pixels around them. When you run these on individual tiles and then merge them, the pixels on the edges of the tiles don't have all the needed info and produce a noticeable line in the final image. You also spend time and storage space tiling and merging.

RCP avoids these downsides by dividing the raster into chunks of a user-defined size, adding a few pixels (again, a user-defined amount) to each side of the chunk to avoid the aforementioned edge problems, then reading only this chunk into memory and working on it, and finally writing it into it's proper location within the output file.

RCP relies on GDAL for all the file access heavy lifting, and I owe a huge shout-out to Frank Warmerdam, Even Rouault, and the rest of the GDAL team.

Processing Methods
Currently, RCP can do several really cool things. The first is a selection of DEM smoothers (blur_mean, blur_gauss, blur_toews, and mdenoise). Next, and really, really fun, is Kennelly and Stewart's skymodel hillshade (along with a bog-standard hillshade algorithm copied shamelessly from the GDAL source). It will also calculate Topographic Position Index (TPI) with a user defined window, which is a step up on the default gdaldem TPI which uses a fixed window size. Finally, it provides a really esoteric contrast stretcher called Contrast Limited Adaptive Histogram Equalization (CLAHE).

DEM Smoothing
Often, a DEM comes to us with some imperfections. Maybe that new half-foot Lidar is too detailed. Maybe you're tired of those contour-line-esque artifacts in the NED 1/3 arc-second DEMs. Maybe you don't want to show so much detail, but want to maintain the crisp mountain ridges. This is where smoothing helps: in preparation for hillshading or some other cartographic, non-analytical process.

blur_mean smoothes the DEM by taking the average of all the pixels within a user-dfined radius/diameter circle around each pixel. This is similar to the ArcGIS Focal Statistics (Mean) tool with a circular neighborhood.

blur_gauss is similar to blur_mean, but it uses a 2d Gaussian distribution kernel for it's average, providing more detailed results than a regular blur_mean. You could do this with a custom neighborhood in Focal Statistics, but RCP makes it trivial to change the neighborhood size.

blur_toews is a smoothing kernel provided by Mike T. at https://gis.stackexchange.com/a/10467. It is slightly different from blur_mean and blur_gauss and may give better results depending on the terrain.

mdenoise uses the mdenoise.exe program that implements Sun et al (2007)'s Mesh Denoise algorithm, again as inspired by Mike T. at https://gis.stackexchange.com/a/12856. The compiled .exe is 32 bit and can only handle small rasters before blowing up and running out of memory. By limiting the size of the chunk we're working on, RCP allows us to use this executable as-is (along with automating the tif->asc->tif conversion process) and to parallelize the process.

Hillshades
My first impetus for this script was Kennelly and Stewart's skymodel hillshade technique, as featured on the cover of the 2016 ESRI Map Book. More optimization work needs to be done, but my ultimate goal is to get this working much faster than ESRI's Terrain Tools script tool through parallelization (and maybe offloading to the GPU).

skymodel is the aforementioned implementation of the skymodel technique, using Dr. Stewart's SkyLum.exe to generate a list of different hillshade azimuths and altitudes that are then combined into the final image.

hillshade is the bog-standard hillshading algorithm lifted from the gdaldem code. This is mainly used to generate all the different hillshades for the skymodel function.

Other Utilities
There are two other utilities I've included as I've played around with different raster techniques.

TPI is the normal Topographic Position Index calculation of the pixel value minus the mean of the surrounding pixels, which can then be categorized and displayed as desired. Coincidentally, this is also a high-pass version of the mean filter (http://fourier.eng.hmc.edu/e101/lectures/Fourier_Analysis/node10.html, http://northstar-www.dartmouth.edu/doc/idl/html_6.2/Filtering_an_Imagehvr.html). Hillshading the results of TPI can be rather interesting, depending on the TPI kernel width.

CLAHE is some really cool image magic that increase the contrast of your image in neat ways (https://imagej.net/Enhance_Local_Contrast_(CLAHE), https://en.wikipedia.org/wiki/Adaptive_histogram_equalization#Contrast_Limited_AHE) Yeah, I don't quite understand all the implications myself... but it can look cool! I've used it to increase the contrast of hillshades that I've then multiplied into aerial imagery to "bumpmap" the imagery. Increasing the contrast allows me to darken the shadows without increasing the darkness of the grey midtones.

WARNING: CLAHE relies on knowing the historgam of the entire image, so running it with chunks can cause visibly different stretches for each chunk. If you make your chunks large enough that the histograms are roughly the same, this difference nearly disappears.

For example, if one chunk covers mainly valley floors (where there are few hills and thus few shadows) and another chunk covers mountains (with lots of whites and lots of darks), they're going to have different histograms and thus different stretches. However, if your chunks are big enough that they cover both valley and mountains, the histograms should be similar enough to get away with.

Check the next posts for how to use it and semi-cartographic examples!

PS: Any feedback on RCP is welcome! 



Monday, January 1, 2018

A Functional Introduction to GDAL, Part 2: GeoTiff Compression

So you just got a hard drive containing some gorgeous new 6" resolution aerial imagery for your hometown and you can't wait to play with it. Upon opening the imagery folder you're hit with a grim reality: high-res imagery file sizes are huge.

Compression techniques allow us to eliminate redundant data in a raster file in such a way that we can recreate the original file (lossless compression) or enough of the original to look good (lossy compression).

Because of it's ubiquitous support in other software and multitude of compression options, this post will focus on some of the compression options available in the GeoTiff driver. The different compression schemes can be specified with the creation option -co compress=<type>. The GeoTiff driver page has a list of all the available compression schemes.

Lossless and Lossy Compression

Compression methods fall into two general categories: "lossy" and "lossless."

"Lossy" compression schemes throw away chunks of data that they think aren't too important for the sake of smaller file sizes. jpeg compression relies on the way the human brain processes visual imagery to throw away varying amounts of "extraneous" data that (depending on the amount of compression specified) we don't usually notice. This results in files that are drastically smaller than if we had full three 8-bit values for each raster cell.

The reason it's called lossy compression is because you will lose data. Once you use a lossy compression scheme on a raster, you can not recover the original data. You can get something that looks very visually similar, but the underlying values are irrevocably changed.

In contrast, "lossless" compression schemes use patterns and other mathematical magic to compress the data in a way that it can be perfectly reproduced when it's decompressed. The compression ratios for lossless compression schemes usually aren't quite as good as lossy compression schemes, but you keep all of your original data.

As a rule, never use lossy compression on data where the cell value is important, like elevation data or reflectance values in multispectral imaging. It does work great for aerial imagery that you're just trying to use as a basemap, however.

Lossy Compression with JPEG

My preferred aerial imagery compression scheme is jpeg with YcBcR color representation (-co compress=jpeg -co photometric=ycbcr). If you're big into color theory you can probably explain better than I can what YcBcR is and why exactly it helps, but long story short it can sometimes give you better jpeg compression ratios (read: smaller files) for aerial imagery.

You can also play around with the jpeg quality creation option (higher quality = less data loss but larger file size), I've found the default of 75 to work pretty good for the projects I've done without major (or even noticeable) visual degradation. This gives me files that are roughly 5% to 10% the size of what they'd be without any compression. I've seen smaller ECW files from vendors, but this is close enough for what I need.

One gotcha I've come across with jpeg compression is artifacts, both inside the image data area and in the NoData areas along some borders. I'll write more on this later, with visual examples, but here are two guidelines to reduce/eliminate jpeg compression artifacts:
  1. To eliminate artifacts on the outside edges, make sure the original image data area is perfectly rectangular. An image of merged tiles that shift down a couple pixels every tile, creating a stair-step border of white NoData areas when merged together, will show some artifacts in the white NoData areas.
  2. To eliminate artifacts inside the data areas on the right and bottom of your image, make sure the tile/stripe size is a multiple of 8 or 16 (see the note about RGB jpeg compression on the GeoTiff driver page linked above): -co blockxsize=256 -co blockysize=256. If that doesn't work, make sure your total image height and width are multiples of your block size (see this mailing list thread). 

Lossless Compression with Deflate and LZW

When I'm working with large elevation datasets and want compression that preserves the original data, I'll usually use deflate compression (-co compress=deflate). This generally seems to give me higher compression ratios than LZW compression (the other main lossless compression scheme)—I've seen it reduce file size by around 50%, but your mileage may vary based on specific datasets.

You can use the -co predictor and -co num_threads options when working with LZW or deflate compression to speed up the compression. If you're just working with aerial imagery, -co predictor=3 doesn't make sense because your data isn't in floating point format (i.e., it's all whole numbers). These can be useful for very large datasets, but the smallish example dataset below only took a few seconds to compress no matter the method.

Compression Comparison

I used the San Francisco Bay image available from Earthstar Geographics to do a basic comparison of the compression schemes we've discussed. The image is 7204 x 7204 pixels in three bands of type byte, originally in the GeoTiff format. Note that the compression percentages will vary between every image (the relatively poor compression for LZW surprised me here) but the general ranking should stay the same.


Compression Scheme File Size % of Original File Size Lossy/Lossless
Uncompressed
155.9 MB
n/a
n/a
Deflate (z=6)
112.7 MB
72.3%
Lossless
LZW
142.9 MB
91.7%
Lossless
jpeg (75% Quality)
25.1 MB
16.1%
Lossy
jpeg (YcBcR, 75% Quality)
9.1 MB
4.8%
Lossy
jpeg (YcBcR, 90% Quality)
14.9 MB
9.49%
Lossy
ECW (2:1, from vendor)
28.3 MB
18.2%
Lossy

Note: The file sizes here were calculated on a Mac, which defines a MB as 1000^2 bytes (decimal-based), while Windows currently defines a MB as 1024^2 bytes (binary-based). If you download the example file yourself, you'll see the metadata file says its 148 MB, which would be the Windows version of the size. Confusing? Yeah. 

The jpeg compression is the clear winner here, and zoomed in to actual size (one pixel in the image is one pixel on my monitor) I couldn't tell any difference even between the 75% quality jpeg and the uncompressed GeoTiff.

Compression and BigTIFF

The default TIFF format only allows for files 4 GB or smaller. GDAL overcomes this barrier with the BigTIFF creation option that allows for really big files. The default option is if_needed, which tries to figure out if the new file will be greater than 4 GB and sets the flag if needed. Because this is the default, you have probably already created uncompressed GeoTiffs larger than 4 GB without even realizing this option exists.

However, compression throws a wrench into this default. The driver doesn't know how well the compression will work and fails to set the BigTIFF option, which often leads to write errors when creating large datasets. Whenever I'm working with large compressed datasets I use -co bigtiff=yes if there's any possibility the output will be greater than 4 GB.

ECW and MrSid

I've mentioned "ECW" and "MrSid" a couple times in this series. These are different formats (not compression options for GeoTiffs) that use mathematical magic known as wavelets to achieve lossy compression ratios greater than what's possible with jpeg, and with better image quality and data access to boot. Several Jpeg2000 compression schemes also use wavelet-based compression in a similar matter. 

The potential downside is that these are not open formats—they are protected by patents and require special (and possibly expensive) licenses. There's usually a free license for reading the data (which is what allows GDAL to read them), but creating them (or in the case of ECW, distributing images from a server as part of a webmap) require ponying up the license fee. You're the only one who can decide if that route is best for you.

Next up

Our next post will investigate gdal_translate, one of the most common and most useful GDAL tools.

Notes
Paul Ramsey's excellently concise compression discussion: http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html
Kersten Clauss' discussion (and test script!) on GDAL's lossless compression options: http://www.digital-geography.com/geotiff-compression-comparison/
A deep discussion on jpeg compression by Even Rouault, one of the GDAL developers: http://erouault.blogspot.com/2014/04/advanced-jpeg-in-tiff-uses-in-gdal.html (Seriously, this guy is awesome. Super knowledgeable and very active on the gdal-dev mailing list).

A Functional Introduction to GDAL, Part 1: Creation Options

Many GDAL tools create a new file that holds the results of its work. Most of these tools support the -of, -ot, and -co switches to set the output format, the output data type, and any format-specific options.

Setting the Raster Format: -of <name>

There are dozens upon dozens of different raster formats in the GIS world, and GDAL can handle many of them. The ability to read or write a format is provided by what GDAL calls "drivers," and there's a specific driver for each format.

While GDAL can read just about anything you throw at it, it can only write a handful of formats. The list of supported formats can be found in the GDAL documentation. The "Create" column will tell you if write support is present for your desired format. "Copy" is slightly different from create and depends on the raster data already being present. If you're opening and modifying an existing raster, you can use a format that has Copy capabilities; if you're creating a new raster, you need to use a Create format. See the CreateCopy() documentation for more info.

Many formats require additional work—for example, the ECW driver requires the ECW SDK from Hexagon Geospatial that has two different types of licensing. The read-only license is freely available for desktop use (which does not allow you to use it with server software distributing imagery), while the server read-only and desktop read-write requires a paid license. Getting this working is beyond the scope of this post, though I think I've seen the stand-alone QGIS installation come with read support enabled.

The name in the "Code" column is the "short format name" mentioned in the help documentation and is the name you'll use with the -of switch. For example, if I want a GeoTiff, I'd type -of GTiff.

Note: As of GDAL 2.3, GDAL tries to guess the output format based on the file extension of the output filename you provide—to get a GeoTiff, you'd just have to have your filename be .tif. In previous versions, it defaults to GeoTiff if no format is specified with -of.

One format that you'll notice doesn't have creation support is jpeg, which seems odd at first blush. However, if you want highly-compressed rasters, the jpeg compression method for GeoTiffs (see the next post on compression options) provides comparable results, just with a .tif file.

Specifying an Output Data Type: -ot <type>

Different datasets will have different basic data types, which define how the data for each raster cell are stored. A DEM with elevations from 1272.3 to 3311.9 may be in Float32, while aerial imagery may have three Byte bands—red, green, and blue, with values from 0 to 255.

Each data type has specific minimum and maximum values it can represent. In addition, Byte and the various Ints (or integers) can only represent whole numbers, while the Float types allow you to represent decimal numbers, like an elevation of 1564.357 (remember, this value by itself is unitless—you need to check your metadata to see whether it's an elevation in feet, or just a brightness value from 0 to 255).

The following table shows the minimum and maximum values for each type (assuming they match the equivalent C++ types). This is taken from the GDAL API reference and the Microsoft C++ reference (the various int min/max values should stay consistent with other compilers and platforms, but there may be slight variations in the Float32 and Float64 values).


Type Minimum Maximum Precision
Byte
0
255
Whole Numbers
Uint16
0
65,535
Whole Numbers
Int16
-32,768
32,767
Whole Numbers
Uint32
0
4,294,967,295
Whole Numbers
Int32
-2,147,483,648
2,147,483,647
Whole Numbers
Float32
3.4x10^-38
3.4x10^38
6 Decimal Places
Float64
1.7x10^-308
1.7x10^308
15 Decimal Places

The -ot switch allows you to specify the type of the output file. There are a couple nuances here, however, that we have to keep in mind:
  • First, specifying a data type does not automatically scale the data to fit into that data type. Rather, it just tries to copy the value straight across, clipping out data that doesn't fit. Going from Byte to Float32 is just fine: 232 becomes 232.0. However, going the other way presents problems: 67.395 becomes just 67, and 5692.845 gets clipped down to 255. Use gdal_translate with the -scale switch to appropriately scale your data into the new data type's ranges.
  • Different types require different amounts of storage for each value. A Byte, as we'd expect, takes up one byte per cell, while a Float32 takes up four bytes (32 bits, 8 bits to a byte). This means a Float32 raster will be about four times as large as a Byte raster of the same dimensions, limiting your storage options and increasing your file read times. This also means processing the same area would require four times the allocated RAM, which could lead to memory size issues on larger data sets and 32-bit environments.
  • Some programs can't handle certain data types. For example, I wanted to make a custom hillshade represented by brightness values of 0-255 with a NoData value, so I specified -ot uint16 and a NoData value of  256. However, when I tried to load it into GIMP to play around with it as a raster image, it failed because GIMP can't handle unit16 data (Note: The default gdaldem hillshade command creates a byte-type raster, using 0 as NoData and 1-255 as the brightness values. My insistence on using the range 0-255 for brightness values was foolishness.).

Creation Options: -co <option>

Creation options allow you to set certain parameters that are specific to each raster format. For example, the GeoTiff driver gives you the ability to compress your image (-co compress=<type>) or to create an accompanying .tfw world file (-co tfw=yes). Multiple -co switches can be used in one command, like this set that creates a GeoTiff with jpeg compression, a YcBcR color scheme, and the ability to create a file bigger than 4GB: -co compress=jpeg -co photometric=ycbcr -co bigtiff=yes.

The creation options available to each driver are listed on the driver's documentation page, which are linked from the format list above. For example, the GeoTiff creation options can be found about halfway through the GeoTiff format documentation.

The documentation usually shows all the creation options in upper case. This isn't absolutely necessary, however. I usually just use lower case for all the options (like you see in these examples) because it's easier to type.

Sidetrack: Configuration Options

There's a bit of a grey area between the definitions of creation options and configuration options (https://trac.osgeo.org/gdal/wiki/ConfigOptions). Generally speaking, creation options change the way data is stored in the file you create while configuration options affect how GDAL functions to create that file. You can specify configuration options as part of a command with the --config switch, like this --config gdal_cachemax 2048. Note that creation options don't have '=' between the option and its value.

Note: The distinction between creation and configuration options gets even grayer in gdaladdo, where you set the compression options for your overviews via the --config switch.

Putting It All Together
Let's say we want to change an Erdas Imagine (.img) file to a jpeg-compressed GeoTiff:
gdal_translate -of GTiff -co compress=jpeg -co photometric=ycbcr -co tiled=yes -co bigtiff=yes in.img out.tif

Coming Up Next

Next post we'll look at the compression options available with the GeoTiff driver.

Notes
More on ECW: http://www.gdal.org/frmt_ecw.html, https://trac.osgeo.org/gdal/wiki/ECW, and https://gis.stackexchange.com/questions/154929/how-to-install-gdal-with-ecw-support-on-windows?rq=1

Tuesday, December 26, 2017

A Functional Introduction to GDAL, Part 0: Installing GDAL

Welcome to GDAL!

This is the first of what should be several posts on GDAL, the bread-and-butter library and toolset for playing with raster data. There are several introductions to GDAL on the web already that eloquently explain what GDAL is and why it is useful—for example, Robert Simmon's 4-part series on Medium.

This series will be geared towards working GIS professionals: people who understand projections, have used the raster tools in ArcMap or QGIS, but may be a little hesitant to dip into the command line. It will be geared around the tools available and how they're used for common tasks—an elaboration on the official docs with an emphasis on practical examples.

Installing GDAL On Windows

Installing GDAL used to be either really simple or really annoying, depending on your computing environment. GDAL is written and actively maintained by developers: people who are comfortable compiling from source and managing environment variables and installations. As such, it works great in their world but requires a little massaging for everyday GISers, especially those of us coming from a Windows-centric ArcMap world.

Then came Conda.

Conda is amazing. Conda is package and environment management for python and python-related packages. Conda makes installing and running GDAL easy.

So, how do you get it?

First, download and install Anaconda. The Anaconda Platform is the support structure for the Conda command line tools and includes helpful utilities and installers.

Next, we'll use the Anaconda Navigator to create a new environment. To do this, go to the Environments pane and select "Create" at the bottom. Give it a name—I call mine "gdal" for clarity's sake—and select the latest 3.x version of Python (unless you've got a good reason to use 2.7). Click "Create" to finish it off.

Note: It doesn't matter what python is currently installed on your computer (like if you've got python 2.7 from ArcGIS Desktop) because Conda creates its own sandbox for each environment. These sandboxes are completely separate from each other and from any other Python installation on your computer—that's the beauty of Conda!

Next, we're going to add the conda-forge repository ("repo"), as it usually has a more up-to-date version of GDAL. To do this, click on the "Channels" button and then select "Add." Type "conda-forge" into the text box and hit Enter.

Now, you can search for and add the gdal packages to your new environment. With your environment selected, change the dropdown menu at the top of the packages pane to "All" and type "gdal" (w/o quotes) into the search box. Select the checkbox next to the entry for gdal when it appears and click "Apply" in the bottom right corner. It will work out what other packages are needed and install everything.

Another Note: Getting the latest version of GDAL gets you the latest updates and bug fixes. For example, GDAL 2.1.3 has a bug that does not allow you to set the altitude parameter when calling DEMProcessing() via the Python bindings (see this bug report). This bug was fixed in 2.2.0.

More Notes! As of this writing (end of 2017), I've noticed that installing the SciPy libraries with the Anaconda Navigator downgrades GDAL to 2.1.3. After you install SciPy, you should see a little green arrow in the checkbox for GDAL in the installed packages view, indicating an update is available. Go ahead and update GDAL back to 2.1.3 by clicking this green arrow and then selecting "Apply" at the bottom.   

Switching Environment
To play in your new environment, navigate to the Anaconda folder group in the Start Menu and select Anaconda Prompt. This opens a new command prompt that is aware of our Conda environments.

Like I mentioned, Conda gives us a separate sandbox for each environment we create. We need to switch to our new environment by typing activate gdal (replacing gdal with whatever environment name you chose earlier). After a pause, the command prompt will change to something like (gdal) C:\Users\jadams. This means that you're now in the gdal sandbox.

To verify GDAL is installed and working properly, type gdalinfo --version and hit Enter. You should get output that looks like GDAL 2.2.2, released 2017/09/15. 

If you create other environments, you can switch between them by activating the other environment with activate foo, just like you'd do to activate your first environment.

Conda on Other OSes

macOS
Installation on macOS is similar to Windows, and Anaconda Navigator works the same in both OSes. The only difference is that we'll use the standard Terminal app instead of the Anaconda Prompt. In addition, we have to type source activate gdal instead of just activate gdal. Beyond that, enjoy the flexibility of bash (or csh, or whatever shell you're using).

Linux
Conda on Linux requires more CLI-based magic than Windows or MacOS. I've not tried it myself, but a quick search for "install conda ubuntu" gives us these instructions, and it should work just the same for other distros as well.

Linux also shares the same source activate requirement as macOS to switch between environments.

Coming Up Next

The next post will focus on some common file creation considerations that you'll use whenever your GDAL commands result in a new file.

Resources

GDAL Homepage: http://www.gdal.org
Conda User Guide: https://conda.io/docs/user-guide/index.html
Anaconda User Guide: https://docs.anaconda.com/anaconda/user-guide/
GISInterals pre-built GDAL binaries for Windows: http://www.gisinternals.com/release.php
Alternate Windows Installation: https://sandbox.idre.ucla.edu/sandbox/tutorials/installing-gdal-for-windows