While th
Monday, February 1, 2010
Two Semantic Web titles that I found very informative
Over the past few months I've been working on a side project involving Semantic Web technologies (more on that in a later post). In addition to the many on-line references, I have found two texts to be very helpful in climbing the learning curve. As a result, I wanted to pass along some information on them in case anyone else was looking for a good place to get started with the Semantic Web.
Semantic Web Programming by John Hebeler, Matthew Fisher, Ryan Blace, Andrew Perez-Lopez and, Mike Dean (Foreword) provides a thorough introduction to the most common Semantic Web standards, libraries and tools. The authors approach the Semantic Web from a developer's standpoint focusing on how to implement semantic functionality. A working example is built upon throughout the book. The example demonstrates many of the concepts and techniques discussed. The example application shows how data in multiple different formats and types of source systems can be linked together. In addition to discussing tools and libraries, a introduction to some of the most important semantic web standards is provided. These include the Resource Description Framework (RDF), RDF Schema (RDFS), and the Web Ontology Language (OWL). Despite the very ambitious amount of material that they attempt to cover, in my opinion the authors do a very good job in 600 pages of providing a good place to jump into Semantic Web programming.
While th
e Semantic Web Programming text introduces the major semantic web standards, it doesn't delve too deeply into the creation of ontologies. This is where Semantic Web for the Working Ontologist - Effective Modeling in RDFS and OWL by Dean Allemang and James Hendler is really strong. The authors provide a comprehensive discussion of modeling in RDF, RDFS and OWL. Sections include practical advice on how to handle challenging modeling situations. Following the coverage of the standards two real world ontologies are discussed as examples of how modeling decisions impact the ontologies. Finally, good and bad modeling practices are presented. While the text covers the first version of OWL and OWL 2 became a W3C recommendation last fall, OWL 2 primarily adds to the language. As a result, this text remain very relevant.
While th
Wednesday, January 20, 2010
GeoServer Doesn't Start with a Corrupted Coordinate System DB
This week I ran into a problem in our CI (Continuous Integration) environment that really had me stumped for a while. I wanted to post the outcome in case anyone else sees similar behavior and is in search of the solution.
We''ve been using the open source map server GeoServer for a while on at least two different projects. Let me say first, that I think the GeoServer team is doing a wonderful job and has produced a very function, flexible, and stable solution. Earlier this week I had checked in configuration changes that added additional data to our default deployment. The next time that our CI environment rebuilt and redeployed GeoServer it failed to start with an out of memory error. I thought, no problem I'll adjust the JVM initialization parameters and restart. At this point, I received the following exception during startup and the application failed to deploy.
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedObject"
An EPSG is a common way of representing the coordinate system that spatial data is associated with. Since new data was recently added, my initial thought was that I madea mistake configuring the new data. The wierd thing was that the EPSG code referenced, 4326, corresponds to one of the most commonly used coordinate systems used with spatial data. If it wasn't available, somethign larger was going on. After sifting through a number of log files, news group posts, etc it appeared that the most likely culprit was an HSQLDB database that is written to disk by GeoServer to aid in coordinate system look-ups. If the database exists, but is corrupted GeoServer will not be able to read it and will indicate that it can't determine referenced coordinate systems. The solution is easy to delete these files and let them be recreated on the next restart. Locating the files was a little challenging. Below are the locations where I have seen the files being written. Delete all of the files in the directory, restart, and the problem should go away.
UNIX/Linux:
/tmp/Geotools/Databases/HSQL
Windows:
%SystemDrive%\Documents and Settings\\Local Settings\tmp\Geotools\Databases\HSQL\
or
%SystemRoot%\tmp\Geotools\Databases\HSQL
We''ve been using the open source map server GeoServer for a while on at least two different projects. Let me say first, that I think the GeoServer team is doing a wonderful job and has produced a very function, flexible, and stable solution. Earlier this week I had checked in configuration changes that added additional data to our default deployment. The next time that our CI environment rebuilt and redeployed GeoServer it failed to start with an out of memory error. I thought, no problem I'll adjust the JVM initialization parameters and restart. At this point, I received the following exception during startup and the application failed to deploy.
org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:4326" from authority "European Petroleum Survey Group" found for object of type "IdentifiedObject"
An EPSG is a common way of representing the coordinate system that spatial data is associated with. Since new data was recently added, my initial thought was that I madea mistake configuring the new data. The wierd thing was that the EPSG code referenced, 4326, corresponds to one of the most commonly used coordinate systems used with spatial data. If it wasn't available, somethign larger was going on. After sifting through a number of log files, news group posts, etc it appeared that the most likely culprit was an HSQLDB database that is written to disk by GeoServer to aid in coordinate system look-ups. If the database exists, but is corrupted GeoServer will not be able to read it and will indicate that it can't determine referenced coordinate systems. The solution is easy to delete these files and let them be recreated on the next restart. Locating the files was a little challenging. Below are the locations where I have seen the files being written. Delete all of the files in the directory, restart, and the problem should go away.
UNIX/Linux:
/tmp/Geotools/Databases/HSQL
Windows:
%SystemDrive%\Documents and Settings\
or
%SystemRoot%\tmp\Geotools\Databases\HSQL
Sunday, August 2, 2009
Scalling dynamic images for display in browsers
Recently I found myself looking into improving the image quality of dynamically created images in a web project that I've been working on. Some of the factors involved in the issues, and the eventual way they were addressed, are specific to the project. However, I wanted to share the way we addressed them in case others found themselves with a similar situation.
One of the pieces of functionality that the project incorporates is the ability for users to create customized displays and save them for later viewing. The displays are made up of background images, multiple foreground graphics, as well as, HTML elements. One of the requirements of the project is that the displays must be saved off as static images and be previewed at different sizes, a thumbnail and a display size.
The problem that we encountered was that we had no control over the screen resolution that the users would save their display using. During the save process we would consolidate all of the elements into a single image using the dimensions from the user's browser. This was required to make sure that all of the elements lined up properly. After the single image was created it was saved and the thumbnail image was created using the Java2D API.
Users of the application primarily use IE6 & 7. We started noticing that when the displays were captured at the full-size preview resolution everything looked good, but if a higher resolution was used for creating the displays the images when shown at the full-size preview dimensions would look distorted. In addition, we noticed that regardless of the screen resolution used for saving the displays, the thumbnail images appeared jagged.
The first problem we were able to address was the thumbnail image quality. After reading Chris Campbell's excellent Java.net article "The Perils of Image.getScaledInstance()" we realized that we were using the default interpolation method of Nearest Neighbor which while fast was producing images of quality less than what we desired. Changing the interpolation method to Bilinear increased the image quality and solved the thumbnail image issue.
An interesting thing we noticed was that in Firefox the saved images appeared a lot clearer than when rendered in IE. As it turns out after a few Google searches apparently Firefox (and the majority of other browsers) use bicubic interpolation for image resizing by default, but IE defaults to nearest-neighbor. Bicubic interpolation produces better results than nearest neighbor, but takes longer to render.
Microsoft added a vendor specific CSS property "-ms-interpolation-mode" that can be used to specify which interpolation method is used as described in this MSDN Library entry. The CSS example below sets the interpolation method to bicubic for smoother image scaling.
Unfortunately, the ms-interpolation-mode property was introduced in IE 7 and our project is still required to support IE 6. As a result, in addition to adding the CSS property we now create a scaled image on the server using bicubic interpolation in the dimensions appropriate for the preview. If the user is viewing the application with IE7, we let IE handle the interpolation but when IE6 is used the pre-scaled image is sent to the browser.
In the end we were able to achieve better image quality for the displayed images through the combination of specifying rendering hints to the Java2D API and the IE specific CSS property. When support for IE 6 is no longer a requirement, we will be able to remove the generation of the pre-scaled display image on the server which will increase performance.
One of the pieces of functionality that the project incorporates is the ability for users to create customized displays and save them for later viewing. The displays are made up of background images, multiple foreground graphics, as well as, HTML elements. One of the requirements of the project is that the displays must be saved off as static images and be previewed at different sizes, a thumbnail and a display size.
The problem that we encountered was that we had no control over the screen resolution that the users would save their display using. During the save process we would consolidate all of the elements into a single image using the dimensions from the user's browser. This was required to make sure that all of the elements lined up properly. After the single image was created it was saved and the thumbnail image was created using the Java2D API.
Users of the application primarily use IE6 & 7. We started noticing that when the displays were captured at the full-size preview resolution everything looked good, but if a higher resolution was used for creating the displays the images when shown at the full-size preview dimensions would look distorted. In addition, we noticed that regardless of the screen resolution used for saving the displays, the thumbnail images appeared jagged.
The first problem we were able to address was the thumbnail image quality. After reading Chris Campbell's excellent Java.net article "The Perils of Image.getScaledInstance()" we realized that we were using the default interpolation method of Nearest Neighbor which while fast was producing images of quality less than what we desired. Changing the interpolation method to Bilinear increased the image quality and solved the thumbnail image issue.
An interesting thing we noticed was that in Firefox the saved images appeared a lot clearer than when rendered in IE. As it turns out after a few Google searches apparently Firefox (and the majority of other browsers) use bicubic interpolation for image resizing by default, but IE defaults to nearest-neighbor. Bicubic interpolation produces better results than nearest neighbor, but takes longer to render.
Microsoft added a vendor specific CSS property "-ms-interpolation-mode" that can be used to specify which interpolation method is used as described in this MSDN Library entry. The CSS example below sets the interpolation method to bicubic for smoother image scaling.
img {
-ms-interpolation-mode:bicubic;
}Unfortunately, the ms-interpolation-mode property was introduced in IE 7 and our project is still required to support IE 6. As a result, in addition to adding the CSS property we now create a scaled image on the server using bicubic interpolation in the dimensions appropriate for the preview. If the user is viewing the application with IE7, we let IE handle the interpolation but when IE6 is used the pre-scaled image is sent to the browser.
In the end we were able to achieve better image quality for the displayed images through the combination of specifying rendering hints to the Java2D API and the IE specific CSS property. When support for IE 6 is no longer a requirement, we will be able to remove the generation of the pre-scaled display image on the server which will increase performance.
Saturday, August 1, 2009
Basic geographic image manipulation with GDAL
Recently I was supporting a demonstration of software that has as one of its capabilities the ability to display its data on a map. To make the demonstration scenario as realistic as possible, we wanted to incorporate aerial imagery into the map display. A requirement of the demonstration was that it must be self-contained on a laptop and not rely on a connection to the Internet (i.e., map services such as Google Maps, ArcWeb Services etc. were not an option).
The approach used to meet this requirement was to use publicly available sample imagery loaded into a locally running map server. In order to use the imagery however, we wanted to perform some basic image manipulation to make it easy to deploy. While there are plenty of software packages that can preform the simple manipulation that I was doing , I used the excellent open source Geospatial Data Abstraction Library (GDAL) package originally developed by Frank Warmerdam and now released by the Open Source Geospatial Foundation (OSGeo). Since I had received a few questions from my co-workers as to how the manipulation was done, I thought I would post a very basic example of some of the manipulation that can be done with GDAL.
The steps below describe how sample imagery can be obtained from a public source, merged into a single image, reprojected into a different coordinate system, and image prynids added to the image. Note, there are a lot of options available using these command line utilities of which these steps make use of a very minimal set. This was done for ease of understanding. Full documentation on the command line options can be found at the website or in the man pages.
1. Download example imagery files
The United States Geological Survey's National Map Seamless Server facilitates downloading of spatial data resources accessible to the public. The controls on the left side of the map can be used to zoom to a specific area and request data from that area to be downloaded. The "Display" and "Download" tabs to the right of the map control the data sets that are displayed on the map and are requested for download respectively.
The following steps will use 4 GeoTIFF images of the historic section of Philadelphia from the 2004 Philadelphia orthoimagery data set. The specific images used in this example are not important, however if you would like to use the same ones they can be downloaded using the following steps.
1.1 Click the XY tool in the upper left panel and enter the following coordinates: -75.147,39.945
1.2 Using In Magnifier in the Scale information display at the top right hand corner of the page, zoo in until the scale is about 1:30,000.
1.3 On the "Display" tab on the right, expand the Orthoimagery node and select "Philadelphia (May 2004).
1.4 Switch to the Download tab and make the same selection as in the previous step. Ensure that there are no other selections in any of the nodes in the tab
1.5 Using the "Define rectangular Download Tool", the first in the Download section of the left hand toolbar, select an area that is about the middle third of the screen as shown below.
1.6 A Download Summary page will be displayed showing the files available to download. Caution, these files are about 90 MB each. If more than four files are shown, adjust the selection area for download.

1.7 Download and extract the ZIP files into a single directory.
2. Install GDAL
The GDAL library can be downloaded from the OSGeo GDAL download site, however the tools are also included in repositories for many Linux distributions. Below are the commands that will install GDAL and the Python utilities in Ubuntu.
sudo apt-get install gdal-bin
sudo apt-get install python-gdal
While we're installing software, if you do not already have a GIS desktop application you might want to consider installing one of the following excellent open source applications: Quantum GIS, or uDig.
3. Viewing image meta-data
The files downloaded from the USGS server received numeric names. Do not worry if the numbers are not the same as those used in these steps. Simply substitute the numbers of the files that you downloaded.
The GDAL package includes a utility named gdalinfo that can be used to view the meta data for a large number of imagery file formats. Below is an example of running the command on one of the files downloaded along with the truncated results.
$ gdalinfo 32304679.tif
Driver: GTiff/GeoTIFF
Files: 32304679.tif
32304679.aux
Size is 5129, 5663
Coordinate System is:
PROJCS["NAD83 / UTM zone 18N",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-75],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","26918"]]
Origin = (487421.099999999976717,4421770.799999999813735)
Pixel Size = (0.149999999999995,-0.149999999999995)
Metadata:
AREA_OR_POINT=Area
TIFFTAG_SOFTWARE=IMAGINE TIFF Support
Copyright 1991 - 1999 by ERDAS, Inc. All Rights Reserved
@(#)$RCSfile: etif.c $ $Revision: 1.10.1.9.1.9.2.11 $ $Date: 2004/09/15 18:42:01EDT $
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1
TIFFTAG_RESOLUTIONUNIT=1 (unitless)
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 487421.100, 4421770.800) ( 75d 8'50.09"W, 39d56'45.49"N)
Lower Left ( 487421.100, 4420921.350) ( 75d 8'50.03"W, 39d56'17.94"N)
Upper Right ( 488190.450, 4421770.800) ( 75d 8'17.67"W, 39d56'45.53"N)
Lower Right ( 488190.450, 4420921.350) ( 75d 8'17.61"W, 39d56'17.98"N)
Center ( 487805.775, 4421346.075) ( 75d 8'33.85"W, 39d56'31.74"N)
...
It can be seen from the output of the command that the file is indeed a GeoTIFF image, and that it is projected into the UTM Zone 18N coordinate reference system. We also see additional information such as the corner and center coordinates.
4. Merging images into a single image file
The downloaded files are individual images that make up a larger imagery set. The graphic below shows how the individual images can be fit together using their embedded spatial meta data.

There are many reasons why one may want to keep these images as separate files, however, for our purposes we'll assume that we want to merge all of them into a single image. A utility Python script is available named, gdal_merge.py for this purpose. The following command first specifies the output merged file name and then lists the files to be included in the output file.
$ gdal_merge.py -o phila_historic.tif 32304679.tif 32833827.tif 39639558.tif 68459239.tif
The gdalinfo command can be used on the merged file to view the mete data for the merged file. As we would expect, the corner coordinates for the merged file now reflect the combined area of all four files.
$ gdalinfo phila_historic.tif
Driver: GTiff/GeoTIFF
Files: phila_histroic.tif
Size is 10256, 11325
Coordinate System is:
PROJCS["NAD83 / UTM zone 18N",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-75],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","26918"]]
Origin = (486653.250000000000000,4422620.100000000558794)
Pixel Size = (0.149999999999995,-0.149999999999995)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 486653.250, 4422620.100) ( 75d 9'22.51"W, 39d57'13.00"N)
Lower Left ( 486653.250, 4420921.350) ( 75d 9'22.39"W, 39d56'17.90"N)
Upper Right ( 488191.650, 4422620.100) ( 75d 8'17.67"W, 39d57'13.08"N)
Lower Right ( 488191.650, 4420921.350) ( 75d 8'17.56"W, 39d56'17.98"N)
Center ( 487422.450, 4421770.725) ( 75d 8'50.03"W, 39d56'45.49"N)
Band 1 Block=10256x1 Type=Byte, ColorInterp=Red
Band 2 Block=10256x1 Type=Byte, ColorInterp=Green
Band 3 Block=10256x1 Type=Byte, ColorInterp=Blue
...
5. Reproject an image into a different coordinate system
Often when working with data sets in different coordinate systems performance can be increased by converting all of the data sets to a single coordinate system. This removes the need for reprojection on every view of the data. For this example, we'll assume that the merged file will now be reprojected into WGS 84. This can be accomplished using the gdalwarp utility. The following command specifies the target coordinate system using the -t_srs flag followed by the input and output file names. Note that the source coordinate system is not needed as it is embedded in the GeoTIFF meta-data.
gdalwarp -t_srs EPSG:4326 phila_historic.tif phila_historic_4326.tif
Again we can use gdalinfo to confirm that the output file is has been reprojected to the specified coordinate system.
Driver: GTiff/GeoTIFF
Files: phila_historic_4326.tif
Size is 11658, 9905
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.2572235630016,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
...
While generally we've stayed with the bare minimum options, one that I did want to mention for gdalwarp is one that controls the resampling method. Using the -r option, the resampling method can be specified. The default value uses a nearest neighbor routine. While fast, this method may not result in the best results. Other options for this parameter include bilineatr, bicubic, cubicspline, and lanczos.
6. Build image pyramids
To increase the performance of viewing large images at different scales it is often a good idea to create pre-scaled versions of the image at different resolutions. This process is referred to as building image pyramids. GeoTIFF images can contain these pyramid images in the original file. The GDAL package include the gdaladdo (add overview image) utility for this purpose. The following command specifies our reprojected image followed by a series of overview levels to create. An overview value of 2 is a 2x reduction of the image.
$ gdaladdo phila_historic_4326.tif 2 4 8
Using gdalinfo again we can see that the overviews are now listed under the band information
Driver: GTiff/GeoTIFF
Files: phila_historic_4326.tif
Size is 11658, 9905
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.2572235630016,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-75.156253194682648,39.953632965384926)
Pixel Size = (0.000001547508316,-0.000001547508316)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( -75.1562532, 39.9536330) ( 75d 9'22.51"W, 39d57'13.08"N)
Lower Left ( -75.1562532, 39.9383049) ( 75d 9'22.51"W, 39d56'17.90"N)
Upper Right ( -75.1382123, 39.9536330) ( 75d 8'17.56"W, 39d57'13.08"N)
Lower Right ( -75.1382123, 39.9383049) ( 75d 8'17.56"W, 39d56'17.90"N)
Center ( -75.1472328, 39.9459689) ( 75d 8'50.04"W, 39d56'45.49"N)
Band 1 Block=11658x1 Type=Byte, ColorInterp=Red
Overviews: 5829x4953, 2915x2477, 1458x1239
Band 2 Block=11658x1 Type=Byte, ColorInterp=Green
Overviews: 5829x4953, 2915x2477, 1458x1239
Band 3 Block=11658x1 Type=Byte, ColorInterp=Blue
Overviews: 5829x4953, 2915x2477, 1458x1239
...
Conclusion:
The GDAL package includes a powerful set of geographic image manipulation utilities. This post only scratches the surface of its capabilities and options. There are other packages, both commercial and open source, that provide some or all of the functionality included in the GDAL utilities. However, the command line nature of these utilities make them an ideal choice for scripting tasks and for their pure convenience and options.
The approach used to meet this requirement was to use publicly available sample imagery loaded into a locally running map server. In order to use the imagery however, we wanted to perform some basic image manipulation to make it easy to deploy. While there are plenty of software packages that can preform the simple manipulation that I was doing , I used the excellent open source Geospatial Data Abstraction Library (GDAL) package originally developed by Frank Warmerdam and now released by the Open Source Geospatial Foundation (OSGeo). Since I had received a few questions from my co-workers as to how the manipulation was done, I thought I would post a very basic example of some of the manipulation that can be done with GDAL.
The steps below describe how sample imagery can be obtained from a public source, merged into a single image, reprojected into a different coordinate system, and image prynids added to the image. Note, there are a lot of options available using these command line utilities of which these steps make use of a very minimal set. This was done for ease of understanding. Full documentation on the command line options can be found at the website or in the man pages.
1. Download example imagery files
The United States Geological Survey's National Map Seamless Server facilitates downloading of spatial data resources accessible to the public. The controls on the left side of the map can be used to zoom to a specific area and request data from that area to be downloaded. The "Display" and "Download" tabs to the right of the map control the data sets that are displayed on the map and are requested for download respectively.
The following steps will use 4 GeoTIFF images of the historic section of Philadelphia from the 2004 Philadelphia orthoimagery data set. The specific images used in this example are not important, however if you would like to use the same ones they can be downloaded using the following steps.
1.1 Click the XY tool in the upper left panel and enter the following coordinates: -75.147,39.945
1.2 Using In Magnifier in the Scale information display at the top right hand corner of the page, zoo in until the scale is about 1:30,000.
1.3 On the "Display" tab on the right, expand the Orthoimagery node and select "Philadelphia (May 2004).
1.4 Switch to the Download tab and make the same selection as in the previous step. Ensure that there are no other selections in any of the nodes in the tab
1.5 Using the "Define rectangular Download Tool", the first in the Download section of the left hand toolbar, select an area that is about the middle third of the screen as shown below.
1.6 A Download Summary page will be displayed showing the files available to download. Caution, these files are about 90 MB each. If more than four files are shown, adjust the selection area for download.

1.7 Download and extract the ZIP files into a single directory.
2. Install GDAL
The GDAL library can be downloaded from the OSGeo GDAL download site, however the tools are also included in repositories for many Linux distributions. Below are the commands that will install GDAL and the Python utilities in Ubuntu.
sudo apt-get install gdal-bin
sudo apt-get install python-gdal
While we're installing software, if you do not already have a GIS desktop application you might want to consider installing one of the following excellent open source applications: Quantum GIS, or uDig.
3. Viewing image meta-data
The files downloaded from the USGS server received numeric names. Do not worry if the numbers are not the same as those used in these steps. Simply substitute the numbers of the files that you downloaded.
The GDAL package includes a utility named gdalinfo that can be used to view the meta data for a large number of imagery file formats. Below is an example of running the command on one of the files downloaded along with the truncated results.
$ gdalinfo 32304679.tif
Driver: GTiff/GeoTIFF
Files: 32304679.tif
32304679.aux
Size is 5129, 5663
Coordinate System is:
PROJCS["NAD83 / UTM zone 18N",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-75],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","26918"]]
Origin = (487421.099999999976717,4421770.799999999813735)
Pixel Size = (0.149999999999995,-0.149999999999995)
Metadata:
AREA_OR_POINT=Area
TIFFTAG_SOFTWARE=IMAGINE TIFF Support
Copyright 1991 - 1999 by ERDAS, Inc. All Rights Reserved
@(#)$RCSfile: etif.c $ $Revision: 1.10.1.9.1.9.2.11 $ $Date: 2004/09/15 18:42:01EDT $
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1
TIFFTAG_RESOLUTIONUNIT=1 (unitless)
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 487421.100, 4421770.800) ( 75d 8'50.09"W, 39d56'45.49"N)
Lower Left ( 487421.100, 4420921.350) ( 75d 8'50.03"W, 39d56'17.94"N)
Upper Right ( 488190.450, 4421770.800) ( 75d 8'17.67"W, 39d56'45.53"N)
Lower Right ( 488190.450, 4420921.350) ( 75d 8'17.61"W, 39d56'17.98"N)
Center ( 487805.775, 4421346.075) ( 75d 8'33.85"W, 39d56'31.74"N)
...
It can be seen from the output of the command that the file is indeed a GeoTIFF image, and that it is projected into the UTM Zone 18N coordinate reference system. We also see additional information such as the corner and center coordinates.
4. Merging images into a single image file
The downloaded files are individual images that make up a larger imagery set. The graphic below shows how the individual images can be fit together using their embedded spatial meta data.

There are many reasons why one may want to keep these images as separate files, however, for our purposes we'll assume that we want to merge all of them into a single image. A utility Python script is available named, gdal_merge.py for this purpose. The following command first specifies the output merged file name and then lists the files to be included in the output file.
$ gdal_merge.py -o phila_historic.tif 32304679.tif 32833827.tif 39639558.tif 68459239.tif
The gdalinfo command can be used on the merged file to view the mete data for the merged file. As we would expect, the corner coordinates for the merged file now reflect the combined area of all four files.
$ gdalinfo phila_historic.tif
Driver: GTiff/GeoTIFF
Files: phila_histroic.tif
Size is 10256, 11325
Coordinate System is:
PROJCS["NAD83 / UTM zone 18N",
GEOGCS["NAD83",
DATUM["North_American_Datum_1983",
SPHEROID["GRS 1980",6378137,298.2572221010002,
AUTHORITY["EPSG","7019"]],
AUTHORITY["EPSG","6269"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4269"]],
PROJECTION["Transverse_Mercator"],
PARAMETER["latitude_of_origin",0],
PARAMETER["central_meridian",-75],
PARAMETER["scale_factor",0.9996],
PARAMETER["false_easting",500000],
PARAMETER["false_northing",0],
UNIT["metre",1,
AUTHORITY["EPSG","9001"]],
AUTHORITY["EPSG","26918"]]
Origin = (486653.250000000000000,4422620.100000000558794)
Pixel Size = (0.149999999999995,-0.149999999999995)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 486653.250, 4422620.100) ( 75d 9'22.51"W, 39d57'13.00"N)
Lower Left ( 486653.250, 4420921.350) ( 75d 9'22.39"W, 39d56'17.90"N)
Upper Right ( 488191.650, 4422620.100) ( 75d 8'17.67"W, 39d57'13.08"N)
Lower Right ( 488191.650, 4420921.350) ( 75d 8'17.56"W, 39d56'17.98"N)
Center ( 487422.450, 4421770.725) ( 75d 8'50.03"W, 39d56'45.49"N)
Band 1 Block=10256x1 Type=Byte, ColorInterp=Red
Band 2 Block=10256x1 Type=Byte, ColorInterp=Green
Band 3 Block=10256x1 Type=Byte, ColorInterp=Blue
...
5. Reproject an image into a different coordinate system
Often when working with data sets in different coordinate systems performance can be increased by converting all of the data sets to a single coordinate system. This removes the need for reprojection on every view of the data. For this example, we'll assume that the merged file will now be reprojected into WGS 84. This can be accomplished using the gdalwarp utility. The following command specifies the target coordinate system using the -t_srs flag followed by the input and output file names. Note that the source coordinate system is not needed as it is embedded in the GeoTIFF meta-data.
gdalwarp -t_srs EPSG:4326 phila_historic.tif phila_historic_4326.tif
Again we can use gdalinfo to confirm that the output file is has been reprojected to the specified coordinate system.
Driver: GTiff/GeoTIFF
Files: phila_historic_4326.tif
Size is 11658, 9905
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.2572235630016,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
...
While generally we've stayed with the bare minimum options, one that I did want to mention for gdalwarp is one that controls the resampling method. Using the -r option, the resampling method can be specified. The default value uses a nearest neighbor routine. While fast, this method may not result in the best results. Other options for this parameter include bilineatr, bicubic, cubicspline, and lanczos.
6. Build image pyramids
To increase the performance of viewing large images at different scales it is often a good idea to create pre-scaled versions of the image at different resolutions. This process is referred to as building image pyramids. GeoTIFF images can contain these pyramid images in the original file. The GDAL package include the gdaladdo (add overview image) utility for this purpose. The following command specifies our reprojected image followed by a series of overview levels to create. An overview value of 2 is a 2x reduction of the image.
$ gdaladdo phila_historic_4326.tif 2 4 8
Using gdalinfo again we can see that the overviews are now listed under the band information
Driver: GTiff/GeoTIFF
Files: phila_historic_4326.tif
Size is 11658, 9905
Coordinate System is:
GEOGCS["WGS 84",
DATUM["WGS_1984",
SPHEROID["WGS 84",6378137,298.2572235630016,
AUTHORITY["EPSG","7030"]],
AUTHORITY["EPSG","6326"]],
PRIMEM["Greenwich",0],
UNIT["degree",0.0174532925199433],
AUTHORITY["EPSG","4326"]]
Origin = (-75.156253194682648,39.953632965384926)
Pixel Size = (0.000001547508316,-0.000001547508316)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( -75.1562532, 39.9536330) ( 75d 9'22.51"W, 39d57'13.08"N)
Lower Left ( -75.1562532, 39.9383049) ( 75d 9'22.51"W, 39d56'17.90"N)
Upper Right ( -75.1382123, 39.9536330) ( 75d 8'17.56"W, 39d57'13.08"N)
Lower Right ( -75.1382123, 39.9383049) ( 75d 8'17.56"W, 39d56'17.90"N)
Center ( -75.1472328, 39.9459689) ( 75d 8'50.04"W, 39d56'45.49"N)
Band 1 Block=11658x1 Type=Byte, ColorInterp=Red
Overviews: 5829x4953, 2915x2477, 1458x1239
Band 2 Block=11658x1 Type=Byte, ColorInterp=Green
Overviews: 5829x4953, 2915x2477, 1458x1239
Band 3 Block=11658x1 Type=Byte, ColorInterp=Blue
Overviews: 5829x4953, 2915x2477, 1458x1239
...
Conclusion:
The GDAL package includes a powerful set of geographic image manipulation utilities. This post only scratches the surface of its capabilities and options. There are other packages, both commercial and open source, that provide some or all of the functionality included in the GDAL utilities. However, the command line nature of these utilities make them an ideal choice for scripting tasks and for their pure convenience and options.
Saturday, April 12, 2008
Drop all objects from an Oracle schema
One of the projects that I have been working on recently supports using an Oracle database to persist data. In our development environment, we've setup an Oracle database instance with which individual developers and testers can utilize for their work.
One of the things that I find myself frequently needing to do is rebuild the project's database schema. As we are in active development, the objects are frequently being added, modified, and\or removed. In order to ensure that I am starting from a clean slate, I have typically dropped and recreated the database user account. However, this has a significant drawback that a second account must be used to drop and recreate the user and that account must have increased permissions in order to drop and create other users.
To solve this problem, I created a SQL*Plus script that uses the Oracle catalog tables to dynamically construct and execute a script which drops all of the user's objects.
The script has the following high -level outline:
Notes:
One of the things that I find myself frequently needing to do is rebuild the project's database schema. As we are in active development, the objects are frequently being added, modified, and\or removed. In order to ensure that I am starting from a clean slate, I have typically dropped and recreated the database user account. However, this has a significant drawback that a second account must be used to drop and recreate the user and that account must have increased permissions in order to drop and create other users.
To solve this problem, I created a SQL*Plus script that uses the Oracle catalog tables to dynamically construct and execute a script which drops all of the user's objects.
The script has the following high -level outline:
- Sets SQL*Plus parameters to enable the output of the script to be run as a script itself
- Remove the contents of the user's Oracle recycle bin
- Start spooling the output of the SQL*plus session to a file which will be executed at the end of the script
- Using the USER_CONSTRAINTS table, execute a select statement that creates disable constraint statements for all the user's constraints.
- Using the USER_OBJECTS table, execute a select statement that creates drop statements for all of the user's objects.
- End spooling the output of the SQL*Plus session
- Execute the spooled file as a script to drop the objects.
SET HEADING OFF;
SET FEEDBACK OFF;
SET ECHO OFF;
SET SQLPROMPT "--";
PURGE RECYCLEBIN;
SPOOL DropAllObjects.out;
SELECT 'ALTER TABLE ' || TABLE_NAME ||
' DISABLE CONSTRAINT ' || CONSTRAINT_NAME || ';'
FROM USER_CONSTRAINTS
ORDER BY TABLE_NAME ASC;
SELECT 'DROP ' || object_type || ' ' ||
object_name ||
DECODE(OBJECT_TYPE,'TABLE',' CASCADE CONSTRAINTS;',';')
from user_objects
ORDER BY OBJECT_TYPE ASC, OBJECT_NAME ASC;
SPOOL OFF;
START DropAllObjects.out;
EXIT;
Notes:
- While running the script it is possible to receive object does not exist errors. This is perfectly normal as the objects may have been dropped by earlier statements in the script do to cascade being used. Running the script a second time should result in no errors, as all of the objects should have been removed.
- Our project doesn't use Oracle user functions or packages so the script hasn't been extensively tested against schema which use them
Welcome!
So, I finally decided to start a public blog after putting it off for some time. Please feel free to leave comments, suggestions, corrections, or improvements regarding anything I may post about.
Thanks,
Matt
Thanks,
Matt
Subscribe to:
Posts (Atom)

