The GeoTools project strives to support as many geographical data formats as possible because getting data into the GeoTools API allows access to a vast suite of tools. In order to transform a data format into the GeoTools2 feature representation one must write an implementation of the DataStore interface.
Once a DataStore implementation is written, any information written in that format becomes available not only for GeoTools users, but also for projects built on top of GeoTools such as GeoServer and uDig.
Writing a new DataStore for GeoTools is one of the best ways to get involved in the project, as writing it will make clear many of the core concepts of the API. Finally, the modular nature of GeoTools allows new DataStores to quickly become part of the next release, so that new formats are can be distributed to to all GeoTools users.
References:
Note
AbstractDataStore is the original GeoTools 2.0 class; since that time we have learned a number of tricks and have a much easier starting point for you in the form of ContentDataStore.
While ContentDataStore is a lot less work to use; it is not yet as fully featured as AbstractDataStore. You may wish to try both tutorials before deciding on a course of action.
Note
Help Review
This article is being updated from GeoTools 2.0 - where it was in docbook.
As is usual for open source documentation is held hostage pending a volunteer to QA, or money. Open source stops with the code, documentation sounds like work so please help with feedback!
In this tutorial we will build a property file based DataStore, and in the process explore several aspects of DataStores and their implementation.
We will be working with content in the following format:
_=id:Integer,geom:Geometry,name:String
rd1=1|wkt|road one
rd2=2|wkt|road two
These examples use the file example.properties.
_=id:Integer,name:String,geom:Point
fid1=1|jody garnett|POINT(0 0)
fid2=2|brent|POINT(10 10)
fid3=3|dave|POINT(20 20)
fid4=4|justin deolivera|POINT(30 30)
If you want to follow along with this tutorial, start a new Java project in your favourite IDE, and ensure that GeoTools is on your CLASSPATH.
The DataStore we will be writing (called “PropertyDataStore”) takes a directory full of .properties files and allows reading and writing to them:
Each of the .properties files represents a “data set” - called a FeatureType by GeoTools
Each of these “data sets” contains a set of Features.
You can think of each of these .properties files as a table in a database or a shapefile (with its corresponding .dbf attributes file).
Each of the .properties is very much like a PSV (Pipe Separated Variety) database file. The first line defines the names (and types) of the columns, and the rest of the lines contain the data; each element (“column”) separated by a ‘|’ (“pipe”) character.
Consider this file (“roads.property”):
_=id:Integer,geom:Geometry,name:String
rd1=1|LINESTRING(0 0,10 10)|road one
rd2=2|LINESTRING(20 20,30 30)|road two
For the moment, ignore everything to the left of an “=”. The first line indicates that there are 3 columns. The first one is called “id” (of type Integer), the next one called “geom” (of type Geometry), and one called “name” (of type String).
The first row has id “1”, geom “LINESTRING(0 0,10 10)”, and name “road one”.
Now, lets consider the information to the left of the “=” sign. The first line begins with “_=”. This indicates this is a special line - it defines the column names and types. The rest of the lines start with a unique identifier (“rd1”, and “rd2”) - these will be the FIDs (Feature IDs) for each row (ie. a single Feature). The FID is completely different from the id attribute - every .properties file will have a FIDs, most will not have an “id” column/attribute.
So, the last data row (a Feature) has FID “rd2”, id “2”, geom “LINESTRING(20 20,30 30)”, and name “road two”.
Definitions
As you walk through this tutorial, please remember the following:
Okay with the background out of the way we can get to work.
The first step is to create a basic DataStore that only supports feature extraction. We will read data from a properties file into the GeoTools2 feature model.
To implement a DataStore we will subclass AbstractDataStore and implement three abstract methods:
To begin create the file PropertyDataStore as follows:
package org.geotools.data.property;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.FilenameFilter;
import java.io.IOException;
import org.geotools.data.AbstractDataStore;
import org.geotools.data.DataSourceException;
import org.geotools.data.DataUtilities;
import org.geotools.data.FeatureReader;
import org.geotools.feature.SchemaException;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
public class PropertyDataStore extends AbstractDataStore {
protected File directory;
protected String namespaceURI;
public PropertyDataStore(File dir) {
this(dir, null);
}
public PropertyDataStore(File dir, String namespaceURI) {
if (!dir.isDirectory()) {
throw new IllegalArgumentException(dir + " is not a directory");
}
if (namespaceURI == null) {
namespaceURI = dir.getName();
}
directory = dir;
this.namespaceURI = namespaceURI;
}
Our constructor is going to hold on to two fields:
Note
As we bring in each snippet of code you will need to import the mentioned classes. In the Eclipse IDE Control-Shift-o will organise imports and as a side effect import anything you are missing.
PropertyDataStore.getTypeNames()
A DataStore may provide access to several different types of information. The method getTypeNames provides a list of the available types.
For our purposes this list will be the name of the property files in a directory.
Add the following implementation for getTypeNames():
/**
* Gets the names of feature types available in this {@code DataStore}.
*
* @return array of type name's published by this datastore
*/
public String[] getTypeNames() {
String list[] = directory.list(new FilenameFilter() {
public boolean accept(File dir, String name) {
return name.endsWith(".properties");
}
});
for (int i = 0; i < list.length; i++) {
list[i] = list[i].substring(0, list[i].lastIndexOf('.'));
}
return list;
}
PropertyDataStore.getSchema( typeName )
Schema information is provided by the FeatureType class. This method provides access to a FeatureType referenced by a type name.
To implement this method we will need to do two things, read a line from a properties file, and interpret the line as a FeatureType.
The DataUtilities class provides an assortment of helper functions. In this method we will use DataUtilities.createType( name, spec ).
Note
DataUtilities is a class especially designed for this tutorial.
Those experienced with GeoTools may find these humble beginnings amusing given how widely used DataUtilities is today.
Add getSchema( typeName ):
/**
* Creates a Schema (FeatureType) from the first line of the .properties
* file
*
* @param typeName TypeName indicating the property file used
*/
public SimpleFeatureType getSchema(String typeName) throws IOException {
String typeSpec = property(typeName, "_");
try {
String namespace = directory.getName();
return DataUtilities.createType(namespace + "." + typeName,
typeSpec);
} catch (SchemaException e) {
e.printStackTrace();
throw new DataSourceException(typeName + " schema not available", e);
}
}
Add property( typeName, key ):
/**
* Opens the file given in typeName and reads through looking for a line
* that begins with key and then "=".
*
* @param typeName indicates file to open
* @param key indicates the line to read
* @return the key's values (everything to the right of the '='
* @throws IOException
*/
private String property(String typeName, String key) throws IOException {
File file = new File(directory, typeName + ".properties");
BufferedReader reader = new BufferedReader(new FileReader(file));
try {
for (String line = reader.readLine(); line != null; line = reader
.readLine()) {
if (line.startsWith(key + "=")) {
return line.substring(key.length() + 1);
}
}
} finally {
reader.close();
}
return null;
}
PropertyDataStore.getFeatureReader( typeName )
FeatureReader is the low-level API provided by DataStore for accessing Feature content.
The method AbstractDataStore.getFeatureReader( typeName ) is required by the superclass AbstractDataStore and is not part of the public DataStore API accessed by user. We will cover how this method is used at the end of this tutorial where we discuss optimisation.
Add PropertyDataStore.getFeatureReader( typeName ):
/**
* Implements access to the "raw" FeatureReader, this method is called
* internally by AbstractDataStore.
*
* @param typeName TypeName indicating property file to read
* @return FeatureReader providing access to contents of file
*/
protected FeatureReader<SimpleFeatureType, SimpleFeature> getFeatureReader(
String typeName) throws IOException {
return new PropertyFeatureReader(directory, typeName);
}
Note
Next up we will be implementing PropertyFeatureReader mentioned above.
If you are in Eclipse you can:
FeatureReader is similar to the Java Iterator construct, with the addition of FeatureType (and IOExceptions).
FeatureReader interface:
To implement our FeatureReader, we will need to do several things: open a File and read through it line by line, parsing Features as we go.
PropertyFeatureReader
Create the file PropertyFeatureReader as follows:
package org.geotools.data.property;
import java.io.File;
import java.io.IOException;
import java.util.NoSuchElementException;
import java.util.logging.Logger;
import org.geotools.data.FeatureReader;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.util.logging.Logging;
import org.opengis.feature.IllegalAttributeException;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
/**
* FeatureReader access to the contents of a PropertyFile.
*
* @author Jody Garnett
* @version 8.0.0
* @since 2.0.0
*/
public class PropertyFeatureReader implements
FeatureReader<SimpleFeatureType, SimpleFeature> {
private static final Logger LOGGER = Logging
.getLogger("org.geotools.data.property");
PropertyAttributeReader reader;
/**
* Creates a new PropertyFeatureReader object.
*
* @param directory Directory containing property file
* @param typeName TypeName indicating file to read
* @throws IOException
*/
public PropertyFeatureReader(File directory, String typeName)
throws IOException {
File file = new File(directory, typeName + ".properties");
reader = new PropertyAttributeReader(file);
}
/**
* Access to schema description.
*
* @return SimpleFeatureType describing attribtues
*/
public SimpleFeatureType getFeatureType() {
return reader.type;
}
/**
* Access the next feature (if available).
*
* @return SimpleFeature read from property file
* @throws IOException
* @throws IllegalAttributeException
* @throws NoSuchElementException
*/
public SimpleFeature next() throws IOException, IllegalAttributeException,
NoSuchElementException {
reader.next();
SimpleFeatureType type = reader.type;
String fid = reader.getFeatureID();
Object[] values = new Object[reader.getAttributeCount()];
for (int i = 0; i < reader.getAttributeCount(); i++) {
try {
values[i] = reader.read(i);
} catch (RuntimeException e) {
values[i] = null;
} catch (IOException e) {
throw e;
}
}
return SimpleFeatureBuilder.build(type, values, fid);
}
/**
* Check if additional contents are available.
*
* @return <code>true</code> if additional contents are available
* @throws IOException
*/
public boolean hasNext() throws IOException {
return reader.hasNext();
}
/**
* Close the FeatureReader when not in use.
*
* @throws IOException
*/
public void close() throws IOException {
if (reader == null) {
LOGGER.warning("Stream seems to be already closed.");
} else {
reader.close();
}
reader = null;
}
}
The helper class PropertyAttributeReader will be used to accomplish the bulk of this work.
Note
Note the use of the GeoTools Logging system. GeoTools provides a wrapper around the usual suspects (Java Logging, Log4J, etc...) allowing users to configure the library to work with the logging system employed by their application.
We are just that cool :-)
The AttributeReader interface is used to provide access to individual attributes from a storage medium. It is hoped that high level operations (such as Joining) could make use of this capability.
Note
If it makes sense for your data format you could just do all the work in your FeaureReader.
Why would you break things up into AttributeReaders? If you had several files you were merging together (such as is the case for Shapefile which has shp, dbf, and shx files).
AttributeReader interface:
Because this class actually does some work, we are going to include a few more comments in the code to keep our heads on straight.
Create the file PropertyAttributeReader as follows:
package org.geotools.data.property;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.NoSuchElementException;
import org.geotools.data.AttributeReader;
import org.geotools.data.DataSourceException;
import org.geotools.data.DataUtilities;
import org.geotools.feature.SchemaException;
import org.geotools.util.Converters;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.feature.type.AttributeDescriptor;
import org.opengis.feature.type.GeometryType;
import org.opengis.referencing.crs.CoordinateReferenceSystem;
import com.vividsolutions.jts.geom.Geometry;
/**
* Simple AttributeReader that works against Java properties files.
* <p>
* This AttributeReader is part of the GeoTools AbstractDataStore tutorial, and
* should be considered a Toy.
* </p>
* <p>
* The content of this file should start with a the property "_" with the value
* being the typeSpec describing the featureType. Thereafter each line will
* should have a FeatureID as the property and the attribtues as the value
* separated by | characters.
* </p>
*
* <pre>
* <code>
* _=id:Integer|name:String|geom:Geometry
* fid1=1|Jody|<i>well known text</i>
* fid2=2|Brent|<i>well known text</i>
* fid3=3|Dave|<i>well known text</i>
* </code>
* </pre>
* <p>
* May values may be represented by a special tag: <code><null></code>. An empty
* element: <code>||</code> is interpreted as the empty string:
* </p>
*
* <pre>
* <code>
* fid4=4||<null> -> Feature( id=2, name="", geom=null )
* </code>
* </pre>
*
* @author Jody Garnett
*/
public class PropertyAttributeReader implements AttributeReader {
BufferedReader reader;
SimpleFeatureType type;
String line;
String next;
String[] text;
String fid;
/**
* Creates a new PropertyAttributeReader object.
*
* @param file File being read
* @throws IOException
* @throws DataSourceException
*/
public PropertyAttributeReader(File file) throws IOException {
String typeName = typeName(file);
String namespace = namespace(file);
reader = new BufferedReader(new FileReader(file));
// read until "_=";
while ((line = reader.readLine()) != null) {
if (line.startsWith("_="))
break;
}
if ((line == null) || !line.startsWith("_=")) {
throw new IOException(typeName + " schema not available");
}
String typeSpec = line.substring(2);
try {
type = DataUtilities.createType(namespace, typeName, typeSpec);
} catch (SchemaException e) {
throw new DataSourceException(typeName + " schema not available", e);
}
line = null;
next = null;
}
/**
* TypeName for the provided file.
*
* @param file File being read
* @return suitable typeName
*/
private static String typeName(File file) {
String name = file.getName();
int split = name.lastIndexOf('.');
return (split == -1) ? name : name.substring(0, split);
}
/**
* Namespace for the provided file
*
* @param file File being read
* @return suitable namespace
*/
private static String namespace(File file) {
File parent = file.getParentFile();
return (parent == null) ? "" : (parent.getName() + ".");
}
Our constructor acquires the type information from the header, using a function from DataUtilities to interpret the type specification. The filename is used as the name for the resulting FeatureType, and the directory name is used for the namespace.
The BufferedReader, reader, is opened and it will be this class that allows us to stream over contents as a series of Features.
Note
We are opening this in the constructor in order raise an IOException if the file cannot be used (rather than wait until next() is called).
We will use a two part strategy for determining if more content is available. We will try and acquire the ‘next’ line in the hasNext() method, using the next() method to update ‘line’ to the contents of ‘next’. All attribute operations will be performed against the current ‘line’.
With these ideas in mind we can implement the required methods:
/**
* Number of attributes to expect based on header information.
*
* @return number of attribtues
*/
public int getAttributeCount() {
return type.getAttributeCount();
}
/**
* AttribtueDescriptor (name and type) for position marked by index.
*
* @param index
* @return AttributeDescriptor describing attribute name and type
* @throws ArrayIndexOutOfBoundsException
*/
public AttributeDescriptor getAttributeType(int index)
throws ArrayIndexOutOfBoundsException {
return type.getDescriptor(index);
}
/**
* Close the internal reader accessing the file.
*
* @throws IOException
*/
public void close() throws IOException {
reader.close();
reader = null;
}
/**
* Check if the file has another line.
*
* @return <code>true</code> if the file has another line
* @throws IOException
*/
public boolean hasNext() throws IOException {
if (next != null) {
return true;
}
next = reader.readLine();
return next != null;
}
/**
* Read the next line from the reader.
*
* @return line
* @throws IOException
*/
String readLine() throws IOException {
while( true ){
String line = reader.readLine();
if( line == null ){
return null; // no more content
}
if( line.startsWith("#") || line.startsWith("!")){
continue; // skip comments
}
else {
return line;
}
}
}
/**
* Retrieve the next line.
*
* @throws IOException
* @throws NoSuchElementException
*/
public void next() throws IOException {
if (hasNext()) {
line = next;
next = null;
int split = line.indexOf('=');
fid = line.substring(0, split);
text = line.substring(split + 1).split("\\|");
if (type.getAttributeCount() != text.length)
throw new DataSourceException("format error: expected "
+ type.getAttributeCount() + " attributes, but found "
+ text.length + ". [" + line + "]");
} else {
throw new NoSuchElementException();
}
}
/**
* Read attribute in position marked by <code>index</code>.
*
* @param index Attribute position to read
* @return Value for the attribtue in position <code>index</code>
* @throws IOException
* @throws ArrayIndexOutOfBoundsException
*/
public Object read(int index) throws IOException,
ArrayIndexOutOfBoundsException {
if (line == null) {
throw new IOException(
"No content available - did you remeber to call next?");
}
AttributeDescriptor attType = type.getDescriptor(index);
String stringValue = null;
try {
// read the value
stringValue = text[index];
// trim off any whitespace
if (stringValue != null) {
stringValue = stringValue.trim();
}
if ("".equals(stringValue)) {
stringValue = null;
}
} catch (RuntimeException e1) {
e1.printStackTrace();
stringValue = null;
}
// check for special <null> flag
if ("<null>".equals(stringValue)) {
stringValue = null;
}
if (stringValue == null) {
if (attType.isNillable()) {
return null;
}
}
// Use of Converters to convert from String to requested java binding
Object value = Converters.convert(stringValue, attType.getType()
.getBinding());
if (attType.getType() instanceof GeometryType) {
// this is to be passed on in the geometry objects so the srs name
// gets encoded
CoordinateReferenceSystem crs = ((GeometryType) attType.getType())
.getCoordinateReferenceSystem();
if (crs != null) {
// must be geometry, but check anyway
if (value != null && value instanceof Geometry) {
((Geometry) value).setUserData(crs);
}
}
}
return value;
}
Finally, since our file format does support FeatureID we will need a way to let our FeatureReader know:
/**
* Retrieve the FeatureId identifying the current line.
*
* @return FeatureID for the current line.
*/
public String getFeatureID() {
if (line == null) {
return null;
}
return fid;
}
We can make use of getFeatureID() to supply a FeatureID for FeatureReader.
Note
Many other DataStores derive a FeatureID from their attributes, or the current line number.
Since a FeatureID must start with a letter a common approach is to prepend the TypeName followed by a dot to the line number, or database row ID number.
FeatureID generation example:
public String deriveFeatureID(){
return type.getTypeName()+"."+id_number;
}
To make your DataStore truly independent and plugable, you must create a class implementing the DataStoreFactorySpi interface.
This allows the Service Provider Interface mechanism to dynamically plug in your new DataStore with no implementation knowledge. Code that uses the DataStoreFinder can just add the new DataStore to the classpath and it will work!
The DataStoreFactorySpi provides information on the Parameters required for construction. DataStoreFactoryFinder provides the ability to create DataStores representing existing information and the ability to create new physical storage.
PropertyDataStoreFactory
Create PropertyDataStoreFactory as follows:
package org.geotools.data.property;
import java.awt.RenderingHints;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.Serializable;
import java.util.Collections;
import java.util.Map;
import org.geotools.data.DataStore;
import org.geotools.data.DataStoreFactorySpi;
/**
* DataStore factory that creates
* {@linkplain org.geotools.data.property.PropertyDataStore}s
*
* @author Jody Garnett
*/
public class PropertyDataStoreFactory implements DataStoreFactorySpi {
/**
* Public "no arguments" constructor called by Factory Service Provider
* (SPI) based on entry in
* META-INF/services/org.geotools.data.DataStoreFactorySpi
*/
public PropertyDataStoreFactory() {
}
/**
* No implementation hints are provided at this time.
*/
public Map<RenderingHints.Key, ?> getImplementationHints() {
return Collections.emptyMap();
}
We have a couple of methods to describe the DataStore.
public String getDisplayName() {
return "Properties";
}
public String getDescription() {
return "Allows access to Java Property files containing Feature information";
}
/**
* Test to see if this datastore is available, if it has all the appropriate
* libraries to construct a datastore. This datastore just returns true for
* now. This method is used for interactive applications, so as to not
* advertise data store capabilities they don't actually have.
*
* @return <tt>true</tt> if and only if this factory is available to create
* DataStores.
* @task <code>true</code> property datastore is always available
*/
public boolean isAvailable() {
return true;
}
The user is expected to supply a Map of connection parameters when creating a datastore.
The allowable connection parameters are described using Parameter (as defined by gt-api docs). This captures the “key” used to store the value in the map, and the expected java type for the value.
public static final Param DIRECTORY = new Param("directory", File.class,
"Directory containting property files", true);
public static final Param NAMESPACE = new Param("namespace", String.class,
"namespace of datastore", false);
/**
* @see #DIRECTORY
* @see PropertyDataStoreFactory#NAMESPACE
*/
public Param[] getParametersInfo() {
return new Param[] { DIRECTORY, NAMESPACE };
}
We have some code to check if a set of provided connection parameters can actually be used.
/**
* Works for a file directory or property file
*
* @param params Connection parameters
* @return true for connection parameters indicating a directory or property
* file
*/
public boolean canProcess(Map<String, Serializable> params) {
try {
directoryLookup(params);
return true;
} catch (Exception erp) {
return false; // can't process, just return false
}
}
/**
* Lookups the directory containing property files in the params argument,
* and returns the corresponding <code>java.io.File</code>.
* <p>
* The file is first checked for existence as an absolute path in the
* filesystem. If such a directory is not found, then it is treated as a
* relative path, taking Java system property <code>"user.dir"</code> as the
* base.
* </p>
*
* @param params
* @throws IllegalArgumentException if directory is not a directory.
* @throws FileNotFoundException if directory does not exists
* @throws IOException if {@linkplain #DIRECTORY} doesn't find parameter in
* <code>params</code> file does not exists.
*/
private File directoryLookup(Map<String, java.io.Serializable> params)
throws IOException, FileNotFoundException, IllegalArgumentException {
File file = (File) DIRECTORY.lookUp(params);
if (!file.exists()) {
File currentDir = new File(System.getProperty("user.dir"));
String path = DIRECTORY.lookUp(params).toString();
file = new File(currentDir, path);
if (!file.exists()) {
throw new FileNotFoundException(file.getAbsolutePath());
}
}
if (file.isDirectory()) {
return file;
} else {
// check if they pointed to a properties file; and use the parent
// directory
if (file.getPath().endsWith(".properties")) {
return file.getParentFile();
} else {
throw new IllegalArgumentException(file.getAbsolutePath()
+ " is not a directory");
}
}
}
Note
The directoryLookup has gotten considerably more complicated since this tutorial was first written. One of the benifits of PropertyDataStore being used in real world situtations.
Armed with a map of connection parameters we can now:
create a Datastore for an existing property file; and
create a datastore for a new property file
Since initially our DataStore is read-only we will just throw an UnsupportedOperationException at this time.
Here is the code that finally calls our PropertyDataStore constructor:
public DataStore createDataStore(Map<String, java.io.Serializable> params)
throws IOException {
File dir = directoryLookup(params);
String namespaceURI = (String) NAMESPACE.lookUp(params);
if (dir.exists() && dir.isDirectory()) {
return new PropertyDataStore(dir, namespaceURI);
} else {
throw new IOException("Directory is required");
}
}
public DataStore createNewDataStore(Map<String, java.io.Serializable> params)
throws IOException {
throw new UnsupportedOperationException("PropertyDataStore is read-only");
}
The Factory Service Provider (SPI) system operates by looking at the META-INF/services folder and checking for implemetnations of DataStoreFactorySpi
To “register” our PropertyDataStoreFactory please create the following file:
This file requires the filename of the factory that implements the DataStoreSpi interface.
Fill in the following content for your org.geotools.data.DataStoreFactorySpi file:
org.geotools.data.tutorial.PropertiesDataStoreFactory
That is it, in the next section we will try out your new DataStore.
In this part we examine the abilities of the PropertyDataStore implemented in Part 2.
Now that we have implemented a simple DataStore we can explore some of the capabilites made available to us.
PropertyDataStore API for data access:
If you would like to follow along with these examples you can PropertyExamples.java.
DataStore.getTypeNames()
The method getTypeNames provides a list of the available types.
getTypeNames() example:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
String names[] = store.getTypeNames();
System.out.println("typenames: " + names.length);
System.out.println("typename[0]: " + names[0]);
Produces the following output (given a directory with example.properties):
typenames: 1
typename[0]: example
DataStore.getSchema( typeName )
The method getSchema( typeName ) provides access to a FeatureType referenced by a type name.
getSchema( typeName ) example:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
SimpleFeatureType type = store.getSchema("example");
System.out.println(" typeName: " + type.getTypeName());
System.out.println(" name: " + type.getName());
System.out.println("attribute count: " + type.getAttributeCount());
AttributeDescriptor id = type.getDescriptor(0);
System.out.println("attribute 'id' name:" + id.getName());
System.out.println("attribute 'id' type:" + id.getType().toString());
System.out.println("attribute 'id' binding:"
+ id.getType().getDescription());
AttributeDescriptor name = type.getDescriptor("name");
System.out.println("attribute 'name' name:" + name.getName());
System.out.println("attribute 'name' binding:"
+ name.getType().getBinding());
Produces the following output:
typeName: example
name: example4149306500761583641:example
attribute count: 3
attribute 'id' name:id
attribute 'id' type:AttributeTypeImpl id<Integer>
attribute 'id' binding:null
attribute 'name' name:name
attribute 'name' binding:class java.lang.String
DataStore.getFeatureReader( query, transaction )
The method getFeatureReader( query, transaction ) allows access to the contents of our DataStore.
The method signature may be more complicated than expected, we certainly did not talk about Query or Transactions when we implemented our PropertyDataStore. This is something that AbstractDataStore is handling for you and will be discussed later in the section on optimisation.
Query.getTypeName()
Determines which FeatureType is being requested. In addition, Query supports the customization attributes, namespace, and typeName requested from the DataStore. While you may use DataStore.getSchema( typeName ) to retrieve the types as specified by the DataStore, you may also create your own FeatureType to limit the attributes returned or cast the result into a specific namespace.
Query.getFilter()
Used to define constraints on which features should be fetched. The constraints can be on spatial and non-spatial attributes of the features.
Query.getPropertiesNames()
Allows you to limit the number of properties of the returned Features to only those you are interested in.
Query.getMaxFeatures()
Defines an upper limit for the number of features returned.
Query.getHandle()
User-supplied name used to describe a query in user’s terms in any generated error messages.
Query.getCoordinateSystem()
Used to force the use of a user-supplied CoordinateSystem (rather than the one supplied by the DataStore). This capability will allow client code to use our DataStore with a CoordinateSystem of their choice. The coordinates returned by the DataStore will not be modified in any way.
Query.getCoordinateSystemReproject()
Used to reproject the Geometries provided by a DataStore from their original value (or the one provided by Query.getCoordinateSystem) into a different coordinate system. The coordinate returned by the DataStore will be processed , either natively by Advanced DataStores, or using GeoTools reprojection routines.
Note
Since this tutorial was writen Query has expanding its capabilities (and the capabilities of your DataStore) to include support for reprojection.
It also offers an “open ended” pathway for expansion using “query hints”.
Transaction
Allows access the contents of a DataStore during modification.
With all of that in mind we can now proceed to our DataStore.getFeatureReader( featureType, filter, transaction ) example:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore datastore = DataStoreFinder.getDataStore(params);
Query query = new Query("example");
FeatureReader<SimpleFeatureType, SimpleFeature> reader = datastore
.getFeatureReader(query, Transaction.AUTO_COMMIT);
try {
int count = 0;
while (reader.hasNext()) {
SimpleFeature feature = reader.next();
System.out.println("feature " + count + ": " + feature.getID());
count++;
}
System.out.println("read in " + count + " features");
} finally {
reader.close();
}
Produces the following output:
feature 0: fid1
feature 1: fid2
feature 2: fid3
feature 3: fid4
read in 4 features
Example with a quick “selection” Filter:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
FilterFactory ff = CommonFactoryFinder.getFilterFactory();
Set<FeatureId> selection = new HashSet<FeatureId>();
selection.add(ff.featureId("fid1"));
Filter filter = ff.id(selection);
Query query = new Query("example", filter);
FeatureReader<SimpleFeatureType, SimpleFeature> reader = store
.getFeatureReader(query, Transaction.AUTO_COMMIT);
try {
while (reader.hasNext()) {
SimpleFeature feature = reader.next();
System.out.println("feature " + feature.getID());
for (Property property : feature.getProperties()) {
System.out.print("\t");
System.out.print(property.getName());
System.out.print(" = ");
System.out.println(property.getValue());
}
}
} finally {
reader.close();
}
Produces the following output:
feature fid1
id = 1
name = jody garnett
geom = POINT (0 0)
DataStore.getFeatureSource( typeName )
This method is the gateway to our high level as provided by an instance of FeatureSource, FeatureStore or FeatureLocking. The returned instance represents the contents of a single named FeatureType provided by the DataStore. The type of the returned instance indicates the capabilities available.
This far in our tutorial PropertyDataStore will only support an instance of FeatureSource.
Example getFeatureSource:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
SimpleFeatureSource featureSource = store.getFeatureSource("example");
Filter filter = CQL.toFilter("name = 'dave'");
SimpleFeatureCollection features = featureSource.getFeatures(filter);
System.out.println("found :" + features.size() + " feature");
SimpleFeatureIterator iterator = features.features();
try {
while (iterator.hasNext()) {
SimpleFeature feature = iterator.next();
Geometry geometry = (Geometry) feature.getDefaultGeometry();
System.out.println(feature.getID() + " location " + geometry);
}
} catch (Throwable t) {
iterator.close();
}
Producing the following output:
found :1 feature
fid3 location POINT (20 20)
FeatureSource provides the ability to query a DataStore and represents the contents of a single FeatureType. In our example, the PropertiesDataStore represents a directory full of properties files. FeatureSource will represent a single one of those files.
FeatureSource defines:
FeatureSource also defines an event notification system and provides access to the DataStore which created it. You may have more than one FeatureSource operating against a file at any time.
While the FeatureSource API does allow you to represent a named FeatureType, it still does not allow direct access to a FeatureReader. The getFeatures methods actually return an instance of FeatureCollection.
FeatureCollection defines:
FeatureCollection is the closest thing we have to a prepared request. Many DataStores are able to provide optimised implementations that handles the above methods natively.
FeatureCollection Example:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
SimpleFeatureSource featureSource = store.getFeatureSource("example");
SimpleFeatureCollection featureCollection = featureSource.getFeatures();
SimpleFeatureIterator features = featureCollection.features();
List<String> list = new ArrayList<String>();
try {
while (features.hasNext()) {
list.add(features.next().getID());
}
} finally {
features.close();
}
System.out.println(" contents:" + list);
System.out.println(" count:" + featureSource.getCount(Query.ALL));
System.out.println(" bounds:" + featureSource.getBounds(Query.ALL));
System.out.println(" size:" + featureCollection.size());
System.out.println(" bounds:" + featureCollection.getBounds());
System.out.println("collection: "
+ DataUtilities.collection(featureCollection).size());
With the following output:
contents:[fid1, fid2, fid3, fid4]
count:-1
bounds:null
size:4
bounds:ReferencedEnvelope[0.0 : 30.0, 0.0 : 30.0]
collection: 4
Note
In the above example, FeatureSource.count(Query.ALL) will return -1, indicating that the value is expensive for the DataStore to calculate, or at least that our PropertyDataStore implementation does not provide an optimised implementation.
FeatureCollection.size() will always produce an answer
You can think of this as:
Care should be taken when using the collection() method to capture the contents of a DataStore in memory. GIS applications often produce large volumes of information and can place a strain on memory use.
In this part we will complete the PropertyDataStore started above. At the end of this section we will have a full functional PropertyDataStore supporting both read and write operations.
The DataStore API has two methods that are involved in making content writable.
AbstractDataStore asks us to implement two things in our subclass:
FeatureWriter defines the following methods:
Change notification for users is made available through several FeatureSource methods:
To trigger the featureListeners we will make use of several helper methods in AbstractDataSource:
Now that we are going to be writing files we can fill in the createNewDataStore method.
Open up PropertyDataStoreFactory and replace createNewDataStore with the following:
public DataStore createNewDataStore(Map params) throws IOException {
File dir = (File)DIRECTORY.lookUp(params);
if (dir.exists()) {
throw new IOException(dir + " already exists");
}
boolean created;
created = dir.mkdir();
if (!created) {
throw new IOException("Could not create the directory" + dir);
}
String namespaceURI = (String) NAMESPACE.lookUp(params);
return new PropertyDataStore(dir,namespaceURI);
}
No surprises here; the code creates a directory for PropertyDataStore to work in.
To start with, we need to make a change to our PropertyDataStore constructor:
public PropertyDataStore(File dir, String namespaceURI) {
super(true); // true indicates we implement getFeatureWriter
if (!dir.isDirectory()) {
throw new IllegalArgumentException(dir + " is not a directory");
}
if (namespaceURI == null) {
namespaceURI = dir.getName();
}
directory = dir;
this.namespaceURI = namespaceURI;
}
This change will tell AbstractDataStore that our subclass is willing to modify Features.
Implement createSchema( featureType)
This method provides the ability to create a new FeatureType. For our DataStore we will use this to create new properties files.
To implement this method we will once again make use of the DataUtilities class.
Add createSchema( featureType ):
public void createSchema(SimpleFeatureType featureType) throws IOException {
String typeName = featureType.getTypeName();
File file = new File(directory, typeName + ".properties");
BufferedWriter writer = new BufferedWriter(new FileWriter(file));
writer.write("_=");
writer.write(DataUtilities.spec(featureType));
writer.flush();
writer.close();
}
Implement getFeatureWriter( typeName )
FeatureWriter is the low-level API storing Feature content. This method is not part of the public DataStore API and is only used by AbstractDataStore.
Add getFeatureWriter( typeName ):
protected FeatureWriter<SimpleFeatureType, SimpleFeature> createFeatureWriter(String typeName,
Transaction transaction) throws IOException {
return new PropertyFeatureWriter(this, typeName);
}
FeatureWriter is less intuitive than FeatureReader in that it does not follow the example of Iterator as closely.
Our implementation of a FeatureWriter needs to do two things: support the FeatureWriter interface and inform the DataStore of modifications.
Create the file PropertyFeatureWriter.java:
package org.geotools.data.property;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.channels.FileChannel;
import org.geotools.data.DataSourceException;
import org.geotools.data.DataUtilities;
import org.geotools.data.FeatureWriter;
import org.geotools.data.Transaction;
import org.geotools.factory.Hints;
import org.geotools.feature.simple.SimpleFeatureBuilder;
import org.geotools.geometry.jts.ReferencedEnvelope;
import org.opengis.feature.IllegalAttributeException;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
/**
* Uses PropertyAttributeWriter to generate a property file on disk.
*
*
* @source $URL: http://svn.osgeo.org/geotools/trunk/modules/plugin/property/src/main/java/org/geotools/data/property/PropertyFeatureWriter.java $
*/
public class PropertyFeatureWriter implements
FeatureWriter<SimpleFeatureType, SimpleFeature> {
PropertyDataStore store;
File read;
PropertyAttributeReader reader;
File write;
PropertyAttributeWriter writer;
SimpleFeature origional = null;
SimpleFeature live = null;
public PropertyFeatureWriter(PropertyDataStore dataStore, String typeName)
throws IOException {
store = dataStore;
File dir = store.directory;
read = new File(dir, typeName + ".properties");
write = File.createTempFile(typeName + System.currentTimeMillis(),
null, dir);
reader = new PropertyAttributeReader(read);
writer = new PropertyAttributeWriter(write, reader.type);
}
Our constructor creates a PropertyAttributeReader to access the existing contents of the DataStore. We made use of PropertyAttributeReader to implement PropertyFeatureReader in Section 1.
We also create a PropertyAttributeWriter operating against a temporary file. When the FeatureWriter is closed we will delete the original file and replace it with our new file.
Add FeatureWriter.getFeatureType() implementation:
public SimpleFeatureType getFeatureType() {
return reader.type;
}
Add hasNext() implementation:
long nextFid = System.currentTimeMillis(); // seed with a big number
public SimpleFeature next() throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
String fid = null;
SimpleFeatureType type = reader.type;
try {
if (hasNext()) {
reader.next(); // grab next line
fid = reader.getFeatureID();
Object values[] = new Object[reader.getAttributeCount()];
for (int i = 0; i < reader.getAttributeCount(); i++) {
values[i] = reader.read(i);
}
origional = SimpleFeatureBuilder.build(type, values, fid);
live = SimpleFeatureBuilder.copy(origional);
return live;
} else {
fid = type.getTypeName() + "." + (nextFid++);
Object values[] = DataUtilities.defaultValues(type);
origional = null;
live = SimpleFeatureBuilder.build(type, values, fid);
return live;
}
} catch (IllegalAttributeException e) {
String message = "Problem creating feature "
+ (fid != null ? fid : "");
throw new DataSourceException(message, e);
}
}
Our FeatureWriter makes use of two Features:
When the FeatureWriter is used to write or remove information, the contents of both live and feature are set to null. If this has not been done already we will write out the current feature.
Add the helper function writeImplementation( Feature ):
private void writeImplementation(SimpleFeature f) throws IOException {
writer.next();
String fid = f.getID();
if( Boolean.TRUE.equals( f.getUserData().get(Hints.USE_PROVIDED_FID) ) ){
if( f.getUserData().containsKey(Hints.PROVIDED_FID)){
fid = (String) f.getUserData().get(Hints.PROVIDED_FID);
}
}
writer.writeFeatureID(fid);
for (int i = 0; i < f.getAttributeCount(); i++) {
Object value = f.getAttribute(i);
writer.write(i, value );
}
}
Add next() implementation:
long nextFid = System.currentTimeMillis(); // seed with a big number
public SimpleFeature next() throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
String fid = null;
SimpleFeatureType type = reader.type;
try {
if (hasNext()) {
reader.next(); // grab next line
fid = reader.getFeatureID();
Object values[] = new Object[reader.getAttributeCount()];
for (int i = 0; i < reader.getAttributeCount(); i++) {
values[i] = reader.read(i);
}
origional = SimpleFeatureBuilder.build(type, values, fid);
live = SimpleFeatureBuilder.copy(origional);
return live;
} else {
fid = type.getTypeName() + "." + (nextFid++);
Object values[] = DataUtilities.defaultValues(type);
origional = null;
live = SimpleFeatureBuilder.build(type, values, fid);
return live;
}
} catch (IllegalAttributeException e) {
String message = "Problem creating feature "
+ (fid != null ? fid : "");
throw new DataSourceException(message, e);
}
}
The next method is used for two purposes:
To access existing Features, the AttributeReader is advanced, the current attribute and feature ID assembled into a Feature. This Feature is then duplicated and returned to the user. We will later compare the original to the user’s copy to check if any modifications have been made.
Add write() implementation:
public void write() throws IOException {
if (live == null) {
throw new IOException("No current feature to write");
}
if (live.equals(origional)) {
writeImplementation(origional);
} else {
writeImplementation(live);
String typeName = live.getFeatureType().getTypeName();
Transaction autoCommit = Transaction.AUTO_COMMIT;
if (origional != null) {
ReferencedEnvelope bounds = new ReferencedEnvelope();
bounds.include(live.getBounds());
bounds.include(origional.getBounds());
store.listenerManager.fireFeaturesChanged(typeName, autoCommit,
bounds, false);
} else {
store.listenerManager.fireFeaturesAdded(typeName, autoCommit,
ReferencedEnvelope.reference(live.getBounds()), false);
}
}
origional = null;
live = null;
}
In the write method we will need to check to see whether the user has changed anything. If so, we will need to remember to issue event notification after writing out their changes.
Add remove() implementation:
public void remove() throws IOException {
if (live == null) {
throw new IOException("No current feature to remove");
}
if (origional != null) {
String typeName = live.getFeatureType().getTypeName();
Transaction autoCommit = Transaction.AUTO_COMMIT;
store.listenerManager.fireFeaturesRemoved(typeName, autoCommit,
ReferencedEnvelope.reference(origional.getBounds()), false);
}
origional = null;
live = null; // prevent live and remove from being written out
}
To implement remove, we will skip over the origional feature (and just won’t write it out). Most of the method is devoted to gathering up the information needed to issue a feature removed event.
Add close() Implementation:
public void close() throws IOException {
if (writer == null) {
throw new IOException("writer already closed");
}
// write out remaining contents from reader
// if applicable
while (reader.hasNext()) {
reader.next(); // advance
writer.next();
writer.echoLine(reader.line); // echo unchanged
}
writer.close();
reader.close();
writer = null;
reader = null;
read.delete();
if (write.exists() && !write.renameTo(read)) {
FileChannel out = new FileOutputStream(read).getChannel();
FileChannel in = new FileInputStream(write).getChannel();
try {
long len = in.size();
long copied = out.transferFrom(in, 0, in.size());
if (len != copied) {
throw new IOException("unable to complete write");
}
} finally {
in.close();
out.close();
}
}
read = null;
write = null;
store = null;
}
To implement close() we must remember to write out any remaining features in the DataStore before closing our new file. To implement this we have performed a small optimization: we echo the line acquired by the PropertyFeatureReader.
The last thing our FeatureWriter must do is replace the existing File with our new one.
In the previous section we explored the capabilities of our PropertyWriter through actual use; now we can go ahead and define the class.
Create PropertyAttributeWriter:
package org.geotools.data.property;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.regex.MatchResult;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.geotools.data.AttributeWriter;
import org.geotools.data.DataUtilities;
import org.geotools.util.Converters;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.feature.type.AttributeDescriptor;
import com.vividsolutions.jts.geom.Geometry;
/**
* Simple AttributeWriter that produces Java properties files.
* <p>
* This AttributeWriter is part of the geotools2 DataStore tutorial, and should
* be considered a Toy.
* </p>
* <p>
* The content produced witll start with the property "_" with the value being
* the typeSpec describing the featureType. Thereafter each line will represent
* a Features with FeatureID as the property and the attribtues as the value
* separated by | characters.
* </p>
*
* <pre>
* <code>
* _=id:Integer|name:String|geom:Geometry
* fid1=1|Jody|<i>well known text</i>
* fid2=2|Brent|<i>well known text</i>
* fid3=3|Dave|<i>well known text</i>
* </code>
* </pre>
*
* @author Jody Garnett
*
*
* @source $URL: http://svn.osgeo.org/geotools/trunk/modules/plugin/property/src/main/java/org/geotools/data/property/PropertyAttributeWriter.java $
*/
public class PropertyAttributeWriter implements AttributeWriter {
BufferedWriter writer;
SimpleFeatureType type;
public PropertyAttributeWriter(File file, SimpleFeatureType featureType)
throws IOException {
writer = new BufferedWriter(new FileWriter(file));
type = featureType;
writer.write("_=");
writer.write(DataUtilities.spec(type));
}
public int getAttributeCount() {
return type.getAttributeCount();
}
public AttributeDescriptor getAttributeType(int index)
throws ArrayIndexOutOfBoundsException {
return type.getDescriptor(index);
}
A BufferedWriter is created over the provided File, and the provided featureType is used to implement getAttribtueCount() and getAttributeType( index ).
Add hasNext() and next() implementations:
public boolean hasNext() throws IOException {
return false;
}
public void next() throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
writer.newLine();
writer.flush();
}
Our FeatureWriter does not provide any content of its own. FeatureWriters that are backed by JDBC ResultSets or random access file may use hasNext() to indicate that they are streaming over existing result set.
Our implementation of next() will start a newLine for the feature that is about to be written.
Add writeFeatureID():
public void writeFeatureID(String fid) throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
writer.write(fid);
}
Our file format is capable of storing FeatureIDs. Many DataStores will need to derive or encode FeatureID information into their Attributes.
Add write( int index, Object value ):
public void write(int position, Object attribute) throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
writer.write(position == 0 ? "=" : "|");
if (attribute == null) {
writer.write("<null>"); // nothing!
} else if( attribute instanceof String){
String txt = encodeString((String) attribute);
writer.write( txt );
} else if (attribute instanceof Geometry) {
Geometry geometry = (Geometry) attribute;
String txt = geometry.toText();
txt = encodeString( txt );
writer.write( txt );
} else {
String txt = Converters.convert( attribute, String.class );
if( txt == null ){ // could not convert?
txt = attribute.toString();
}
txt = encodeString( txt );
writer.write( txt );
}
}
/**
* Used to encode common whitespace characters and | character for safe transport.
*
* @param txt
* @return txt encoded for storage
* @see PropertyAttributeReader#decodeString(String)
*/
String encodeString( String txt ){
// encode our escaped characters
// txt = txt.replace("\\", "\\\\");
txt = txt.replace("|","\\|");
// encode whitespace constants
txt = txt.replace("\n", "\\n");
txt = txt.replace("\r", "\\r");
return txt;
}
Our implementation needs to prepend an equals sign before the first Attribute, or a bar for any other attribute.
We also make sure to encode any newlines in String content, Geometry as wkt, and use the Converters class to handle any other objects correctly.
Add close():
public void close() throws IOException {
if (writer == null) {
throw new IOException("Writer has already been closed");
}
writer.close();
writer = null;
type = null;
}
Finally, to implement our FeatureWriter.close() optimization, we need to implement echoLine():
public void echoLine(String line) throws IOException {
if (writer == null) {
throw new IOException("Writer has been closed");
}
if (line == null) {
return;
}
writer.write(line);
}
In this part we will explore the full capabilities of our completed PropertyDataStore.
Now that we have completed our PropertyDataStore implementation, we can explore the remaining capabilities of the DataStore API.
PropertyDataStore API for data modification:
The DataStore.getFeatureSource( typeName ) method is the gateway to our high level api, as provided by an instance of FeatureSource, FeatureStore or FeatureLocking.
Now that we have implemented writing operations, the result of this method supports:
FeatureStore provides Transaction support and modification operations. FeatureStore is an extension of FeatureSource. You may check the result of getFeatureSource( typeName ) with the instanceof operator.
Example of FeatureStore use:
PropertyDataStore datastore = new PropertyDataStore(directory);
SimpleFeatureSource featureSource = datastore
.getFeatureSource("example");
if (!(featureSource instanceof SimpleFeatureStore)) {
throw new IllegalStateException("Modification not supported");
}
SimpleFeatureStore featureStore = (SimpleFeatureStore) featureSource;
FeatureStore defines:
Once again, many DataStores are able to provide optimised implementations of these operations.
Transaction Example:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
Transaction t1 = new DefaultTransaction("transaction 1");
Transaction t2 = new DefaultTransaction("transactoin 2");
SimpleFeatureType type = store.getSchema("example");
SimpleFeatureStore featureStore = (SimpleFeatureStore) store
.getFeatureSource("example");
SimpleFeatureStore featureStore1 = (SimpleFeatureStore) store
.getFeatureSource("example");
SimpleFeatureStore featureStore2 = (SimpleFeatureStore) store
.getFeatureSource("example");
featureStore1.setTransaction(t1);
featureStore2.setTransaction(t2);
System.out.println("Step 1");
System.out.println("------");
System.out.println("start auto-commit: "+DataUtilities.fidSet(featureStore.getFeatures()) );
System.out.println("start t1: "+DataUtilities.fidSet(featureStore1.getFeatures()) );
System.out.println("start t2: "+DataUtilities.fidSet(featureStore2.getFeatures()) );
// select feature to remove
FilterFactory ff = CommonFactoryFinder.getFilterFactory(null);
Filter filter1 = ff.id(Collections.singleton(ff.featureId("fid1")));
featureStore1.removeFeatures(filter1); // road1 removes fid1 on t1
System.out.println();
System.out.println("Step 2 transaction 1 removes feature 'fid1'");
System.out.println("------");
System.out.println("t1 remove auto-commit: "+DataUtilities.fidSet(featureStore.getFeatures()) );
System.out.println("t1 remove t1: "+DataUtilities.fidSet(featureStore1.getFeatures()) );
System.out.println("t1 remove t2: "+DataUtilities.fidSet(featureStore2.getFeatures()) );
// new feature to add!
SimpleFeature feature = SimpleFeatureBuilder.build(type, new Object[] {
5, "chris", null }, "fid5");
feature.getUserData().put(Hints.USE_PROVIDED_FID,true);
feature.getUserData().put(Hints.PROVIDED_FID, "fid5");
SimpleFeatureCollection collection = DataUtilities.collection(feature);
featureStore2.addFeatures(collection);
System.out.println();
System.out.println("Step 3 transaction 2 adds a new feature '"+feature.getID()+"'");
System.out.println("------");
System.out.println("t2 add auto-commit: "+DataUtilities.fidSet(featureStore.getFeatures()) );
System.out.println("t2 add t1: "+DataUtilities.fidSet(featureStore1.getFeatures()) );
System.out.println("t1 add t2: "+DataUtilities.fidSet(featureStore2.getFeatures()) );
// commit transaction one
t1.commit();
System.out.println();
System.out.println("Step 4 transaction 1 commits the removal of feature 'fid1'");
System.out.println("------");
System.out.println("t1 commit auto-commit: "+DataUtilities.fidSet(featureStore.getFeatures()) );
System.out.println("t1 commit t1: "+DataUtilities.fidSet(featureStore1.getFeatures()) );
System.out.println("t1 commit t2: "+DataUtilities.fidSet(featureStore2.getFeatures()) );
// commit transaction two
t2.commit();
System.out.println();
System.out.println("Step 5 transaction 2 commits the addition of '"+feature.getID()+"'");
System.out.println("------");
System.out.println("t2 commit auto-commit: "+DataUtilities.fidSet(featureStore.getFeatures()) );
System.out.println("t2 commit t1: "+DataUtilities.fidSet(featureStore1.getFeatures()) );
System.out.println("t2 commit t2: "+DataUtilities.fidSet(featureStore2.getFeatures()) );
t1.close();
t2.close();
store.dispose(); // clear out any listeners
This produces the following output:
Step 1 ------ start auto-commit: [fid3, fid4, fid1, fid2] start t1: [fid3, fid4, fid1, fid2] start t2: [fid3, fid4, fid1, fid2] Step 2 transaction 1 removes feature 'fid1' ------ t1 remove auto-commit: [fid3, fid4, fid1, fid2] t1 remove t1: [fid3, fid4, fid2] t1 remove t2: [fid3, fid4, fid1, fid2] Step 3 transaction 2 adds a new feature 'fid5' ------ t2 add auto-commit: [fid3, fid4, fid1, fid2] t2 add t1: [fid3, fid4, fid2] t1 add t2: [fid3, fid4, fid1, fid2, fid5] Step 4 transaction 1 commits the removal of feature 'fid1' ------ t1 commit auto-commit: [fid3, fid4, fid2] t1 commit t1: [fid3, fid4, fid2] t1 commit t2: [fid3, fid4, fid2, fid5] Step 5 transaction 2 commits the addition of 'fid5' ------ t2 commit auto-commit: [fid3, fid4, fid2, fid5] t2 commit t1: [fid3, fid4, fid2, fid5] t2 commit t2: [fid3, fid4, fid2, fid5]
Note
Please review the above code example carefully as it is the best explanation of transaction independence you will find.
Specifically:
“auto-commit” represents the current contents of the file on disk
Notice how the transactions only reflect the changes the user made relative to the current file contents.
This is shown after t1 commit, where transaction t2 is seeing 4 features (ie the current file contents plus the one feature that has been added on t2).
This really shows that FeatureSource and FeatureStore are “views” into your data.
FeatureLocking is the last extension to our high-level API. It provides support for preventing modifications to features for the duration of a Transaction, or a period of time.
FeatureLocking defines:
The concept of a FeatureLock matches the description provided in the OGC Web Feature Server Specification. Locked Features can only be used via Transactions that have been provided with the correct authorization.
We have a number of FeatureWriters available for different uses; these implementations are used by the default implementation of AbstractFeatureStore and AbstractFeatureLocking.
These classes serve as a good example of how to use FeatureWriter.
The DataStore.getFeatureWriter( typeName, filter, transaction ) method creates a FeatureWriter used to modify features indicated by a constraint.
Example - removing all features:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
Transaction t = new DefaultTransaction("transaction");
try {
FeatureWriter<SimpleFeatureType, SimpleFeature> writer = store
.getFeatureWriter("example", Filter.INCLUDE, t);
SimpleFeature feature;
try {
while (writer.hasNext()) {
feature = writer.next();
System.out.println("remove " + feature.getID());
writer.remove(); // marking contents for removal
}
} finally {
writer.close();
}
System.out.println("commit " + t); // now the contents are removed
t.commit();
} catch (Throwable eek) {
t.rollback();
} finally {
t.close();
store.dispose();
}
This FeatureWriter does not allow the addition of new content. It can be used for modification and removal only.
DataStores can often provide an optimized implementation.
The DataStore.getFeatureWriter( typeName, transaction ) method creates a general purpose FeatureWriter. New content may be added after iterating through the provided content.
Example - completely replace all features:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
final SimpleFeatureType type = store.getSchema("example");
final FeatureWriter<SimpleFeatureType, SimpleFeature> writer;
SimpleFeature f;
SimpleFeatureCollection collection = FeatureCollections.newCollection();
f = SimpleFeatureBuilder
.build(type, new Object[] { 1, "jody" }, "fid1");
collection.add(f);
f = SimpleFeatureBuilder.build(type, new Object[] { 2, "brent" },
"fid3");
collection.add(f);
f = SimpleFeatureBuilder
.build(type, new Object[] { 3, "dave" }, "fid3");
collection.add(f);
f = SimpleFeatureBuilder.build(type, new Object[] { 4, "justin" },
"fid4");
collection.add(f);
writer = store.getFeatureWriter("road", Transaction.AUTO_COMMIT);
try {
// remove all features
while (writer.hasNext()) {
writer.next();
writer.remove();
}
// copy new features in
SimpleFeatureIterator iterator = collection.features();
while (iterator.hasNext()) {
SimpleFeature feature = iterator.next();
SimpleFeature newFeature = writer.next(); // new blank feature
newFeature.setAttributes(feature.getAttributes());
writer.write();
}
} finally {
writer.close();
}
The DataStore.getFeatureWriterAppend( typeName, transaction ) method creates a FeatureWriter for adding content.
Example - making a copy:
Map<String, Serializable> params = new HashMap<String, Serializable>();
params.put("directory", directory);
DataStore store = DataStoreFinder.getDataStore(params);
FeatureReader<SimpleFeatureType,SimpleFeature> reader;
FeatureWriter<SimpleFeatureType,SimpleFeature> writer;
SimpleFeature feature, newFeature;
SimpleFeatureType type = store.getSchema( "example" );
SimpleFeatureType type2 = DataUtilities.createType( "duplicate", "id:Integer,geom:Geometry,name:String" );
Query query = new Query( type.getTypeName(), Filter.INCLUDE );
reader = store.getFeatureReader( query, Transaction.AUTO_COMMIT );
store.createSchema( type2 );
writer = store.getFeatureWriterAppend( "duplicate", Transaction.AUTO_COMMIT );
try {
while (reader.hasNext()) {
feature = reader.next();
newFeature = writer.next();
newFeature.setAttribute("id", feature.getAttribute("id"));
newFeature.setAttribute("geom", feature.getAttribute("geom") );
newFeature.setAttribute("name", feature.getAttribute("name"));
writer.write();
}
}
finally {
reader.close();
writer.close();
}
DataStores can often provide an optimised implementation of this method.
In this part we will explore several optimisation techniques using our PropertyDataStore.
AbstractDataStore provides a lot of functionality based on the five methods we implemented in the Tutorials. By examining its implementation we have an opportunity to discuss several issues with DataStore development. Please keep these issues in mind when applying your own DataStore optimisations.
In general the “Gang of Four” decorator pattern is used to layer functionality around the raw FeatureReader and FeatureWriters you provided. This is very similar to the design of the java-io library (where a BufferedInputStream can be wrapped around a raw FileInputStream).
From AbstractDataStore getFeatureReader( featureType, filter, transaction ):
Note
Historically Filter.ALL and Filter.NONE were used as placeholder, as crazy as it sounds, Filter.ALL filters out ALL (accepts none) Filter.NONE filters out NONE (accepts ALL)/
These two have been renamed in GeoTools 2.3 for the following:
Here is an example of how AbstractDataStore applies wrappers around your raw feature reader:
public FeatureReader<SimpleFeatureType, SimpleFeature> getFeatureReader(Query query,Transaction transaction) throws IOException {
Filter filter = query.getFilter();
String typeName = query.getTypeName();
String propertyNames[] = query.getPropertyNames();
....
if (filter == Filter.EXCLUDES) {
return new EmptyFeatureReader(featureType);
}
String typeName = featureType.getTypeName();
FeatureReader reader = getFeatureReader(typeName);
if (filter != Filter.INCLUDES) {
reader = new FilteringFeatureReader(reader, filter);
}
if (transaction != Transaction.AUTO_COMMIT) {
Map diff = state(transaction).diff(typeName);
reader = new DiffFeatureReader(reader, diff);
}
if (!featureType.equals(reader.getFeatureType())) {
reader = new ReTypeFeatureReader(reader, featureType);
}
return reader;
}
Support classes used:
From AbstractDataStore getFeatureWriter( typeName, filter, transaction):
public FeatureWriter getFeatureWriter(String typeName, Filter filter,
Transaction transaction) throws IOException {
if (filter == Filter.ALL) {
FeatureType featureType = getSchema(typeName);
return new EmptyFeatureWriter(featureType);
}
FeatureWriter writer;
if (transaction == Transaction.AUTO_COMMIT) {
writer = getFeatureWriter(typeName);
} else {
writer = state(transaction).writer(typeName);
}
if (lockingManager != null) {
writer = lockingManager.checkedWriter(writer, transaction);
}
if (filter != Filter.NONE) {
writer = new FilteringFeatureWriter(writer, filter);
}
return writer;
}
Support classes used:
Every helper class we discussed above can be replaced if your external data source supports the functionality.
All JDBC DataStores support the concept of Transactions natively. JDBDataStore supplies an example of using Transaction.State to store JDBC connection rather than the Difference map used above.:
public class JDBCTransactionState implements State {
private Connection connection;
public JDBCTransactionState( Connection c) throws IOException{
connection = c;
}
public Connection getConnection(){
return connection;
}
public void commit() throws IOException {
connection.commit();
}
public void rollback() throws IOException {
connection.rollback();
}
}
For the purpose of PropertyDataStore we could create a Transaction.State class that records a temporary File name used for a difference file. By externalising differences to a file rather than Memory we will be able to handle larger data sets; and recover changes in the event of an application crash.
Another realistic example is making use of Java Enterprise Edition session information allow “per user” edits.
Several DataStores have an environment that can support native locking. By replacing the use of the InProcessLockingManager we can make use of native Strong Transaction Support.
We have a total of three distinct uses for FeatureWriters:
AbstractDataStore.getFeatureWriter( typeName, transaction )
General purpose FeatureWriter
AbstractDataStore.getFeatureWriter( typeName, filter, transaction )
An optimised version that does not create new content can be created.
AbstractDataStore.getFeatureWriterAppend( typeName, transaction)
An optimised version that duplicates the origional file, and opens it in append mode can be created. We can also perform special tricks such as returning a Feature delegate to the user, which records when it has been modified.
DataStore, FeatureSource and FeatureStore provide a few methods specifically set up for optimisation.
AbstractDataStore leaves open a number of methods for high-level optmisations:
These methods are designed to allow you to easily report the contents of information that is often contained in your file header. Implementing them is optional, and each method provides a way for you to indicate if the information is unavilable.
By default the implementations returned are based on FeatureReader and FeatureWriter. Override this method to return your own subclasses that are tuned for your data format.
DataStores operating against rich external data sources can often perform high level optimisations. JDBCDataStores for instance can often construct SQL statements that completely fulfil a request without making use of FeatureWriters at all.
When performing these queries please remember two things:
A common optimisation is to trade memory use for speed by use of a cache. In this section we will present a simple cache for getBounds() and getCount(Query.ALL).
The best place to locate your cache is in your DataStore implementation, you will need to keep a separate cache for each Transaction by making use of Transaction.State. By implementing a cache in the DataStore all operations can benefit.
Another popular technique is to locate the cache in an instance of FeatureSource. While the cache will be duplicated when multiple FeatureStores are in use, it is convenient to locate the cache next to the high-level operations that can best benefit.
Finally FeatureResults represents a great opportunity to cache results, rather than reissue them repeatedly.
FeatureListener (and associated FeatureEvents) provides notification of modification which can be used to keep your cache implementation in sync with the DataStore.
We can fill in the following methods for PropertyDataStore:
getCount( Query )
We would like to improve this by recognizing the special case where the user has asked for the count of all of the features. In this case the number of Features is the same as the number of lines in the file (minus one for the header information and any comments).
Things to look out for when reviewing the code:
We can offer a simple optimisation by counting the number of lines in our file, when the Query requests all features (using Filter.INLCUDE):
int cacheCount;
long cacheTimestamp;
protected int getCount(Query query) throws IOException {
if (query.getFilter() == Filter.INCLUDE) {
String typeName = query.getTypeName();
File file = new File(directory, typeName + ".properties");
if (cacheCount != -1 && file.lastModified() == cacheTimestamp) {
return cacheCount;
}
cacheCount = PropertyDataStore.countFile(file);
cacheTimestamp = file.lastModified();
return cacheCount;
}
else {
return -1; // too expensive count the features
}
}
/** Used to carefully count the lines in the provided properties file */
static int countFile(File file) {
LineNumberReader reader = null;
try {
int skip=1; // header
reader = new LineNumberReader(new FileReader(file));
String line;
while ((line = reader.readLine()) != null){
if( line.startsWith("#")||line.startsWith("!")){
skip++; // comment
}
if( line.endsWith("\\") ){
skip++; // multiline
}
}
return reader.getLineNumber() - skip;
} catch (IOException e) {
return -1; // could not quickly determine length
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException e) {
// we tried
}
}
}
}
getBounds( Query )
Our file format does not offer an easy way to sort out the bounds (spatial file formats often include this information in the header). As such we won’t be implementing getBounds()
protected ReferencedEnvelope getBounds(Query query) throws IOException {
return null; // to expensive - calculate by visiting all the features
}
getFeatureSource( typeName )
We will be returning the following classes (which we will create in the next section).
Here is what that looks like:
public SimpleFeatureSource getFeatureSource(final String typeName)
throws IOException {
File file = new File( this.directory, typeName+".properties");
if( !file.exists()){
throw new FileNotFoundException( file.getAbsolutePath() );
}
if( file.canWrite() ){
return new PropertyFeatureStore(this, typeName);
//return new PropertyFeatureLocking(this, typeName);
}
else {
return new PropertyFeatureSource(this, typeName);
}
//return super.getFeatureSource(typeName);
}
For a writable file we extend AbstractFeatureLocking which supports thread-safe Locking, and provides the correct hooks into the AbstractDataStore listenerManager.
To implement a caching example we are going to produce our own implementation of FeatureSource:
Create the file PropertyFeatureSource:
package org.geotools.data.property;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.LineNumberReader;
import java.util.Set;
import org.geotools.data.AbstractFeatureSource;
import org.geotools.data.DataStore;
import org.geotools.data.FeatureListener;
import org.geotools.data.Query;
import org.geotools.data.QueryCapabilities;
import org.geotools.data.Transaction;
import org.geotools.geometry.jts.ReferencedEnvelope;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.filter.Filter;
/**
*
*
* @source $URL: http://svn.osgeo.org/geotools/trunk/modules/plugin/property/src/main/java/org/geotools/data/property/PropertyFeatureSource.java $
*/
public class PropertyFeatureSource extends AbstractFeatureSource {
String typeName;
SimpleFeatureType featureType;
PropertyDataStore store;
long cacheTimestamp = 0;
ReferencedEnvelope cacheBounds = null;
PropertyFeatureSource(PropertyDataStore propertyDataStore, String typeName)
throws IOException {
this.store = propertyDataStore;
this.typeName = typeName;
this.featureType = store.getSchema(typeName);
this.queryCapabilities = new QueryCapabilities() {
public boolean isUseProvidedFIDSupported() {
return true;
}
};
}
public PropertyDataStore getDataStore() {
return store;
}
public void addFeatureListener(FeatureListener listener) {
store.listenerManager.addFeatureListener(this, listener);
}
public void removeFeatureListener(FeatureListener listener) {
store.listenerManager.removeFeatureListener(this, listener);
}
public SimpleFeatureType getSchema() {
return featureType;
}
We are extending AbstractFeatureSource here, as such we not need to implement FeatureSource.getCount( query ) as the default implementation will call up to PropertyDataStore.getCount( query ) implemented earlier.
We can however generate the bounds information and cache the result.
Once again we are using a timestamp of the file to notice if the file is changed on disk.
public ReferencedEnvelope getBounds() {
File file = new File(store.directory, typeName + ".properties");
if (cacheBounds != null && file.lastModified() == cacheTimestamp) {
// we have the cache
return cacheBounds;
}
try {
// calculate and store in cache
cacheBounds = getFeatures().getBounds();
cacheTimestamp = file.lastModified();
return cacheBounds;
} catch (IOException e) {
}
// bounds are unavailable!
return null;
}
Earlier we modified PropertyDataStore to create an instance of this class if the file was read-only.
We are going to perform a similar set of optimisations to PropertyFeatureStore; with the added wrinkle of listening to feature events so we can update our cached values in the event modifications are made.
Create PropertyFeatureStore as follows:
package org.geotools.data.property;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.LineNumberReader;
import org.geotools.data.AbstractFeatureLocking;
import org.geotools.data.DataStore;
import org.geotools.data.FeatureEvent;
import org.geotools.data.FeatureListener;
import org.geotools.data.Query;
import org.geotools.data.QueryCapabilities;
import org.geotools.data.Transaction;
import org.geotools.data.simple.SimpleFeatureCollection;
import org.geotools.data.simple.SimpleFeatureIterator;
import org.geotools.feature.FeatureCollection;
import org.geotools.geometry.jts.ReferencedEnvelope;
import org.opengis.feature.simple.SimpleFeature;
import org.opengis.feature.simple.SimpleFeatureType;
import org.opengis.filter.Filter;
/**
* Implementation used for writeable property files.
* Supports limited caching of number of features and bounds.
*
*
* @source $URL: http://svn.osgeo.org/geotools/trunk/modules/plugin/property/src/main/java/org/geotools/data/property/PropertyFeatureStore.java $
*/
public class PropertyFeatureStore extends AbstractFeatureLocking {
String typeName;
SimpleFeatureType featureType;
PropertyDataStore store;
long cacheTimestamp = 0;
ReferencedEnvelope cacheBounds = null;
int cacheCount = -1;
FeatureListener watcher = new FeatureListener() {
public void changed(FeatureEvent featureEvent) {
if (cacheBounds != null) {
if (featureEvent.getType() == FeatureEvent.Type.ADDED) {
cacheBounds.expandToInclude(featureEvent.getBounds());
} else {
cacheBounds = null;
}
}
cacheCount = -1;
}
};
PropertyFeatureStore( PropertyDataStore propertyDataStore, String typeName ) throws IOException{
this.store = propertyDataStore;
this.typeName = typeName;
this.featureType = store.getSchema( typeName );
store.listenerManager.addFeatureListener( this, watcher);
this.queryCapabilities = new QueryCapabilities() {
public boolean isUseProvidedFIDSupported() {
return true;
}
};
}
FeatureEvent provides a bounding box which we can use to selectively invalidate cacheBounds
Yes it is a little awkward not being able to smoothly extend PropertyFeatureSource (this is one of the fixes we have addressed for ContentDataStore covered in the next tutorial).
public PropertyDataStore getDataStore() {
return store;
}
public void addFeatureListener(FeatureListener listener) {
store.listenerManager.addFeatureListener(this, listener);
}
public void removeFeatureListener(
FeatureListener listener) {
store.listenerManager.removeFeatureListener(this, listener);
}
public SimpleFeatureType getSchema() {
return featureType;
}
This time we can implement getCount( query ) locally; being sure to check both the filter (includes all features) and the transaction (auto_commit):
public int getCount(Query query) throws IOException {
if( Filter.INCLUDE == query.getFilter() && getTransaction() == Transaction.AUTO_COMMIT ){
File file = new File( store.directory, typeName+".properties" );
if( cacheCount != -1 && file.lastModified() == cacheTimestamp){
return cacheCount;
}
cacheCount = PropertyDataStore.countFile(file);
cacheTimestamp = file.lastModified();
return cacheCount;
}
return -1;
// return super.getCount(query); // super class checks transaction state diff
}
In a similar fashion getBounds( query ) can be generated and cached:
public ReferencedEnvelope getBounds() {
File file = new File( store.directory, typeName+".properties" );
if( cacheBounds != null && file.lastModified() == cacheTimestamp ){
// we have the cache
return cacheBounds;
}
try {
// calculate and store in cache
cacheBounds = getBoundsInternal(Query.ALL);
cacheTimestamp = file.lastModified();
return cacheBounds;
} catch (IOException e) {
}
// bounds are unavailable!
return null;
}
We have already modified PropertyDataStore to return an instance of this class if the file was writable.
Since this tutorial has been written the gt-property module has been pressed into service as a supported module in its own right.
References:
To get an idea of what kind of “extra” is required for a supported module:
A bug report asking that gt-properties support multi-line string values.
The suggestion was to use the Properties class, which we could not do and still retain our idea of streaming the content from disk.
The following change was made To PropertyAttributeReader allow for multi-line entries:
/** * Check if the file has another line. * * @return <code>true</code> if the file has another line * @throws IOException */ public boolean hasNext() throws IOException { if (next != null) { return true; } next = readLine(); return next != null; } String readLine() throws IOException { StringBuilder buffer = new StringBuilder(); while( true ){ String txt = reader.readLine(); if( txt == null ){ break; } // skip comments if( txt.startsWith("#") || txt.startsWith("!")){ continue; } // ignore leading white space txt = trimLeft( txt ); // handle escaped line feeds used to span multiple lines if( txt.endsWith("\\")){ buffer.append(txt.substring(0,txt.length()-1) ); buffer.append("\n"); continue; } else { buffer.append(txt); break; } } if( buffer.length() == 0 ){ return null; // there is no line } String raw = buffer.toString(); // String line = decodeString(raw); // return line; return raw; } /** * Used to decode common whitespace chracters and escaped | characters. * * @param txt Origional raw text as stored * @return decoded text as provided for storage * @see PropertyAttributeWriter#encodeString(String) */ String decodeString( String txt ){ // unpack whitespace constants txt = txt.replace( "\\n", "\n"); txt = txt.replaceAll("\\r", "\r" ); // unpack our our escaped characters txt = txt.replace("\\|", "|" ); // txt = txt.replace("\\\\", "\\" ); return txt; } /** * Trim leading white space as described Properties file standard. * @see Properties#load(java.io.Reader) * @param txt * @return txt leading whitespace removed */ String trimLeft( String txt ){ // trim int start = 0; WHITESPACE: for( int i=0; i < txt.length(); i++ ){ char ch = txt.charAt(i); if( Character.isWhitespace(ch)){ continue; } else { start = i; break WHITESPACE; } } return txt.substring(start); }
References:
Another reported issue. You can fill in a “info” data strucutre to more accurately describe your information to the uDig or GeoServer catalog.
public ServiceInfo getInfo() { DefaultServiceInfo info = new DefaultServiceInfo(); info.setDescription("Features from Directory " + directory); info.setSchema(FeatureTypes.DEFAULT_NAMESPACE); info.setSource(directory.toURI()); try { info.setPublisher(new URI(System.getProperty("user.name"))); } catch (URISyntaxException e) { } return info; }