Register ArcSDE Layers to GeoServer using ArcGIS 10

Recently i gave the geoserver REST-API a try to manage the layer registration process directly from ArcGIS 10…

SDE-layers&styles -> geoserver

My situation:

-Debian 6 64bit
-Postgres 8.4 (without PostGIS)
-ArcSDE 10 for Postgres 8.4
-GeoServer 2.1.2 with SDE-extension
-one vector store for SDE
-SDE Raster and Vector all managed from ArcGIS 10 client

My needs:

  1. register SDE raster&vector to geoserver using arcpy and ArcGIS toolbox capabilities
  2. assign Styles to newly registered Geoserver-Layers using arcpy

My solution

for need (1) I implemented two simple python scripts… note that only input datasets with epsg:4326 are supported… In geoserver all registerd datasets are published as epsg:900913–you can change this setting, to fit to your needs… (add some tool-param for proj inside the xml or json body)

one for vector:

#Register SDE Vector Layer to GeoServer
#author: Hubert Asamer 2012
import arcpy, os, sys, string
import subprocess

inputFeatures = string.split(string.replace(arcpy.GetParameterAsText(0),"\'",""), ";")
sld_file = arcpy.GetParameterAsText(1)
gs_host = arcpy.GetParameterAsText(2)
gs_user = arcpy.GetParameterAsText(3)
gs_pw = arcpy.GetParameterAsText(4)
gs_workspace = arcpy.GetParameterAsText(5)
gs_store =arcpy.GetParameterAsText(6)

for name in inputFeatures:
featureClassName = name.upper()
arcpy.AddMessage(featureClassName)
p = subprocess.Popen(['curl',
'-XPOST','-u',''+gs_user+':'+gs_pw+'', '-d',
'{"featureType":{"name":"'+featureClassName+'","nativeCRS":"EPSG:4326","srs":"EPSG:900913","projectionPolicy":"REPROJECT_TO_DECLARED"}}',
'-H', 'Content-Type:application/json',
'http://'+gs_host+'/rest/workspaces/'+gs_workspace+'/datastores/'+gs_store+'/featuretypes'],
shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
arcpy.AddMessage(line),
retval = p.wait()

one for raster: note that there will be created a store called as the input-grid and a corresponding layer

#Register SDE Raster Layer to GeoServer
#author: Hubert Asamer 2012
import arcpy, os, sys, string
import subprocess

inputFeatures = string.split(string.replace(arcpy.GetParameterAsText(0),"\'",""), ";")
sld_file = arcpy.GetParameterAsText(1)
gs_host = arcpy.GetParameterAsText(2)
gs_user = arcpy.GetParameterAsText(3)
gs_pw = arcpy.GetParameterAsText(4)
gs_workspace = arcpy.GetParameterAsText(5)
sdeusername=arcpy.GetParameterAsText(6)
sdepassword=arcpy.GetParameterAsText(7)
sdehost=arcpy.GetParameterAsText(8)
sdeport=arcpy.GetParameterAsText(9)

for name in inputFeatures:
featureClassName = name.upper()
arcpy.AddMessage(featureClassName)
p = subprocess.Popen(['curl', '-XPOST', '-u', ''+gs_user+':'+gs_pw+'', '-H', 'Content-Type:text/xml', '-d', '<coverageStore><name>'+featureClassName+'</name><type>ArcSDE Raster</type><enabled>true</enabled><url>sde://'+sdeusername+':'+sdepassword+'@'+sdehost+':'+sdeport+'/#'+featureClassName+'</url></coverageStore>', 'http://'+gs_host+'/rest/workspaces/'+gs_workspace+'/coveragestores'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
arcpy.AddMessage(line),
retval = p.wait()
q = subprocess.Popen(['curl', '-XPOST', '-u', ''+gs_user+':'+gs_pw+'', '-H', 'Content-Type:text/xml', '-d', '<coverage><name>'+featureClassName+'</name><nativeFormat>ArcSDE Raster</nativeFormat><nativeCRS>EPSG:4326</nativeCRS><srs>EPSG:900913</srs><parameters><entry><string>OVERVIEW_POLICY</string><string>QUALITY</string></entry></parameters></coverage>', 'http://'+gs_host+'/rest/workspaces/'+gs_workspace+'/coveragestores/'+featureClassName+'/coverages'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line1 in q.stdout.readlines():
arcpy.AddMessage(line1),
retval = q.wait()

for need (2) check this out:

#Register SLD to GeoServer-Layer
#author: Hubert Asamer 2012
import arcpy, os, sys, string
import subprocess

inputFeatures = string.split(string.replace(arcpy.GetParameterAsText(0),"\'",""), ";")
sld_file = os.path.abspath(arcpy.GetParameterAsText(1))
gs_host = arcpy.GetParameterAsText(2)
gs_user = arcpy.GetParameterAsText(3)
gs_pw = arcpy.GetParameterAsText(4)
gs_workspace = arcpy.GetParameterAsText(5)

for name in inputFeatures:
featureClassName = name.upper()
arcpy.AddMessage(featureClassName)
arcpy.AddMessage(sld_file)
p = subprocess.Popen(['curl','-XPOST','-u',''+gs_user+':'+gs_pw+'','-d','<style><name>'+featureClassName+'</name><filename>'+featureClassName+''+'.'+'sld</filename></style> ','-H','Content-Type: text/xml',
'http://'+gs_host+'/rest/styles'], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
arcpy.AddMessage(line),
retval = p.wait()
arcpy.AddMessage('Style created!')
q = subprocess.Popen(['curl', '-XPUT', '-u', ''+gs_user+':'+gs_pw+'', '-d', '@'+sld_file, '-H', 'Content-Type:application/vnd.ogc.sld+xml', 'http://'+gs_host+'/rest/styles/'+string.replace(featureClassName,".","%2E")+''], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line1 in q.stdout.readlines():
arcpy.AddMessage(line1),
retval = q.wait()
arcpy.AddMessage('SLD-File uploaded!')
r = subprocess.Popen(['curl','-XPUT','-u',''+gs_user+':'+gs_pw+'','-d','<layer><defaultStyle><name>'+featureClassName+'</name></defaultStyle><enabled>true</enabled></layer> ','-H','Content-Type: text/xml',
'http://'+gs_host+'/rest/layers/'+gs_workspace+':'+string.replace(featureClassName,".","%2E")], shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line2 in r.stdout.readlines():
arcpy.AddMessage(line2),
retval = r.wait()
arcpy.AddMessage('Style linked with layer!')

here my tbx-files (ArcGIS 10) and python code all in one file…

Note that you can produce your sld-files with any tool you like…it must be valid and all attribute matches inside the sld must be correct… I prefer arc2earth community edition to produce sld’s for single vector layers… Also qgis 1.8 offers sld import/export…

GeoServer REST API vs. SDE-Layers

Yesterday i gave the GeoServer REST API a try to register approx. 250 Raster and vector layers all coming from an SDE-Database. My debian6 server hosts postgres 8.4 with SDE 10.0 and Geoserver 2.1.2 (tomcat6)…Previously I installed the sde- extension for geoserver

From my experience the most important issue on publishing hundreds of layers from an GDB is a 100 percent consistent naming convention. Note that in ArcGIS SDE-GDB the cool feature-dataset structure is not represented through the JAVA sde-driver, meaning that the ability to group layers must be possible trough their names…

My aim was to do all basic geoserver-gui tasks in a simple batch file from my win7 client:

  • create raster stores
  • register raster layers
  • register vector layers from existing store

The geoserver doc has some examples on how to interact with the REST API. I also found this project site (opengeoportal wiki)  very useful. My example uses curl…

Raster-Registration based on textfile containing sde layer names (newLayerListRaster.txt):

Result is:

  • raster store (as named in textfile)
  • raster layer with default style (as named in textfile)
@echo off
 SET username="????"
 SET password="????"
 SET sdeusername="????"
 SET sdepassword="????"
 SET sdehost="127.0.0.1"
 SET sdeport="????"
 SET geoserverhostandpath="????/geoserver"
 SET workspace="sde"
 SET list=%1
 echo ---GeoServer Raster Registration ArcSDE---
 echo REQUEST...
 echo arg1=NewStoreName: --from list--
 echo arg2=SDE-RasterTableName(DB.SCHEMA.TABLENAME): --from list--
 echo.
 echo RESPONSE...
 FOR /F %%i in (%list%) do curl -XPOST -u %username%:%password% -H "Content-Type:text/xml" -d "<coverageStore><name>%%i</name><type>ArcSDE Raster</type><enabled>true</enabled><url>sde://%sdeusername%:%sdepassword%@%sdehost%:%sdeport%/#%%i</url></coverageStore>" http://%geoserverhostandpath%/rest/workspaces/%workspace%/coveragestores
 FOR /F %%i in (%list%) do curl -XPOST -u %username%:%password% -H "Content-Type:text/xml" -d "<coverage><name>%%i</name><nativeFormat>ArcSDE Raster</nativeFormat><requestSRS><string>EPSG:4326</string></requestSRS><responseSRS><string>EPSG:4326</string></responseSRS><parameters><entry><string>OVERVIEW_POLICY</string><string>QUALITY</string></entry></parameters></coverage>" http://%geoserverhostandpath%/rest/workspaces/%workspace%/coveragestores/%%i/coverages
 echo.

 

Vector Registration (store must exist):

Result is:

  • vector layer (as named in textfile)
@echo off
 SET username=????
 SET password=????
 SET geoserverhostandpath=????/geoserver
 SET workspacename=????
 SET datastoreName=????
 SET list=%1
 echo ---GeoServer Vector-Layer Registration in existing Store for ArcSDE ---
 echo REQUEST...
 echo arg1=SDE-VectorTableName(DB.SCHEMA.TABLENAME): --from list--
 echo arg2=newStyleName: --from list--
 echo.
 echo RESPONSE...
 @echo on
 FOR /F %%i in (%list%) do curl -XPOST -u %username%:%password% -d {"featureType":{"name":"%%i"}} -H "Content-Type:application/json" http://%geoserverhostandpath%/rest/workspaces/%workspacename%/datastores/%datastoreName%/featuretypes
 @echo off
 echo.
 rem curl -XPOST -u %username%:%password% -H "Content-type: text/xml" -d "<style><name>%stylename%</name><filename>%styleName%.sld</filename></style>" http://%geoserverhostandpath%/rest/styles
 echo.

These two examples can easily be adopted to define projection settings and other layer settings. Just edit the xml or json part of the XPOST request. A good starting point is to define a layer fully by hand in the geoserver gui and look into the resulting layer.xml or coverage.xml in the geoserver data dir. The structure of these xml’s is quite straightforward. Then try to place the tags you want into the batch request…

The next thing i want to automate is style handling and assigning specific styles to layers… One approach is to extend the layerlist from our batch routine with a second or third attribute delimited by e.g. semicolon, holding style info…

Single commands: create style > upload sld file for style > assign style to layer

  • curl -u admin:geoserver -XPOST -H 'Content-type: text/xml' -d '<style><name>newStyleName</name><filename>newStyleName.sld</filename></style>' http://?????/geoserver/rest/styles
    
  • curl -u admin:geoserver -XPUT -H 'Content-type: application/vnd.ogc.sld+xml'-d '@newStyleName.sld' http://?????/geoserver/rest/styles/schwemmland
    
  • curl -u admin:geoserver -XPUT -H 'Content-type: text/xml' -d '<layer><defaultStyle><name>newStyleName</name></defaultStyle></layer>' http://?????/geoserver/rest/layers/LayerName
    

Conclusion:

Depending on the amount of layers the REST API can save a lot of time.