Thursday, August 10, 2017

Using SDE Feature Classes Inside a Python Geoprocessing Service

I finally had a breakthrough today at work on a problem I've been working on off-and-on for a while now at work, and I want to share it with you (and write it down so I remember it...).

Like a lot of places, we use a backend SDE to hold the data that we work on internally and expose publicly through map services. I've been trying to create a Geoprocessing Service based on a  Python script tool to do some things we need done. Within this script, I directly access the parcels feature class in our SDE that has tens of thousands of features.

The script tool runs just fine from within ArcMap, with the path to the feature class either hardcoded in or provided as a tool parameter:

parcels_fc = "Database Connections\\connection.sde\\Cadastral\\parcels"
or
parcels_fc = arcpy.GetParameterAsText[4]

The Problem
However, sharing the results to our ArcGIS Server as a GP service never worked out. If I hardcoded the path, it would upload but the service would always fail when I'd try to run it in a web app. If I got it through a parameter and selected "Constant Value" for that parameter when sharing the service, the server would say that it couldn't find the data in its data store. 

I tried all sorts of different permutations to try to get this to work, to get the server to recognize that it had the SDE connections registered in its data store, but to no effect. A "Failed to consolidate data" error was common when trying to stage the service prior to uploading to the server.

Solution
Ok, enough background, what fixed it? The key was to put a copy of the SDE connection file in a place where it would get copied into the server's data store and to use a relative path to access it from within the script. 

First, I created a folder in the same directory as the script (which we'll call "path" for this example) and copied the SDE connection file into it. Next, I used the sample code from the Esri documentation (scroll down to "Relative Paths to Folders") to create something like this:
script_path = sys.path[0] # where the script lives
folder_path = os.path.join(script_path, "path")
parcels_fc = os.path.join(folder_path, "Cadastral\\parcels")

In the script tool properties, make sure "Store absolute paths as relative paths" is checked.

And voila. When I share the results as a GP service, the server picks up the "path" folder and copies it to its data store along with the connection file inside of it. The script knows to look for the connection file in the folder "path" that resides in the same directory as the script, then uses that connection to access the SDE and do whatever magic it is that you've told it to do. 

Resources:
https://gis.stackexchange.com/questions/10503/creating-a-gp-script-to-update-a-table-in-sde
https://gis.stackexchange.com/search?q=geoprocessing+sde