Data Gathering

Goals and Objectives:

The goal of this lab is to learn how to find and download data from a variety of sources online, manipulate and join it using ArcGIS, project it into one coordinate system, and build and design a geodatabase to store the data.  It is also intended to provide more exposure to python coding in order to automate geoprocessing of data.

Methods:

The basic workflow of downloading each data set for this lab began by downloading zip files into a temporary directory so the large files can be easily deleted later.  They were then extracted into a working folder.  The data sets were projected and clipped/extracted and then loaded into a geodatabase.

The process began by finding and downloading several data sets.  The first one was the Polyline Railway Network file from the US Department of Transportation.  Next, the land cover for the state of Wisconsin (NLCD 2011 Land Cover) was downloaded via the USGS National Map Viewer.  The same website was used to download the elevation data sets (1/3 arc-second DEM).  This required downloading both n44w092_13 and n45w092_13 in order to get data for all of Trempealeau County. From the USDA Geospatial Data Gateway, the Wisconsin Cropland Land Cover was downloaded. From the Trempealeau County Land Records division website, the entire Trempealeau County file geodatabase was downloaded.  Finally, from the USDA NRCS Web Soil Survey website, the Trempealeau County soils data was downloaded.

The soils data was downloaded as a Microsoft Access geodatabase.  In order to be able to properly use this data, it was manipulated with Microsoft Access by importing the .txt files into the geodatabase schema.  This connected the actual data to the geodatabase template file.  In ArcCatalog, the soils shapefile was then imported into the TMP geodatabase. The component table was also imported into the TMP geodatabase. A relationship class was created to join the component information to the new soils feature class. Both of these were added to ArcMap and joined based on the relationship class.  The NTAD rail lines shapefile was added to the map, clipped to the Trempealeau County boundary, and added to the TMP geodatabase.  This projected the data into the proper coordinate system. The DEMs were then added to the map and combined using the Mosaic to New Raster tool.

The 3 raster datasets that were downloaded, along with the TMP geodatabase were moved to their own separate folder.  Then Pyscripter was used to create a python script that projected the rasters, extracted them to the TMP county boundary, and loaded the .tifs into the geodatabase. The code that was created to do this can be seen in the Python Scripts post below.  The results of this script can be seen in Figure 1 below.

Figure 1: The resulting layers that were created from using Python to project and extract the 3 raster datasets.
Legends for Land Cover and Crop Cover were omitted intentionally due to lack of space available. Information about the significance of these values can be found at the source websites linked above. 

Data Accuracy:

Figure 2:  Analysis of the metadata for each dataset downloaded for this lab.

Conclusions:

In general, the data seemed to come from reputable sources.  It will be important to keep in mind the scale when conducting further analyses in order to preserve data integrity.  Many of the datasets only provided a few of the data quality parameters so the others were estimated based on what was given. This could be a potential concern relating to the datasets used.  It was especially a concern when the scale was estimated based on the resolution values. While the sources seemed reputable, the often incomplete metadata leaves a level of uncertainty when it comes to the quality of the data. It was especially concerning when metadata did not include information about the various different accuracy levels, or the accuracy levels were left to be inferred by other information.  The USDA soils dataset seemed to have the most complete metadata available.  It was difficult to evaluate the TMP geodatabase because there was very little metadata for the geodatabase as a whole.  The majority of information available is for individual feature classes and many of these feature classes have very different metadata associated with them.  Overall, it would be a lot better if the metadata were to explicitly state these data quality measures instead of relying on users inferring and estimating them.  This would improve confidence levels greatly. 

Python Scripting

Python scripting is used to automate GIS processes. It is done in order to be able to process large amounts of data more efficiently to reduce overhead.  Once proficient in the scripting language, one can execute processes much faster than running them all manually.

#-------------------------------------------------------------------------------
# Name:        Exercise 5: Data Gathering
#
# Author:      Kevin Trushenski
#
# Created:     08/03/2017
#
# Purpose: To learn how to write a python script to project, clip, and load data into a geodatabase.
#-------------------------------------------------------------------------------

#import python module and spatial analyst

import arcpy
from arcpy import env
from arcpy.sa import *

#check out the spatial analyst extension
arcpy.CheckOutExtension("spatial")

#set environment settings
arcpy.env.workspace = "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\Ex5\Data"
arcpy.env.overwriteOutput = True
print "{}" .format(env.workspace)

#get list of rasters from workspace
rs_list = arcpy.ListRasters()
for raster in rs_list:
    print(raster)


#loop through the rasters in the loop
for raster in rs_list:
    #define the outputs
    rasterOut = "{}_Out.tif".format(raster)
    rasterExtract = "{}_Extract.tif".format(raster)

    #project the rasters
    arcpy.ProjectRaster_management(raster, rasterOut, "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\Ex5\Data\TrempWebDATA.gdb\Boundaries\County_Boundary")

    #extract the raster and copy the raster into the geodatabase
    outExtractByMask = ExtractByMask(rasterOut, "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\Ex5\Data\TrempWebDATA.gdb\Boundaries\County_Boundary")
    outExtractByMask.save(rasterExtract)
    arcpy.RasterToGeodatabase_conversion(rasterExtract, "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\Ex5\Data\TrempWebDATA.gdb")
    print "Raster to Geodatabase Conversion {} Successful".format(rasterExtract)


print "The script is complete"



#-------------------------------------------------------------------------------
# Name:      Ex 7 Network Analysis
# Purpose: To create a script that will select active mines that don't have a rail loading station on-site.  It also eliminates mines within 1.5 km of a rail because it is likely a rail spur has already been added.
#
# Author:      Kevin Trushenski
#
# Created:     10/04/2017
#-------------------------------------------------------------------------------

#import system modules
import arcpy
#set environments
from arcpy import env
env.workspace = "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\ex7\ex7.gdb"
arcpy.env.overwriteOutput = "true"

#Set variables

all = "all_mines"
active= "active_mines"
mines = "Status_mine"
norail = "mines_norail"
wi= "wi"
rail = "rails_wtm"
worail = "mines_norail_final"

#Set up the field delimiters for the SQL statements

field1= arcpy.AddFieldDelimiters(all, "Site_Statu")
field2= arcpy.AddFieldDelimiters(all, "Facility_T")

#SQL statement to select active mines

activeSQL = field1 + "=" + "'Active'"

#SQL statement for field facility_T LIKE mine

mineSQL = field2 + "LIKE" + "'%Mine%'"

#SQL statement for field facility_T NOT rail

norailSQL = "NOT" + field2 + "LIKE" + "'%Rail%'"

#Make a Layer from the feature class with mine status = active

arcpy.MakeFeatureLayer_management(all, active, activeSQL)

#Make a layer from the feature class with facility type = mine

arcpy.MakeFeatureLayer_management(active, mines, mineSQL)

#Make a layer from the feature class without rails

arcpy.MakeFeatureLayer_management(mines, norail, norailSQL)

#Select

arcpy.SelectLayerByLocation_management(norail, "INTERSECT", wi)
arcpy.SelectLayerByLocation_management(norail, "WITHIN_A_DISTANCE", rail, "1.5 KILOMETER", "REMOVE_FROM_SELECTION")

arcpy.CopyFeatures_management(norail, worail)


print "The script is complete."


#-------------------------------------------------------------------------------
# Name:  Exercise 8: Raster Analysis Python
# Purpose: To write a python script to generate a weighted index model in order to place more emphasis on a certain variable in the Risk Model from Exercise 8
#
# Author:      Kevin L Trushenski
#
# Created:     16/05/2017
#-------------------------------------------------------------------------------
#import system modules
import arcpy

#set environments
from arcpy import env
env.workspace = "Q:\StudentCoursework\CHupy\GEOG.337.001.2175\TRUSHEKL\Ex5\Data\TrempWebDATA.gdb"
arcpy.env.overwriteOutput = "true"
arcpy.CheckOutExtension("Spatial")

#Create variables
Parks= arcpy.Raster("parksReclass")
Res= arcpy.Raster("res_reclass")
River= arcpy.Raster("river_clip_reclass")
Schools= arcpy.Raster("schoolReclass")
Farm= arcpy.Raster("farm_reclass")

#Weight most important variable
outweight= (Res*1.5)

#Set up equation for weighted index model
Weighted= (outweight + Parks + River +Schools + Farm)

#Save to geodatabase
Weighted.save("weighted_result")

print "The Script is Complete."

Sand Mining in Western Wisconsin Overview

Frac sand mining is an industry that has been present in Wisconsin for over 100 years.  There is a huge demand for the sand because it is used for hydraulic fracturing, or "fracking", as well as some manufacturing.   The sand that is sought after is quartz and has to have a very specific grain size and shape.  According to the Wisconsin Geological and Natural History Survey, frack sand must be "nearly pure quartz, very well rounded, extremely hard, and of uniform size". 

The sand can be found in certain sandstone formations in western and central Wisconsin (see Figure 1).

Figure 1: Locations of frac sand mining sites in Wisconsin,
Courtesy of the Wisconsin Geological and natural history Survey

Frac sand mining has been very controversial due to the implications that are associated with it.  In western Wisconsin, there are issues surrounding what mining will do to the natural environment, as well as to people that live near the sites.  The Wisconsin DNR has had to invoke several regulations to ensure the protection of natural resources while still allowing the sand mining industry to flourish.  According to the DNR, "Industrial sand mines and other related operations must follow the same state requirements to protect public health and the environment as other nonmetallic mining operations in Wisconsin".  They go on to note that the permits for sand mining carry regulations pertaining to storm water, air quality, wetlands, high capacity wells, solid/hazardous waste, drinking water, and endangered/threatened species.  

Whenever a new mine is planned to be opened, there is often a lot of controversy surrounding it, especially for the community in which the mine will be near.  One recent example of this is with the controversial sand mining company Pattison Sand Co.  They plan to expand their underground mining operation in Clayton County, Iowa.  According to an article in Urban Milwaukee from May, 2016, since 2005 this company has "racked up more workplace violations than any other industrial sand mine in the United States".

One tool that can assist in exploring the issues surrounding sand mining is Geographic Information Systems (GIS). Through using this tool, one can analyze the spatial trends associated with fracking.  This will be helpful when examining the regulations, as well as being able to look at geological information. It can be a tool to help give a visual perspective to some of these issues in order to potentially help find a solution.

Works Cited:

http://dnr.wi.gov/topic/Mines/Sand.html

http://urbanmilwaukee.com/2016/05/08/frac-sand-mine-is-nations-worst-violator/

http://wcwrpc.org/frac-sand-factsheet.pdf