Python field calculate sequence

This is a quick snippet that creates sequential numbers for unique values.
Note that it does require that the shapefile/feature class to be hard sorted on the value in question.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
counter=2
oldval=''
def foo(newval):
 global counter
 global oldval
 if oldval=='':
    oldval=newval
 if newval != oldval:
    counter=counter+1
    oldval=newval
 return counter

Goo = 
foo(!TLID!)

Positive lookahead assertion

I don’t claim to know regular expressions all that well, but I do love them for their power and simplicity.

This stack overflow answer to “How to remove duplicate rows in Notepad++” really caught my eye this morning.
It’s one way to replace duplicate rows in a text file in Notepad++; note that the top answer to the link utilizes the TextFX extension to Notepad++, which I love as well, but regular expressions can be used in a multitude of platforms and environments so I would argue that this answer is far more valuable.

1
^(.*?)$\s+?^(?=.*^\1$)

regex

As the author of the answer explains:
^ matches the start of a row.
(.?) matches any characters 0 or more times, but as few as possible (it matches exactly on row, this is needed because you have to check the “. matches newline” in the replace dialog.). The matched row is stored, because of the parens around this section.
$ matches the end of a row
\s+?^ matches all whitespace characters (newlines) until the start of the next row.
(?=.^\1$) The positive lookahead assertion. The topic of the post. Note that we’re using the \1 to reference the row/text found in the first part of the expression. As the author states, a row is only matched and removed when there is exactly the same row following it somewhere else in the file.

Here’s some more info on positive lookaheads if you haven’t gotten enough.

Python iterate files (weak sauce)

I am seemingly always having to rake directories listing off this or that.
I’m posting this here for my future self…If you get something out of it, all the better.
It is not rocket science, it is not even science, it is not… you get the point.

1
2
3
4
5
6
7
8
import time, os
for dirname, dirnames, filenames in os.walk(r'D:/eath_star/'):
    for filename in filenames:
        if filename=='tatooine_cantinas.shp':
            file = os.path.join(dirname, filename)
            created = time.ctime(os.path.getctime(file))
            size = (os.path.getsize(file)/1024)/1024
            print file, str(size)+'mb', created

Convert all files in a directory using PIL.

1
2
3
4
5
6
7
import Image, os

for dirname, dirnames, filenames in os.walk('D:/browse_graphic/'):
    for filename in filenames:
        if filename.endswith('.png'):
            im = Image.open(dirname+os.sep+filename)
            im.convert('RGB').save("d:/browse_graphic/"+filename.replace('png','jpg'))

DnB of the week:

Python cheats

I’m not an expert on Python’s list comprehensions or even built-in functions for that matter, but I often feel like I should be since I am amazed at how useful they can be.

Stumbled across this one this morning:

Check if multiple strings exist in another string

The built in ‘any’ function Returns True if any element of the iterable is true:

therefore if we try to do this:

1
2
3
4
5
6
a = ['a', 'b', 'c']
str = "a123"
if a in str:
  print "some"
else:
  print "none"

it fails and we get this

1
>>> TypeError: 'in <string>' requires string as left operand, not list

Using a list comprehension we can easily overcome this:

1
2
3
4
5
6
a = ['a', 'b', 'c']
str = "a123"
if any(x in str for x in a):
	print "some"
else:
	print "none"

Yay

This

Tobin Bradley is a force of nature. I really enjoy his work.
This one in particular is pretty amazing.

In his screencast he shows how to stand up a virtual Ubuntu server on Digital Ocean and install an mbtiles server. The whole thing takes less than 15 minutes.
Insanely simple and amazingly effective.

Tilemill - polygon compositing operations

Tilemill is a pretty great map rendering engine IMHO. The compositing options really help avoid extraneous datasets that are often used purely for cartography hacks in Arcmap.

The mxd for a ‘heavy jungles’ map service has three layers 1) jungle polygons within a given study area, 2) the study area area (polygon) - just symbolized on the outline and, 3) A polygon ‘mask’ layer that covers any other features outside the study area. (basically the inverse of the study area).

I’m not sure how the ‘mask’ was created but it is essentially a giant rectangle stretching well beyond the extent of the service with a study area hole cut out of it. Is there any way to simply use the study area itself to mask features outside it? I don’t think it’s possible in ArcMap, but in Tilemill, the polygon-comp-op rocks it!

The jungles layer:
glog

The study area:
study_area

The ‘mask’:
mask

The first two layers are stored in our enterprise environment, but the third exists solely for cartographic purposes. So how can we make this work in Tilemill with just the ‘valid’ datasets?

The Tilemill carto documentation describes ‘src’ as:

The src and dst composite operations show only the source and destination layers, respectively. Neither are of much use in TileMill (where you can just as easily hide the layers). The src-over comp-op is another one you won’t be uding much. It draws the source and destination normally, the same as not applying a comp-op at all. The rest of the alpha blending compositing operations may be useful for cartography, however.

“Neither are of much use in Tilemill” unless used in conjunction with polygon-comp-op and in a scenario as I’ve described.

Here’s the carto css and output with the jungle polygons and the study area. Note that I’ve made the map background red to drive the point home. Also note that I simplified the jungle symbology for brevity here, obviously the thumbnails have different symbology - please try your hardest to look past this inconsistency ;)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Map{background-color:red//rgba(233,233,202,.3);
}

#jungles{
    polygon-fill:#FBF190;
    line-color:#FBF190;
    line-width:.7;
}

#study_area {
  line-color:#999;
  line-width:2;
  line-dasharray:3,3;
  line-opacity:.5;
  line-join:round;
  polygon-opacity:0;
}

For this to work, we need the study area underneath the jungles:
toc

Output with red background
redback
Add the polygon-comp-op:src; to the studyarea mss:
polygoncompop
Make the map background the preferred mask color:
final
Voila! no more need for a mask layer. Thanks Tilemill!

Don’t you want to know what I’m listening to?

Spatial views in Oracle

Creating spatial views in Oracle SDE is pretty easy and can provide a slick means of limiting access to features, fields, spatial extents etc..
Following ESRI’s guidelines on the creation of a spatial view, I cruised in to the database and whispered some primitive incantations.

1
2
create view white_hats_vw as select * 
from nsa_employees where alignment in ('Lawful Good','Neutral Good','Chaotic Good')

Note that you could do a lot more with the above statement, limit the spatial extent, join to other tables etc. Note also that if this view is used for security-ish purposes, you’ll want to grant the appropriate (SELECT) privileges to your target user.

Behold, a view is born:
view21

So what can we do with this view?
When we ‘view’ it (pun intended) in ArcCatalog it looks like this:
view3
A table with a square in it, to suggest that there is something ‘spatial’ about it.

Can we preview it?
Yes, we can.
view9
But that’s funny… the table with a square in it now resembles a polygon feature class… ok. innocent until proven guilty I guess.

Can we ArcMap it? Yes we can. But since it’s not registered, ArcMap will need a unique ID to support selecting. It’ll also need to know a few details about the geometry but it can infer these from the ST_Geometry field.

view5

Once we get past that dialog we seem to be golden with support for spatial selections, select by attributes, viewing the attribute table etc. etc.
view10

So now what about distributing this view? Will your constituents need to deal with the likely confusing previous dialog?
Nope. Just save the view as a layer file and the Objectid and geom operators will be stored for future use.

Note that another way to create a view is to right click on the database connection in ArcCatalog. - >New -> View
view7

We then receive this dialog:
view11

View name I can dig but ‘view definition’? I think a better name might be in order like SQL SELECT STATEMENT. At any rate this is the select statement that the view will be based on. Whereas above we began our sql statement with ‘CREATE VIEW’, here we merely include the select bits. I haven’t played around with this one as much but my hunch is that some sparks might fly if you try to utilize st_geometry relational operators etc. etc.

Good stuff, all is good except for the leftover uncertainty about registering views with SDE and then registering them with the geodatabase. When right-clicking a view in ArcCatalog, we don’t have the option to register with the geodatabase. Presumably due to the fact that it’s not registered with SDE. Curiously, the 10.0 version of this method includes the sdelayer - o register command line bits (at the very bottom of the page) to register with SDE while the 10.1 stub (link above) does not (mysterious!). Until I need more functionality than being able to view, select and easily distribute my layer file, I will avoid that bridge.

have fun, be safe.

SQL Developer Explain Plan

In SQL Developer, feel free to hit F10 to crack open the Explain Plan window. Watch out for too many Full Table Scans ;).

1
2
3
4
5
select ttlid, atlid from 
(SELECT t.shape as taxshape,t.tlid as ttlid, n.shape as addpoint, n.tlid as atlid
                      FROM   taxlots t, addpts n
                      WHERE sde.St_envintersects (t.shape, n.shape) = 1)
              WHERE sde.St_contains(taxshape, addpoint)=1

Explain yourself:ExplainPlan