Retrieve KML links

kmlhell

On my previous post I worked with a kml that was splitted in various networked kmz files. This was particularly annoying given that I had to track the files in my computer.

I am ashamed, this should have been automatized! And since a friend prepared a nice script for it, here it is.

Simple idea

We have the urls to download the kmz files, but we can’t do it directly because the server will refuse our petition.

The server will only accept connections from google earth, so we have to generate that petition, telling the server that we are google earth.

Get the User-Agent

The User-Agent is the header of any http connection telling the server who is talking to him, firefox? chrome? or google earth?.

With the help of wireshark we can monitor the packets going through the network, the task is to obtain the ones generated by google earth, so we know that the user agent is:

GoogleEarth/7.1.2.2041(X11;Linux (3.13.0.0);en;kml:2.2;client:Free;type:default

The script

I did not have the pleasure to write this script, the credit goes to fvila. But since it has been very valuable to retrieve everything, i put it all in here.

But let’s have a quick look on what it is using:

from zipfile import ZipFile
from os.path import join, isfile, isdir
import urllib2
import sys

It imports some standard libraries to do the basic job. Zipfile to unpack the kml files, os.path to handle the file paths, urllib2 to download the remote kmz and sys for a basic set of command line parameters.

After that, apart from the basic script managing, the meat of it can be found in the function downloadKMLresources. There is nothing out of the ordinary in this function, it uses urllib2 to download a file, but using the small trick of changing the user agent

req = urllib2.Request(url)
		req.add_header('User-Agent',
        	           'GoogleEarth/7.1.2.2041(X11;Linux (3.13.0.0);en;kml:2.2;client:Free;type:default')

The full script, ready to use to download the networked kmz of any kmz file follows:

from zipfile import ZipFile
from os.path import join, isfile, isdir
import urllib2
import sys

try:
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET

def usage():
	print "Usage: %s [KMZ file] [destination]" % sys.argv[0]
	exit(-1)

def downloadKMLresources(kmz_file, dest):
	kmz = ZipFile(kmz_file, 'r')
	kml = kmz.open('doc.kml')
	xml = ET.ElementTree(file=kml)
	root = xml.getroot()

	elements = root.findall('.//{http://www.opengis.net/kml/2.2}href')
	urls = [e.text for e in elements if e.text[-3:] == 'kmz']
	downloaded_files = []

	for url in urls:
		name = join(dest, url[url.rfind('/')+1:])
		downloaded_files.append(name)
		print "Creating %s..." % name

		req = urllib2.Request(url)
		req.add_header('User-Agent',
        	           'GoogleEarth/7.1.2.2041(X11;Linux (3.13.0.0);en;kml:2.2;client:Free;type:default')

		response = urllib2.urlopen(req)
		contents = response.read()
		with open(name, 'wb') as f:
			f.write(contents)

		response.close()

	return downloaded_files

if __name__ == '__main__':
	if len(sys.argv) != 3:
		usage()

	kmz_file = sys.argv[1]
	destination = sys.argv[2]

	if not(isfile(kmz_file)) or not(isdir(destination)):
		usage()

	downloadKMLresources(kmz_file, destination)

Conclusion

This was the last piece that I needed to automate all the process described in KML hell post.
Once each kmz file is downloaded, I can process them separately:

  • Unzip the kmz
  • Copy the doc.kml with an appropiate name
  • Remove any non-ascii character
  • Run through my slightly modified gdal
  • GDAL reprojection from WGS 84 to UTM zone 33N. I did not mention that in my previous post, but it is quite important.
Advertisements

Leave a comment

Filed under gis, tools

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s