Saved it! I am still looking for good intro resources into PostGIS and PostgreSQL.
Also looking for advice: what should I learn first, python or postgis?
My goal is to perform 3 clips and some visualization stuff on the resulting layers as efficiently as possible since I need to perform that task at least twice per week. The underlying data is From OSM, currently in shape format, but I would prefer osm.pbf (global extract).
While skimming through the python doc of arcgis and qgis I couldn’t find anything on how to visualize data.
I am currently reading Automate the boring stuff and have some other GIS and Geospatial python books lying around.
So one approach would be learning Python and then writing the script.
The other approach I recently thought of is loading the global extract into PostGIS and always work with the DB.
Haven’t made up my mind yet on how to visualize afterwards and I don’t have any background in SQL, so there’s more knowledge in the python Departement.
What do you think is easier, faster to learn and more time efficient after it’s implementation?
The OSM planet extract is massive, so it's not the easiest place to start your databases work. If you want to automate the visualization and the clipping (assuming it's a relatively small clip), I would suggest first learning to automate things with Python and then automate your processes with ArcGIS or QGIS script builder. Or it depends what is more painful for you at the moment: the data part (loading, moving and clipping) or the visualization?
How long did that network analysis take to process?!
Don’t remember exactly, but more than 15 minutes and less than few hours.
Routes were calculated from the API with Python
Routes were calculated from the API with Python
Sorry, that's what I meant. I imagine calculating all the route weights took far longer than generating the image itself, which is why I wondered whether a faster language could be used instead of Python, and maybe get away with not having to pre-process anything.
From the point of view of making this into an App, and especially for large maps, rather than having to wait ages for it to finish calculating all the routes before generating an image, it would be cool to watch it generate the routes in real-time, branching out slowly from your starting point.
I've downloaded some map data to play around with, thanks for sharing!
Just to clarify, I had the (java) routing engine running locally on my machine and just did the API calls with Python. I think you can’t avoid pre-processing. But I would be happy to be wrong.
Did you try OSRM? It should be considerably faster.
I have tried it, but setting up GraphHopper worked fast enough for my purpose.
Load more comments
I know that arcmap can do this. Would be interested if Topi [OP] ever tried it out and wether he can compare both approaches.
I don't have accessibility to any ESRI licenses at the moment as I do these just for fun on my own laptop. So no, I haven't tried it out.
Great post, very thorough.
What's the background map on the train visualisation?
Cheers. The background is all Finnish roads (and nothing else). It's open data and I have it in my PostGIS. Just looks like a background map when visualized that way.
Five hours of train traffc from last Sunday in Finland animated. Circle size relative to train speed. Link to my original tweet
1. What is it with the circles? Can't you see the train speed anyhow from the animation? Sure, but I just thought it looked nice and made the trains look like bubbles.
2. What tools did you use? Python, PostGIS, QGIS with Time Manager plugin and GIMP
3. Where's the code for this? Well there's only the code for getting the data from the API (other things done in the database and on desktop software) and that's here
4. Can you do one for [insert country name here]? Sure if you point me to an open GeoJSON API first.
Nice work! The circle size represents train speed? It could be interesting to test color instead, especially if the train speed is changing along the route. It might also help to add a supporting chart (maybe line?) to make the overall amount of traffic easier to see.
Good idea. Thanks!
Created with QGIS. Data from NOAA and Natural Earth
Got to ask content creator u/Tjukanov
It’s really simple Python script to get the data. I’ll put it to GitHub later.
Here it is. And please note that I'm not really a Python dev, but hope this is helpful to someone: https://gist.github.com/tjukanovt/83c88ea930a4b88cf6d91e9cdecbe145
This is awesome. I’m a GIS Analyst but over the years have never ventured too far from the ESRI environment, mostly because my jobs have all been with ESRI software or proprietary tools. I’m starting to dip my toe into open source projects and would love to do projects like this in my free time.
Other than documentation, can you recommend any helpful resources or tutorials?
YouTube is a great resource for QGIS stuff, as stupid as it may sound. Here’s one tutorial I wrote earlier https://medium.com/@tjukanov/animated-routes-with-qgis-9377c1f16021
Welcome to the world of open source!
What do the different colors mean?
The colors also represent the age. Dark purple = very old & light yellow = new. TBH maybe it needs a legend in the last frame.
Year in the bottom right shows the year and buildings appear at the time when they were (according to the data) built. Thanks to u/daniskarma for the the great static maps with the same data. Those gave me the idea to do this animation. I mostly post this kind of stuff on my Twitter. Created with QGIS Time Manager + PostGIS.. Data can be found here.
Of course, for theses maps I used a shortcut. QGIS has a plugin named Spanish Inspire Catastral Downloader that let you choose a province and automatically download the data. In the building layer you have a field named "end" which is the end of construction date.
If you want to make a manual donwload, you'll have to go to the catastro main page https://www.sedecatastro.gob.es/ and select: "Descarga de datos y cartografía por municipio", then you have to donwload the shapefiles and the CAT files (which contain the age of construction data). But a electronic certificate is needed in order to download this data (It's free to make but it takes some paperwork to get one). I prefer to use the QGIS plugin.
All of this work for all Spain but Navarra and Euskadi, both of them have their own Catastro system. Althought the process is quite similar.
Awesome maps and thanks for the data link. Loading the data now myself...
/u/Tjukanov does a lot of cool stuff with QGIS, Graphhopper and GeoData in general.
An example of such a visualisation here: Animated optimal routes from San Francisco to ~2000 locations in the U.S., Disscussion here.
Thanks for sharing!
I would like to ask if you had a list of hurricanes that reached Indiana, obviously they would have been weaker tropical storm by then. But I'm still curious.
Yes it could be extracted from the data.
Any chance you have the time to make this happen? I am quite interested in seeing this actually. I'm sure it would be a pain in the balls but it would be really interesting to see each path go through a color gradient corresponding to the the strength of the storm during its journey. I'm sure the data for that kind of thing would only be able to be applied to the more recent storms though.
Edit: Come to think of it, is there even record of the categories reached by each storm going back 100 years?
I might do it in the whole world version. Thanks for the feedback.
I teach 6th grade social studies and would love to have a high res copy of this to put on my wall. Is that something you would be ok with?
You mean doing a high res yourself? Sure, go for it! Meaning that I'd do a high res? Not promising anything, as I have quite a lot of other projects ongoing.
also /u/Tjukanov how did you pull the OSM road data into QGIS? I'm assuming that's a huge dataset and I thought with OSM you could only pull a certain extent at a time. Would be curious how you did it.
The planet file has been imported to PostGIS
Hey, this is incredible congrats on such a cool project. We're an Irish travel company specialising in US travel and was hoping you'd allow us to share this with graphic with our audience with a full credit linking back to your Twitter/Reddit or other account? Thanks in advance, Donnacha USIT Travel
Sure. Just add a link to my Twitter profile and remember to credit Openstreetmap (that's the original source of the road data) as their license requires by adding the following text there: "© OpenStreetMap contributors". Check: http://www.openstreetmap.org/copyright/en
Apologies if this has been asked, but I didn't see an answer -- what do the different colors represent? Boyfriend and I are arguing over this :)
The dots are green, but the features have a blending mode where overlapping features turn lighter and eventually white. So more white --> more destinations are using that route.