Automating Complex Workflows using Processing Modeler (QGIS3)

GIS Workflows typically involve many steps - with each step generating intermediate output that is used by the next step. If you change the input data or want to tweak a parameter, you will need to run through the entire process again manually. Fortunately, QGIS has a graphical modeler built-in that can help you define your workflow and run it with a single invocation. You can also run these workflows as a batch over a large number of inputs.

Overview of the task

We will take a point layer of maritime piracy incidents and create a processing model to produce a density map by aggregating them over a global hexagonal grid.

Other skills you will learn

  • Using a global equal area projection and setting the Project CRS.

  • Applying a Graduated symbology to a polygon layer.

Get the data

National Geospatial-Intelligence Agency’s Maritime Safety Information portal provides a shapefile of all incidencts of maritine piracy in the form on Anti-shipping Activity Messages. Download the Arc Shape file version of the database.

Natural Earth has several global vector layers. Download the 10m Physical Vectors - Land containing Land polygons.

For convenience, you may directly download a copy of the above layers from below:

ASAM_shp.zip

ne_10m_land.zip

Data Source: [NGA_MSI] [NATURALEARTH]

Procedure

  1. In the QGIS Browser Panel, locate the directory where you saved your downloaded data. Expand the ne_10m_land.zip and select the ne_10m_land.shp layer. Drag the layer to the canvas. Next, locate the ASAM_shp.zip file. Expand it and select the asam_data_download/ASAM_events.shp layer and drag it on to the canvas.

../../_images/1108.png
  1. The ASAM_events.shp layer does not have projection information associated with it, so you will be prompted to select a CRS in the Coordinate Reference System Selector. Here, the points are in the Latitude and Longitude coordinates, so select the WGS 84 CRS and click OK.

../../_images/2104.png
  1. Once the layer is loaded, you can see the individual points representing incidents of piracy locations. Let’s start building our Processing model to process these layers. Go to Processing ‣ Graphical Modeler….

../../_images/350.png
  1. In the Processing Modeler dialog, locate the Model Properties panel. Enter piracy hexbin as the Name of the model and projects as the Groups. Click the Save button.

../../_images/424.png
  1. Save the model as piracy_hexbin.

../../_images/522.png
  1. Now we can start building a graphical model of our processing pipeline. The Processing modeler dialog contains a left-hand panel and a main canvas. On the left-hand panel, locate the Inputs panel listing various types of input data types. Scroll down and select the + Vector Layer input. Drag it to the canvas.

../../_images/622.png
  1. Enter Input Points as the Parameter name and Point as the Geometry type. This input represents the piracy incidents point layer.

../../_images/721.png
  1. Next, drag another + Vector Layer input to the canvas. Enter Base Layer as the Parameter name and Polygon as the Geometry type. This input represents the natural earth global land layer.

../../_images/821.png
  1. As we are generating a global hexagonal grid, we can ask the user to supply us the grid size as an input instead of hard-coding it as part of our model. This way, the user can quickly experiment with different grid sizes without changing the model at all. select a + Number input and drag it to the canvas. Enter Grid Size as the Parameter name and click OK.

../../_images/921.png
  1. Now that we have our user inputs defined, we are ready to add processing steps. All of the processing algorithms are available to you under the Algorithms tab. The first step in our pipeline will be to reproject the base layer to the Project CRS. Search for Reproject layer algorithm and drag it to the canvas.

Note

The necessity of this reprojection step will become clear shortly. The grid generation algorithm requires us to specify the extent of the grid in the unit of the Project CRS. We can supply this reprojected layer to compute this extent.

../../_images/1029.png
  1. In the Reproject layer dialog, select Base Layer as the Input layer. Check the Use project CRS as the Target CRS. Click OK.

../../_images/1132.png
  1. In the Processing Modeler canvas, you will notice a connection appear between the + Base Layer input and the Reproject layer algorithm. This connection indicates the flow of our processing pipeline. Next step is to create a hexagonal grid. Search for the Create grid algorithm and drag it to the canvas.

../../_images/1230.png
  1. In the Generate grid dialog, choose Hexagon (polygon) as the Grid type. Select Extent of 'Reprojected' from algorithm 'Reproject Layer' as the Grid extent. Click the 123 button under the Horizonal spacing label and choose Model input.

../../_images/1328.png
  1. Select Grid Size input for Using model input. Repeat the same process for Vertical Spacing. Click OK.

../../_images/1426.png
  1. At this point, we have a global hexagonal grid. The grid spans the full extent of the base layer, including land areas and places where there are no points. Let’s filter out those grid polygons where there are no input points. Search for Extract by location algorithm and drag it to the canvas.

../../_images/1523.png
  1. For Extract features from, select 'Grid' from algorithm 'Generate Grid', Where the features (geometric predicate) as Intersect and By comparing to the features from as Input points. Click OK.

../../_images/1622.png
  1. Now we have only those grid polygons that contain some input points. To aggregate these points, we will use Count points in polygon algorithm. Search and drag it to the canvas.

../../_images/1723.png
  1. Select 'Extracted (location)' from algorithm 'Extract by location' as the value for Polygons. The Points layer would be Input Points. At the bottom, name the Count output layer as Aggregated. Click OK.

../../_images/1820.png
  1. The model is now complete. Click the Save button.

../../_images/1917.png
  1. Switch to the main QGIS window. You can find your newly created model in the Processing Toolbox under Models ‣ projects ‣ piracy_hexbin. Now it is time to run and test the model. As our goal is to aggregate the input points over hexagonal grids, it is important that the grids are generated using a equal-area projection. This will ensure that regardless of the location of the grid, it will cover exactly the same area. Our model doesn’t explicitely ask for a CRS, but uses whatever CRS is set as the Project CRS. Let’s choose a global equal area projection as the Project CRS. Go to Project ‣ Properties.

../../_images/2016.png
  1. In the Project Properties dialog, switch to the CRS tab. We will use a global Mollweide projection for this exercise which is a equal area projection. Search for Mollweide in the Filter box and select World_Mollweide EPSG:54009 as the CRS. Click OK.

../../_images/2119.png
  1. You will see the layers getting reprojected on-the-fly to the selected CRS. Locate the piracy_hexbin model in the Processing Toolbox and double-click it.

../../_images/2218.png
  1. Our Base Layer is the ne_10m_land and the Input Points layer is ASAM_events. The Grid Size needs to be specified in the units of the selected CRS. The World_Mollweide CRS unit is meters, so we specify 100000 m (100 Kms) as the Grid Size. Click Run to start the processing pipeline. Once the process finishes, click Close.

../../_images/2316.png
  1. You will see a new layer Aggregated loaded as the result of the model. As you explore, you will notice the layer contains an attribute called NUMPOINTS containing the number of piracy incidents points contained within that grid feature. Let’s style this layer to display this information better. Right-click the Aggregated layer and select Properties.

../../_images/2415.png
  1. Switch to the Symbology tab. Select Graduated symbology and NUMPOINTS as the Column. Click Change.. next to Symbol label.

../../_images/2514.png
  1. Select Simple fill symbol and check the Transparent Stroke button under Stroke color. This is to make the hexagon edges transparent.

../../_images/2614.png
  1. Click the dropdown next to Color ramp and select the Viridis ramp. Click the dropdown again and select Invert Color Ramp to reverse the order of color.

../../_images/2714.png
  1. The Graduated symbology will divide the values in the selected column into distinct classes and assign a different color to each of the classes. Select Natural Breaks (Jenks) as the Mode and click Classify and click OK.

Note

see Basic Vector Styling for a detailed explanation of different modes.

../../_images/2814.png
  1. Back in the main QGIS window, turn off the ASAM_events layer. You will see a nice visualization of piracy hotspots across the globe.

../../_images/2913.png

Now that you have encoded the full data pipeline in the model, it is easy to reproduce your results. A model also allows you to experiment quickly without manually repeating each intermediate step every time. If your inputs change over time, say an updated database of piracy is released after a few months, you can run your model on that input to generate a similar visualization without having to remember each step.


If you want to give feedback or share your experience with this tutorial, please comment below. (requires GitHub account)