The idea of developing a QGIS plugin came from my willingness to refresh my knowledge after having collaborated to the development of a plugin a few years ago. At that time, rather inexperienced, I developed features for a plugin called Roadnet having a very detailed scope, the spatial data management of the Local Street Gazetter according to the British Standard. Back then in 2015, the latest QGIS version was 2.8, plugins were in Python 2 and Qt/PyQt latest version was 4.x. Within that project I deepened my Python knowledge, I learnt how to create Qt UI objects and its processing threading classes QThreadPool and QThread.
Fast Forward 2019, Python 3 is now the standard, QGIS supporting this Python version is now at 3.4 and Qt/PyQt last version is now 5.x, I am a less inexperienced GIS developer and I felt ready to contribute to the QGIS community with a plugin built entirely by myself. My first intention was to write a story on how to implement a general multi-purpose threaded process in QGIS, I eventually ended up building a QGIS plugin which is using a threaded process but is doing something a bit more interesting than geometry validation/reporting. Qweetgis uses the Twitter streaming API with concurrent processing in Python to download and display on a map real time tweets without locking the resources in the main QGIS UI thread. You can watch tweets pop up as points in your map while they are created without having to wait for the download process to finish. If this seems interesting enough for you, please download the plugin, use it in QGIS and keep reading this story. Here is the link for the the GitHub repository. Qweetgis is based on the work of Riccardo Klinger - Geolicious- who developed Geotweet, a twitter plugin for QGIS 2.x.
Qweetgis uses a Python package called Tweepy that makes the requests to the Twitter streaming API. Standard QGIS Python does not come with Tweepy and it has to be installed separately. There are a lot of guides online that teach you how to do it and there is not a single method that works for every user since operating systems and Python configurations may differ. I installed it on both a Windows 10 and Linux (Ubuntu 18.04) machine. On Ubuntu (where I also have Anaconda) I simply did a
$ pip3 install tweepy from the terminal and Tweepy installed into the global environment Python 3 which is the same used in QGIS. On Windows, the solution that worked for me is here.🔴 I did it on a standalone QGIS install, if you have QGIS installed within OSGEO4W this method may brake you Python2 environment 🔴. For QGIS in OSGEO4W I also managed to install Tweepy following this example to set up the OSGEO4W Python to point to Python 3, the command is
py3_env. After that, I downloaded the tar.gz Tweepy module from PyPi, extracted it and from the OSGEO4W shell, cd-ing into the extracted Tweepy folder
tweepy-3.7.0, I ran the
python3 setup.py install .
The other prerequisite is having the four authentication tokens to access the Twitter API, the two Twitter API developer keys and the two user tokens used by OAuth to authorise Qweetgis to use your account. To get the first two keys you will have to sign-up for a developer Twitter API account and create an application, instructions are here. Once you get these two keys you will be able to generate your OAuth access tokens, instructions are here. I know it’s awkward, you install an application and you’d like to use it straight away, without having to subscribe to a Twitter Dev account that you probably will never use if it wasn’t for this plugin. I tried to make the authentication process as seamless as possible, creating an authentication web service providing the OAuth tokens based on my personal DEV keys, stored in a secure server. The problem is that the Twitter API requires DEV keys in every request, these keys must be stored secretly and I couldn’t distribute mine for you to use. The risk is that someone with malicious intents could use these keys to read the tokens of the users subscribed to the plugin and potentially hack plugin users’ Twitter accounts.
Once obtained your four items for authentication, the first time you’ll start Qweetgis after it is installed you will be presented with the following login dialog.
Note: If you are behind a proxy server, Qweetgis could be set to point to it, it has to be configured before logging in, see instructions here.
Every time a new session is started the plugin looks into the
config/config.jsonfile in the plugin root directory. If credentials are not found you will be prompted to input them. When you click on OK, the inserted tokens are written and stored in the config file and you won’t need to supply them again. You can also test the credentials once inserted before proceeding. Qweetgis will ping Twitter and search for 10 random tweets to check if the connection is up and the API is responsive.
When you are happy with the inserted tokens and clicked on OK you will proceed to the main plugin dialog window. Qweetgis will also set your QGIS project projection to WGS84 long-lat and zoom to a preset extent (-180, -90, 180, 90), this is because geotagged tweets are in this coordinate system and potentially no tweets are emitted outside of the maximum extent boundary. If you want a world map to get started with, you will find a world countries boundaries shapefile inside the
test_data directory in the plugin root directory.
Streaming tweets works on two basic concepts, the tweets location and the tweets filtering. In the Twitter API, the tweet object read by Qweetgis contains the location as metadata, more details here. On the top right “Search Accuracy” dropdown list, you can select what type of location you want your streamed tweets to be georeferenced to. It can either be the tweet location or the user place.
- Tweet Location
With “tweet location” the user who emits a tweet consent to publish the GPS position recorded from the device together with the tweet, the location is recorded in the
coordinatesmetadata item.This type of geotagging isn’t always chosen from Twitter’ s users and tweets arrive quite slowly. On the other hand the location recorded in these tweets is accurate as they are the representation of the tweet “exact-point” of emission, of course the GPS error carried by common handheld GPS devices may occur.
2. User Place
The second type of geotagging ‘User Place’ is based on the location contained in the
placemetadata of the tweet object. From the twitter API docs “Places are specific, named locations with corresponding geo coordinates […] Tweets associated with Places are not necessarily issued from that location but could also potentially be about that location”. The
place metadata item is not represented with a couple of coordinates as a point but it’s a polygon with four long-lat coordinates. In Qweetgis the place is represented as a long-lat coordinates point since it is assigned the calculated centre of the polygon. This type of accuracy search retrieves tweets much faster than “tweet location” but the tweets may not always be a true reflection of the user/tweet actual location at the moment it is emitted.
Filtering the tweets stream
The search type buttons allow to switch between two different types of filtering on the tweets stream, you can filter by keywords of by a rectangular search area. Let’s see the filtering in detail.
In the text box insert the filtering keywords, the searched field is the
full_text in the
extended_tweet nested object. The plugin will display on the map only those tweets that contain the searched words you inserted, the search method of Twitter keyword filter is explained in detail here. Multiple words will be searched with the OR Boolean and may not necessarily be subsequent words. For example searching “love pizza bananas” will return a tweet like “I love my pizza with whipped cream and bananas” but also “lovely pizza” will be returned as a valid result. The text returned is the full tweet and whenever possible the message will not be truncated.
2. Location based
All tweets that have coordinates falling into a specified bounding box are returned from the stream and limits are those of the coordinate reference system (max extent is -180, -90, 90, 180). The returned results will be punctual features containing the same attributes as the textual search, the tweet will not be truncated.
Starting a keyword filtered streaming session
With the desired search accuracy, input your keywords separated with a space in the text box and press on the play button. The layer to be populated with the tweets will be added to the map canvas as a memory layer, the progress bar will activate, showing a busy indicator and the tweets counter will signal the total number of tweets received. When a new tweet is received it will instantly pop up on the map canvas, flashing for a few seconds. You can move and zoom the map, open the attribute table and explore the content of the results anytime the streaming is on. As no limits are applied to the search the streaming will continue until you don’t press on the pause button; in that case the stream listener will be shut down and you will be able to start a new streaming session with the same or different keywords. If you start a session with the same method, with the same keywords and the layer is not made permanent Qwitter will add the tweets to the memory layer with the same name on the layer register. If you don’t want this to happen make the layer permanent and save it in your preferred format on disk, Qweetgis will add a new memory layer to the map canvas.
Starting a location based filtered streaming session
When you switch search type to “Geo Search” the main dialog will show the extent selector for your filtering rectangle. The extents shown in the text boxes are those that will be used in the to filter the incoming tweets and by defaults are those of the current map extent. You can decide to change them, using the extent of all layers in the project or the extent of one particular layer in your layer register, selecting it from the dropdown list. The concept is the same as the previous filtering method, operating on the two buttons to play/pause the data stream. As for saving the layer, same rules of keywords filtering apply.
In case you wanted to return a large number of tweets for a geolocated tweet stream or for a relatively small area, the streaming session could be active for several minutes, if not hours before reaching the number of wanted tweets. Qweetgis can help you with that since you can set a limit to the number of tweets to listen for and when that limit is reached the listener will shut down on its own. This is possible with the ‘first’ limit that will return the first n results, specified in the numerical input.
The second type, the “up to” limit, will let you set a max number of tweets to receive but when the limit is reached the streaming will continue and tweets received will be updated, removing the older ones as the new ones are added. This feature could be useful for long streaming runs, for example to appreciate particular patterns of keywords or tweets’ locations that can change over time. Applying this limit will result in a less cluttered map and it is a simple yet effective way to display some geographic real-time data as an animation. The tweet streaming with this limit applied together with the capacity of QGIS 3 of displaying the point layers as a heat map, could turn the streaming in a nice and practical visualisation technique without putting too much effort into the design of sophisticated animations. The flashing of the new received tweet for this type of limit is disabled as users could set a very low limit for rapidly flowing tweet streams, the effect was making QGIS very slow and prone to crash.
Last time I worked on a QGIS plugin I thought that setting up a development environment for a QGIS plugin was a hard task. Developing this project I could happily verify that I was wrong as in a few moments thanks to the plugin builder I was ready to start. A few years later, the development of this project helped me to get to know more closely the latest available tools to develop plugins and most importantly to develop a small project on my own, from start to end. At the beginning trying to find relevant and interesting data it was a bit of a ramble but the wandering was worth it because I discovered a number of streamed geographic data sources than I am now more curious to try and potentially be the basis for the development of other applications. While working for this plugin I have found some others open live data streaming APIs, especially for transport tracking (i.e. Transport for London, Chicago Transit Authority, Aviation Edge Free API). I am fascinated by the opportunities offered by the way these datasets are delivered and I intend to explore them further in the future. If you’ve read this story, you’ve got until here and you have more live data streaming (possibly free) public APIs to suggest please leave their reference in the comments.
Proxy server settings
Qweetgis can work behind a proxy (provided that the proxy grants access to Twitter), the functionality is offered by Tweepy. In order to set-up the proxy connection, in the root folder of the plugin you will find the
config.jsonfile in the
configfolder. Under the
PRODsection insert your proxy URL, for example:
/*in case your proxy requires authentication*/