Electoral Radar – Evaluating a Political Campaign Using Data from Social Networks
In 2014, I developed a Business Intelligence application to monitor a political campaign in Brazil. It allows a candidate to follow what is going on in his region during the campaign through social network posts. This service was contracted by 5 candidates over the period July to November 2014.
I kept a few days of data online for demo purposes. You can access this application here and actually use it as I describe below. Don’t use a tablet or cell phone, ok? Since a lot of data is loaded into the browser, you really need a desktop. The application is in Portuguese but I’m sure you can figure out what’s happening with a few explanations.
Here is the main dashboard, split into two figures to help viewing:
- The application loads all posts in the browser so wait a bit for loading.
- There are 4 main panels in the application: main dashboard, posts listing, who-is-talking (people panel) and graphs panel. You can access the 4 panels by clicking on the appropriate icon on the left-hand side.
- The main dashboard shows charts: a bar chart showing who is ahead (in volume of posts), a sentiment pie chart, 2 choropleth charts showing sentiment and volume geographically, a post counter, a time chart of post volume, a theme chart (topic modeling), top influencers (negative and positive), as well as a picture of the leading candidate.
- What makes the interface powerful is the set of filters that can be used to drill down into the data. There are filters on the candidate (click on the bar chart), sentiment (click on the pie chart), geography (click on the maps), date and time, theme (topic), source (Facebook, Twitter, Instagram, Youtube, RSS) and by search the text (top right). The filters are combined and all other panels are updated to reflect the aggregate of posts that matched the filters.
- You can reset the filters using several reset buttons as shown in the above images.
- As an example, here is the main dashboard after filtering Facebook on a particular region of the map. Compared to the first dashboard, the other candidate is ahead in that region, at least on Facebook.
- The second panel deals with individual posts. There is a region to visualize the first post and first video where “first” may be chosen to be: most important, most frequent (similar posts are detected as repetitions), most offensive and most recent.
- The individual posts are shown in the order chosen. Many images are missing because Facebook has closed down on their API since then.
- The third panel shows posts organized by author: what are the candidates themselves saying, the press, some political authorities, as well as political parties. See the 4 screen shots below:
- The fourth panel show many graphs, always as a result of the filtering applied:
Some words about implementation technologies
- First, let me say that I had some help from 2 engineers and 1 designer (whom I thank) to put this application together; nevertheless, I did most of the development, from post extraction to front end.
- The post extractors are written in Python using public APIs for Facebook, Twitter, Instagram and Youtube. The RSS extractor uses feedparser.
- There is some screen scraping to extract text from unstructured web pages (following links extracted from posts).
- The database is mongodb
- Post filtering uses whoosh
- Language detection uses guess_language
- Sentiment is done using the Google Prediction Engine
- Charts are done with dc.js, dimple, crossfilter and d3.js
- Geographic information to build maps comes from OpenStreetMap with many scripts to massage the data to be usable by dc.js