Welcome

Well. It’s always important to get off to a good start isn’t it…?

Welcome – I hope to post some projects in areas of interest to me as I explore the world of Geographic Information Systems. My day-in day-out experience is very ESRI-orientated, however I am looking to span out into Open Source (QGIS, uDIG) and online (CartoDB, MapBox) etc. Here goes nothing…!

On the left-hand side you should see my projects, with a small but rapidly expanding list beneath it. Have a look at whatever takes your fancy but remember this is a blog. Something to do with the space/time continuum means the first bit you probably should read is actually at the bottom. Older things are heavier. Lawsuit.

On that note, the following applies: the information contained is for general information purposes only. I have tried to ensure any information used/relayed is up-to-date and correct, but make no representation or warranty of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the information, products, services, or related graphics contained here (or associated links) for any purpose. Any reliance you place on such information is therefore strictly at your own risk. In no event should liability for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this website (or associated links).

I also stole the header from here: http://sanctuaries.noaa.gov/library/gis-header.jpg (NOAA)

 

Project: #EUreferendum – Week 4 Opinion

You best read this before the below, to ensure all makes sense and you understand how the following has been arrived at.

Week 4 = Sunday the 5th to Saturday the 11th of June

What has been the opinion on Twitter regarding the upcoming UK EU Referendum in Week 4? In Week 1 , Week 2 and Week 3 it looked like “leave” was certainly the louder side of the debate; but with a few doubts over whether it was coming or going.

On paper, Week 4 saw “leave” successfully counter-act these suspicions…

  • Leave = 6,843 tweets in Week 4 (3,291 tweets in Week 3; 3,429 tweets in Week 2 and 2,566 in Week 1)
  • Remain = 1,971 tweets in Week 4 (1,258 tweets in Week 3; 1,103 tweets in Week 2 and 727 in Week 1)

As can be seen this is a significant week-on-week increase for “leave” when compared to Week 3. Where “remain” has increased by 56% since Week 3; “leave” has rocketed by 108%!

“Leave” certainly remains the louder, and by my metrics I make that a 266% increase in “leave” tweets when compared to when I started this assessment in Week 1. For balance however, over that same period of time “remain” tweets has seen a 271% increase. So there is always that potential of a silent majority that I referred to last week. However the spike seen this week for “leave” appears significant:

week_1_to_4

See below for a temporal torque map together with an underlying spatial heatmap demonstrating the rough geography of the debate. Of the 8,844 tweets I was able to georeference 4,005 using CartoDB’s inbuilt georeferencing tools (45%):

https://tgf-mapping.cartodb.com/viz/6a665e5a-30a4-11e6-991d-0ecd1babdde5/public_map

Obviously: this is not intended to try and predict the result of the referendum in any way.

Project: #EUreferendum – Week 3 Opinion

You best read this before the below, to ensure all makes sense and you understand how the following has been arrived at.

Week 3 = Sunday the 29th of May to Saturday the 4th of June

What has been the opinion on Twitter regarding the upcoming UK EU Referendum in Week 3? In Week 1 and Week 2 it looked like “leave” was winning…

Week 3 represents a continuation of the trend: those expressing their opinions in such a way to trigger my “leave” criteria still out-number those that appear to want to “remain”:

  • Leave = 3,291 tweets in Week 3 (3,429 tweets in Week 2 and 2,566 in Week 1)
  • Remain = 1,258 tweets in Week 3 (1,103 tweets in Week 2 and 727 in Week 1)

Week-on-week however, this demonstrates a 4% reduction in “leave” tweets when compared to Week 2; but is still a 28% increase on Week 1. “Remain” tweets have increased by 14% when compared to Week 2; and 73% when compared to Week 1.

Could this be a sign of a quieter “remain” majority starting to express their opinion as the hyperbole becomes extreme and time increasingly narrow, for what is considered the biggest decision in generations?

See below for a temporal torque map together with an underlying spatial heatmap demonstrating the rough geography of the debate. Of the 4,549 tweets I was able to georeference 2,011 using CartoDB’s inbuilt georeferencing tools (44%).

https://tgf-mapping.cartodb.com/viz/5a724bd8-2b1f-11e6-a475-0e31c9be1b51/public_map

Obviously: this is not intended to try and predict the result of the referendum in any way.

Project: #EUreferendum – Week 2 Opinion

You best read this before the below, to ensure all makes sense and you understand how the following has been arrived at.

Week 2 = Sunday the 22nd to Saturday the 28th of May

What has been the opinion on Twitter regarding the upcoming UK EU Referendum in Week 2? Last week it looked like “leave” was winning…

In short, nothing has really changed. Those expressing their opinions in such a way to trigger my “leave” criteria still out-number those that appear to want to “remain”:

  • Leave = 3,429 tweets in Week 2 (up from 2,566 in Week 1)
  • Remain = 1,103 tweets in Week 2 (up from 727 in Week 1)

However this means Leave have a 34% week-on-week increase; but Remain have a 52% increase.

See below for a temporal torque map together with an underlying spatial heatmap demonstrating the rough geography of the debate. Of the 4,532 tweets I was able to georeference 2,116 using CartoDB’s inbuilt georeferencing tools (46%).

https://tgf-mapping.cartodb.com/viz/1bbfbf7c-268a-11e6-83dc-0e31c9be1b51/public_map

Obviously: this is not intended to try and predict the result of the referendum in any way.

Project: #EUreferendum – Week 1 Opinion

You best read this before the below, to ensure all makes sense and you understand how the following has been arrived at.

Week 1 = Sunday the 15th to Saturday the 21st of May

What has been the opinion on Twitter regarding the upcoming UK EU Referendum in Week 1?

In short, those expressing their opinions in such a way to trigger my “leave” criteria (as explained in my post linked to above) far out-number those that appear to want to “remain”:

  • Leave = 2,566 tweets
  • Remain = 727 tweets

See below for a temporal torque map together with an underlying spatial heatmap demonstrating the rough geography of the debate. Of the 3,293 tweets I was able to georeference 1,324 using CartoDB’s inbuilt georeferencing tools (at 40% this isn’t a terrible result!).

https://tgf-mapping.cartodb.com/viz/e50dea62-267a-11e6-a894-0e787de82d45/public_map

Obviously: this is not intended to try and predict the result of the referendum in any way.

Project: #EUreferendum – the aim

Whilst no-one is reading anything else on this page…

In the UK, the current Conservative government won the 2015 General Election promising the populus a referendum on that juicy bone of contention: the European Union. Do we want to remain or do we want to leave?

Unfortunately for the Conservatives, they won that general election out-right, and now actually have to offer this referendum. Cynics amongst me suggest they never expected to win a majority and that this would be compromised out in a coalition. It didn’t work out that way; for better for their party, they won; for worse they are now tearing themselves apart internally over minor disagreements such as economic stability, foreign policy and something similar to a Hitler/WWIII/Zombie Apocalypse.

The EU referendum is on the 23rd of June 2016. Being the 23rd of May today I thought it would be nice to measure the mood over the next few weeks. I wondered, where would be best data-source for a reasoned, balanced debate on such an emotive issue – and obviously didn’t conclude on Twitter (sorry!). But I did figure out how to harvest Twitter data, so there we go. As of today I have started harvesting Twitter data every 15minutes for the following categories:

  • Tweets that contain #leave AND #Brexit; one of #EUreferendum OR #EUref; and that explicitly do not contain #remain OR #StrongerIn.
  • Tweets that contain#remain OR #StrongerIn; one of #EUreferendum OR #EUref; and that explicitly do not contain #leave OR #Brexit.

Obviously the above will not completely capture one-side; or the other. The aim was to take a regular, consistent sample however. The majority will not have geolocation, so I will have to see how best I can geo-reference the user location based on their own assessment of where exactly they think they are (“Moon, lol”). The aim however is to produce either some heat-maps or temporal torque maps as the weeks go on. Rough time-lines:

  1. 30/05/2016 – maps for 15-21 May: here
  2. 30/05/2016 – maps for 22-28 May: here
  3. 05/06/2016 – maps for 29-4 June
  4. 12/06/2016 – maps for 5-11 June
  5. 19/06/2016 – maps for 12-18 June
  6. 22/06/2016 – maps for the whole time-period…

We’ll see how that works out. And also that minor issue of being a “European”.

EDIT: Data was also collected for w/c 16th so the above timelines have been updated.

 

Project: UK MP Expenses – 2015/16

As explained in the mist of time or here, if you can click. A spatial summary of MP expense claims made in the financial year of 2015/16. The data only extends to January 2016 currently, but this will be updated in July. According to my calculations, some key facts/figures:

  • Nothing else happened this year. I’m sick of jokes and tom-foolery.
  • The greatest number of expense claims in a constituency so far is 657 (Berwickshire, Roxburgh & Selkirk). The least remains 0 (Richmond Park).
  • The greatest amount paid-out by the tax-payer in a constituency so far is £72,694.45 (Glasgow South). The least remains £0 (Richmond Park).
  • The greatest amount of claims that were rejected in a constituency so far is £6,000 (Folkestone & Hythe).
  • The greatest re-payment so far is £5,444 (Leigh).
  • I could go on and repeat this for each category of expense as well (starting up costs; staffing; office costs; accommodation; travel; miscellaneous; and winding up costs) but I won’t, as that is the point of the map below….

https://tgf-mapping.cartodb.com/viz/7598c352-1f66-11e6-b3c5-0ecfd53eb7d3/public_map

 

Project: UK MP Expenses – 2014/15

As explained even further down there somewhere; or here, if you can click. A spatial summary of MP expense claims made in the financial year of 2014/15. According to my calculations, some key facts/figures:

  • David Moyes was sacked as manager of Manchester United (Louis van Gaal replaced him – and was sacked, today. This is one of those moments.)
  • The greatest number of expense claims in a constituency was a moderate 1,053 (Tewkesbury). The least was an unrestrained 0 (Richmond Park). I told you about this one last time.
  • The greatest amount paid-out by the tax-payer in a constituency was £212,924.04 (Heywood and Middleton). The least was £0 (Richmond Park). I know.
  • The greatest amount of claims that were rejected in a constituency was £10,108.09 (Plymouth, Sutton & Devonport).
  • The greatest re-payment was £5,456.82 (Weaver Vale).
  • Once again. I could go on and repeat this for each category of expense as well (starting up costs; staffing; office costs; accommodation; travel; miscellaneous; and winding up costs) but I won’t, as that is the point of the map below….

https://tgf-mapping.cartodb.com/viz/790e92dc-1f66-11e6-8281-0ea31932ec1d/public_map

Project: UK MP Expenses – 2013/14

As explained down there somewhere; or here, if you can click. A spatial summary of MP expense claims made in the financial year of 2013/14. According to my calculations, some key facts/figures:

  • Blurred Lines by Pharrell Williams/Robin Thicke/Marvin Gaye was No 1…
  • The greatest number of expense claims in a constituency was a tiny 1,374 (Washington & Sunderland West). The least was a huge 0 (Richmond Park).
  • The greatest amount paid-out by the tax-payer in a constituency was £230,481.60 (Wythenshawe & Sale East). The least was £0 (Richmond Park – you may get the picture about this one…)
  • The greatest amount of claims that were rejected in a constituency was £3,725.12 (Ipswich).
  • The greatest re-payment was £7,500 (Bolton North East).
  • I could go on and repeat this for each category of expense as well (starting up costs; staffing; office costs; accommodation; travel; miscellaneous; and winding up costs) but I won’t, as that is the point of the map below….

https://tgf-mapping.cartodb.com/viz/5cc219be-1f66-11e6-9933-0ecd1babdde5/public_map

Project: UK MP Expenses – the aim

 So. Let’s start by keeping it relatively light and relaxed…

For those of you not aware, 2008 was a big year – the entire world economy went to pot; and, in the UK, in 2009 it didn’t get much better as it was revealed that those in charge were, literally, on the gravy-train. Over-claiming food receipts was one thing, as was charging for a duck-house, or moat-maintenance – but there were some particularly dodgy things going on, particularly with second-homes and mortgages, that an establishment already shocked to the core economically probably should have avoided. But no-one claimed for that much advised time-machine… read on.

Anyway. In response, the government set up the Independent Parliamentary Standards Authority (IPSA) to manage, maintain and generally ruin their fun. (Side-note: it then proceeded to give MPs a 10% pay rise years later…). To IPSA’s credit they do publish all expense claims and even have an interactive map. However I’m personally not a fan of the ‘narrow the map quickly’ approach before revealing the facts & figures. It always makes me feel like I’m missing the wider picture – and I can hardly click on every region and zoom in now can I! (Yes). Add to this the fact that the IPSA spreadsheets reveal a lot more than the map does (claims categories/if they were rejected or not/any repayments etc) – I thought this would be a nice project.

Working using the archived data for 2013/14, 2014/15 and 2015/16 I created an Excel workbook with VBA macros that formatted the data so it was workable i.e. transactions became floats; extraneous columns deleted; names formatted with no gaps etc. They were then imported into ArcGIS where a Model pulled the 500,000 transactions apart by category; summarised them per constituency code; and joined them to the shapefile with the 632 constituency boundaries and codes included. This enabled analysis at the constituency level. It had to be as automated as possible to enable updating for the rest of the 2015/16 data, and perhaps onwards.

My aim was to spatially demonstrate where expense claims were being made UK-wide; for what categories; and if anything warranted a closer-look (i.e. lots of repayments). My aim wasn’t to try and add a lat/long for each transaction or even to attribute them to a particular MP – this can be deduced from the archive spreadsheets if needed – the aim was just to lay the details out on the table a little bit more. Transparency is a fantastic thing, but only if everyone takes full advantage.

These maps have now been added to CartoDB (first-time using it – very impressed!) and I will add them above together with a few facts and figures.

Dive-in!