Raspberry Pi GPS tracker – Converting Code to PHP – Part 1

Last week I looked at getting the hardware up and running for a Raspberry Pi GPS tracker. However, as I said I was using some Python code and I don’t speak Python so I wanted to convert it to PHP. So this week I am going to look at what I did.

Before I could even begin to look at the converting the code I had to see if it was even possible for PHP to access the serial port. Turns out this is exactly what Direct IO (dio) is for but it isn’t included as standard so you have to download and install as follows:

First you need the prerequisites which are pear and php5-dev which is required for compiling additional modules:

sudo apt-get install php-pear php5-dev -y

Next download and compile dio:

sudo pecl install channel://pecl.php.net/dio-0.0.7

This will show the following output:

Build process completed successfully
Installing '/usr/lib/php5/20131226/dio.so'
install ok: channel://pecl.php.net/dio-0.0.7
configuration option "php_ini" is not set to php.ini location
You should add "extension=dio.so" to php.ini

Pay particular attention to the last line!

Reading from the serial port is pretty easy and very similar to reading from a file, so to open the port you do the following:

if ( !$t = fopen('dio.serial:///dev/ttyACM0','r+b',false, $c) )  echo 'Failed to open ttyACM0';

Then to read from the port:

$line=fgets($t,1024);

In our case we want to continuously cycle reading the port looking for valid GPS location records.

// cycle round forever looking for the correct GPS record
while(true) {
    // get a record from the GPS
    $line=fgets($t,1024);
    // we are only interested in GPMRC records
    if (!empty($line)){
        if ( strpos($line,"GPRMC")){
            // break up the string
            $resp = explode(",", $line);
            // we are only interested in valid GPS records
            if ($resp[2] == "A") {
                $lat_decimal = degrees_to_decimal($resp[3], $resp[4]);
                $lon_decimal = degrees_to_decimal($resp[5], $resp[6]);
                file_put_contents($logs.date('Ymd').'-simple-log.txt', $resp[9].",".$resp[1].",".$lat_decimal.",".$lon_decimal.PHP_EOL,FILE_APPEND);
            }
        }
    }
}

Rather than reproduce all the code here which will become out of date as I improve it I have published it to Github here.

Next time I will look at some interesting quirks of this and how to improve the code’s usefulness.

Raspberry Pi GPS tracker – Getting it Together

Having secured a Pi Zero from the cover of a the magazine MagPi I thought I would start out simple by trying one of the projects shown there. The one that caught my eye and required no soldering was to build a GPS tracker.

Getting hold of a cheap USB unit was pretty easy but make sure it is Linux compatible. I bought this one from eBay.

The first thing I did when I got the unit was to try it out on the Pi and while it was working I could see that it had got a satellite fix the record containing the actual location wasn’t populated and showed as invalid.

Satellite fixes in Ublox U-center software

Satellite fixes in Ublox U-center software

Assuming that it was a an issue with the unit I downloaded ublox software on a Windows machine and tried from there. Again the same thing. So I contacted the company that sold me the unit and they came back with this very full explanation.

Invalid records in Ublox U-center software

Invalid records in Ublox U-center software

The first time these devices are powered up in a new location, they firstly locate as many satellite signals as possible, then download the almanac data (future predicted satellite positions), whilst trying to calculate the current position.

If the signals aren’t strong enough, the satellites can be seen and identified by number, however the position cannot be calculated and the almanac data cannot be downloaded.

When I test all of these modules, I do so outside with a laptop, a clear view of the sky, and a hot mug of coffee (as each module can take 5-15mins the first time depending on satellite positions).

The device will likely output the current satellite time with a signal which is much weaker than the signal level required to get a location fix.

Whilst the device may give a satellite lock indoors after the initial location fix, you are unlikely to get a first location fix at all indoors, believe me I’ve tried this without much success on cold days to avoid going outside, ultimately they need a clear view of the sky to work.

So I put the kettle on and set off outside with the GPS unit once again connected to the Pi. Almost immediately it got a fix and the green light started to blink. Success!

The next step was putting it all together with some code to record the GPS records as I moved around. For this I used the recommended Python script, GPS Experimentation, which was easy to setup and configure. Unfortunately I don’t speak Python so one of my first tasks will be to see if I can re-write it in PHP.

And so I set off out with my Heath Robinson GPS tracker. As you can see from the images below the combined battery, wires and Pi resembles a cinematic bomb so I spent quite a bit of time searching for the perfect container in which to house it. In the end Poundland came up trumps with a box of screen wipes and it was cheap too (a pound, obviously).

The first surprise was that the battery,  a freebie 2000mAh usb power bank, lasted a full 10 hours just driving the Pi and GPS unit. That was seriously impressive. The second was just how accurate the tracker was when out in the open. As you can see when I plotted the data in GPSPrune.

A meander around Reading Town

A meander around Reading Town

However, there were times when things were not all that they seemed. The image below shows the hour I spent sat in a coffee shop working and highlights the small variations on GPS location that are returned. I think one improvement I will make to the code is to stop tracking when the speed is zero, or very close to it.

An hour sat in the same place

An hour sat in the same place

And at one point the tracker lost the plot completely and decided that I was in France which led to this anomaly!

A day trip to France

A day trip to France

 

On the whole this was an easy and fun project to put together and all relatively cheap. The next step is to convert the code to PHP and start to make some improvements. Check back next week for an update.

Kickstarter is a stock market not a super market

It’s no secret that I am a big fan of Kickstarter having backed many, many, many projects but, as you can see from the above screen grab, not everyone is of the same opinion and I believe that stems from a fundamental misunderstand about just what exactly a crowd funding site is.

The reason for the ire shown by the commenters above is that the project delivery end date is slipping and so people are becoming unhappy. In this particular project’s case the delivery date has slipped by approximately one month – in Kickstarter terms that is nothing, as you will see below. These sorts of comments are becoming more and more typical as Kickstarter, Indiegogo and other sites of a similar ilk are becoming more popular.

The stats

I decided to take a look at the projects that I had backed to work out how close the project delivery came to the date that the project originally quoted. To date I have backed 35 projects across a variety of categories but, unsurprisingly, skewed towards those with a technology bent.

The raw data is below but here are some headlines:

TWO projects out of 35 delivered “on time”.

101 days that on average a project is late.

971 days that one particular project is late (that’s about 2.5 years but it will deliver).

ONE project that looks unlikely to deliver and could be fraudulent (422 days late).

It takes, on average, 2.71 times longer to deliver than a project estimates.

In other words people who start projects on Kickstarter hugely underestimate the time that it will take to deliver. Having spent part of my life as a project manager in IT I am not overly surprised by this. This late delivery wouldn’t be a great problem where it not for two things:

  • lack of communication on the part of the project
  • unrealistic expectations on the part of the backer

The Communications Issue

Backers get excited about a project and are hungry for information as much as they are for delivery of the product itself. In my experience project owners never update backers as often as they would like and increasingly I am seeing this being done as part of the project comments rather than an update. This means that not everyone gets to see the update compounding the issue.

I was also taught to give bad news early and some projects do this well others not at all. PIECE has communicated a delay and looks to be delivering soon which is ironic given the comments at the top of the page are from this project.

The Unrealistic Expectations Issue

Which leads nicely onto unrealistic expectations. If you walk into a shop you can pick up an item now, take it home and have it up and running the same day. Kickstarter isn’t like that. It’s a slow burn. You have to wait, perhaps, for the design to be finished, manufacturers selected, any issues from the initial production runs sorted and finally fulfillment. No project ever runs smoothly and this inevitably pushes out the end date. However, many, many backers see the delivery date as fixed in stone but it very rarely is as the data shows.

Crowdfunding is a Stockmarket

Backing a project on a crowdfunding site is akin to to investing on the stock market. In other words there is a risk involved and you might not get back what you have invested. Even with the changes to Kickstarter that risk is not properly communicated to backers and Indiegogo is even worse.

As the popularity of the crowdfunding sites increases and they get more publicity for big projects such as Pebble they attract more people who back without fully appreciating what they are getting into.

The sites themselves need to do more to educate potential backers and not simply hide behind the terms and conditions saying that they aren’t responsible. Perhaps as a backer you should have to “sign” a disclaimer before backing saying that you accept that the project will almost certainly be late and you might lose your money.

Something needs to be done and if done properly can benefit not just the backers but also the project owners and crowdfunding sites too.

ProjectClosedEstimatedActualDays OverPredicted F2DActual F2DOut byOutcome
iPen: the first active stylus for iPad!27/12/201130/01/201216/03/201246.0034802.35Sold
The tiltpod - world's smallest articulating iPhone stand01/04/201230/04/201230/05/201230.0029592.03Still use
Pebble: E-Paper Watch for iPhone and Android19/05/201230/09/201230/03/2013181.001343152.35Upgraded
Une Bobine - For People Who Love iPhone... and Android16/06/201230/07/201230/09/201262.00441062.41Sold
HAND Stylus17/06/201230/07/201230/09/201262.00431052.44Sold
Scanbox - Turn your iPhone into a portable scanner08/07/201230/07/201230/10/201292.00221145.18Use rarely
Smarter Stand for iPad13/08/201230/10/201230/11/201231.00781091.40Discarded
Blacktop Royalty's Debut Album29/07/201230/11/201230/12/201230.001241541.24Listen Occasionally
Try iOS: iPhone App Development Course29/08/201230/11/201204/12/20124.0093971.04Discarded
pIO - microSD Adapter for Raspberry Pi06/09/201230/09/201207/11/201238.0024622.58Still use
Charlie Kaufman's Anomalisa09/09/201230/05/2013971.0026312344.69Not received
myLED - Bringing External Notifications to iPhone and iPad28/12/201230/04/201306/07/201367.001231901.54Discarded
1 Second Everyday App27/12/201230/12/201210/01/201311.003144.67Discarded
Smarter Stand for iPhone and iPod touch12/04/201330/05/201315/06/201316.0048641.33Discarded
The Earbud Shield04/05/201330/06/201315/07/201315.0057721.26Still use
The Practical Meter: Know your power!25/07/201330/09/201315/11/201346.00671131.69Still use
Une Bobine for iPhone 5|C|S09/11/201330/11/201316/01/201447.0021683.24Use rarely
the COBURNS – iPad stands06/12/201330/12/201330/12/20130.0024241.00Still use
JUMP Cable13/02/201430/05/201415/08/201477.001061831.73Sold
Swisskey™ A key / tool organizer that syncs to your phone28/02/201430/05/201431/07/201462.00911531.68Still use
SIDEKICK - Pebble Dock09/02/201430/05/201430/05/20140.001101101.00Discarded
#BuildBearHQ: Our shipping container DINER16/05/201430/05/201430/11/2014184.001419814.14Not yet redemed
Not Knot: Cable organiser & headphones protector26/05/201430/06/201430/06/20140.0035351.00Gifts
iStick™: USB Flash Drive with Lightning for iPhone and iPad17/06/201430/08/201403/12/201495.00741692.28Still use
StormTag - A Bluetooth Weather Station. On Your Keyring.25/07/201430/11/2014422.001285504.30Not received
BelayCords - Reversible USB Charging Cords iPhone & Android02/11/201430/11/201406/04/2015127.00281555.54Still use
memobottle - A4, A5 & Letter Reusable Water Bottles12/10/201430/12/201415/07/2015197.00792763.49Still use
The Roost Smart Battery - Smarten your Smoke Alarms19/12/201430/05/201515/11/2015169.001623312.04Not received
Gmail for Mac: Finally, Gmail is a true desktop email client09/12/201430/01/201515/06/2015136.00521883.62Discarded
Pebble Time - Awesome Smartwatch, No Compromises28/03/201530/07/201517/09/201549.001241731.40Still use
Lignite Collection for Pebble - Quality Watchfaces and Apps28/04/201530/06/201515/08/201546.00631091.73Still use
dio Naked Reversible USB cables for Lightning and Micro22/05/201530/06/201525/08/201556.0039952.44Still use
Halo Back: World's First Smart Screen Protector12/06/201530/08/201521/10/201552.00791311.66Still use
ZNAPS -The $9 Magnetic Adapter for your mobile devices14/08/201530/11/201557.001081651.53Not received
PIECE - Change The Way You Use Smart Phones10/09/201530/10/201588.00501382.76Not received

Useful Excel for Mac Keyboard Shortcuts

Like so many others I am a regular Excel user but having grown up on Windows finding equivalent keyboard shortcuts for the Mac can sometimes be a pain. Specifically how to manipulate cells.

So presented here as an aide memoir for me and hopefully it will prove useful to others:

Ctrl + u – edit a cell

Cmc + Alt + Enter – new line in cell

Enter – move to next cell below

Shift + Enter – move to next cell above<

Microsoft Universal Folding Keyboard

Now that screens are getting bigger on mobiles it is becoming increasingly more practical to do work on them, especially now that Microsoft Office is available on most mobile devices.

The biggest issue I have though is typing on the screen. While it is fine for short emails and text messages it isn’t really practical for longer pieces such as this blog post. One issue is that the on-screen keyboard covers too much of the screen real estate. To overcome this I’ve had portable keyboards before but they have either been clumsy or heavy to carry.

Now enter Microsoft to the rescue with their Microsoft Universal Foldable Keyboard which is a version of the keyboard that ships with the Surface. The keyboard is made of a durable material that looks as if it should be able to withstand the rigours of travel and even the odd splash of coffee here or there. It folds in half and is about the size of a CD case (for those of you that remember such things!). It has an internal battery charged through a standard micro usb port and is suggested that a single charge should last three months.

The outstanding feature of the keyboard is that it is compatible with not just Windows but also iOS and Android too. In fact it can be paired with two different devices at the same time and you can easily switch between them through a key at the top right hand corner.

Typing on the keyboard takes some getting used to, particularly when you have been used to a full sized keyboard with plenty of travel and space between the keys. The keys nearest the fold are a little odd as they are huge compared to the others but this is simply to aid with the overall placement.

With a little practice I was able to type reasonably quickly although I do still turn on the caps lock by mistake occasionally and there is nothing worse than looking at the screen only to find you have shouted the last sentence!

All in all it is a nice piece of kit and comes recommended.

Now that Microsoft is Shutting Sunrise

I have been a big fan of Sunrise but there was always a danger when it was taken over by Microsoft that they would eventually shut it down and that is now on the cards. I really like the way that Sunrise allows me to see in my calendar information other than my diary entries such as Tripit travel plans and Foursquare checkins and I didn’t want to lose that.

My initial thought was to do something through IFTTT but not all services were covered and I couldn’t get the options I wanted to change the entry colour, for example. It turns out that the answer is pretty simple and that is to use calendar feeds, such as iCal or a dedicated feed for Google Calendar which is what I use.

The following instructions show how to add Foursquare checkins to your Google Calendar but the process is very similar for other services.

Login to your Foursquare account and then go to this link: https://foursquare.com/feeds/. You will be taken to a page at the bottom of which are a number of links for different calendar feeds. For Google Calendar I clicked “Add my check-in history to my Google Calendar”

Foursquare

This took me directly to my calendar where I was asked to confirm that I wanted to add this calendar

I clicked Yes and after a short wait I saw the following notification:Google_Calendar_-_Week_of_28_Dec_2015

The calendar feed then appears on the left hand side from where you can click on the (tiny) down arrow and make changes such as to the colour of the entries.

Google_Calendar_-_Week_of_28_Dec_2015

After a little tweaking I saw my checkins appear in my calendar:

Google_Calendar_-_Week_of_14_Dec_2015

I have been able to add other services also covered by Sunrise as follows:

So while I was initially disappointed about the demise of Sunrise as it turns out I can have exactly the same functionality in my native calendar, Google Calendar, and there are plenty of mobile clients for it too so all in all I am pretty happy with the solution.

Installing ModSecurity & OWASP Core Rule Set on an Amazon EC2 Linux (CentOS) Instance

NOTE: This post has been updated to include the requirement of mod_unique_id.

As part of some investigations at work I have been playing around with ModSecurity, the open source web application firewall (WAF), and the standard set of rules provided by OWASP. All our infrastructure is hosted with Amazon AWS so I thought that it would be useful to drop down the steps I took to get this working on a bare bones Amazon Linux box.

Install ModSecurity

You can, of course, compile ModSecurity from the sources but it is easier to install via yum, however, it is part of the epel repo which isn’t enabled by default so you need to run as follows:

sudo yum install mod_security --enablerepo=epel

Now check that the ModeSecurity engine is on:

more /etc/httpd/conf.d/mod_security.conf

And look for the following line:

SecRuleEngine on

Download & configure the OWASP files

The Open Web Application Security Project (OWASP) has created a set of rules for ModSecurity – the OWASP ModSecurity Core Rule Set, whose “goal is to provide an easily “pluggable” set of generic attack detection rules that provide a base level of protection for any web application.”.

You can add these rules to ModSecurity as follows:

cd ~
 sudo wget https://github.com/SpiderLabs/owasp-modsecurity-crs/zipball/master
 sudo unzip -q master
 cd /etc/httpd
 sudo mv ~/SpiderLabs-owasp-modsecurity-crs-* modsecurity-crs
 cd modsecurity-crs
 sudo cp modsecurity_crs_10_setup.conf.example modsecurity_crs_10_config.conf

Next you need to ensure that the OWASP config files are included when ModSecurity loads. To do this edit mod_security.conf

 cd ../conf.d
 sudo nano mod_security.conf

And add the following lines in the section # ModSecurity Core Rules Set and Local configuration

 Include modsecurity-crs/local_rules/*.conf
 Include modsecurity-crs/modsecurity_crs_10_config.conf
 Include modsecurity-crs/base_rules/*.conf

The easiest way to do this is with the following SED command:

 sudo sed -i "s/Include modsecurity.d\/local_rules\/\*.conf/Include modsecurity.d\/local_rules\/\*.conf\n Include modsecurity-crs\/modsecurity_crs_10_config.conf\n Include modsecurity-crs\/base_rules\/\*.conf/" /etc/httpd/conf.d/mod_security.conf

Next you need to enable mod_unique_id Apache module which is required by mod_security:

sudo sed -i "s/#LoadModule unique_id_module modules\/mod_unique_id.so/LoadModule unique_id_module modules\/mod_unique_id.so/" /etc/httpd/conf/httpd.conf

Finally restart Apache:

sudo service httpd restart

If apache fails to start with a message similar to:

[alert] (EAI 2)Name or service not known: mod_unique_id: unable to find IPv4 address of “ip-x-x-x-x”

Add a line to your instance’s host file:

sudo nano /etc/hosts
 127.0.0.1 ip-x-x-x-x localhost

Testing that the rules are working

If you are lucky all this will work with your application without modification, although that isn’t guaranteed of course. The question is how do you safely test that the rules are working. One way is to try with this simple SQL injection attempt on the URL as follows:

http://yourdomain.com/?username=1'%20or%20'1'%20=%20'1&password=1'%20or%20'1'%20=%20'1

If the rules are working then you should be shown a 403 error. You can also check the httpd error log (/var/log/httpd/error_log) to check any messages written there by ModSecurity.

Things to watch out for

During my tests with ModSecurity I discovered a couple of things that you should be aware of. Firstly you need to be testing using a domain name and not directly using the instances IP address. Secondly, ModSecurity and CloudFlare don’t play nicely together. ModSecurity seems to take exception to the cookies that are placed there by CloudFlare. There maybe a workaround for this but none was immediately obvious to me.

Conclusion

Protecting your web application should be a high priority for all and ModSecurity and the OWASP rules give you a good starting point. The issue is that you are relying on a set of rules written by someone else that you (probably) don’t understand and won’t be certain if they are going to conflict with your application. They are also no substitute for ensuring your application itself is as watertight as possible. However, if you have nothing else and cannot afford a commercial WAF then they are a great addition.

WordPress, Cloudflare, Caching & Version Numbers

While I love WordPress one thing that really bugs me is its insistence on adding the version number to the end of resources, so you end up with links links this:

http://www.spokenlikeageek.com/wp-content/themes/required/style.css?ver=4.4

The version number is added to enable browser caching of files thereby speeding up page loads. The issue with this is if you make a change to the resource it is difficult to get the changes showing without clearing your cache. Additionally, as I am using CloudFlare this step by WordPress is superfluous.

All is not lost however, you can remove these version number by adding the following snippet of code to your themes functions.php file, ideally in the child theme.


function remove_cssjs_ver( $src ) {
if( strpos( $src, '?ver=' ) )
$src = remove_query_arg( 'ver', $src );
return $src;
}
add_filter( 'style_loader_src', 'remove_cssjs_ver', 10, 2 );

It is then a simple matter of either refreshing your browser to see the change or to remove the copy from CloudFlare’s cache. Go to the Caching section and click the Purge Individual Files button:

Caching__spokenlikeageek_com___CloudFlare_-_Web_Performance___Security

Then add the files you want purging from the cache in the box:

Caching__spokenlikeageek_com___CloudFlare_-_Web_Performance___Security

Give it a minute or so, refresh you page and you will no longer have the version numbers appended to the end of resources and you will quickly be able to see the results of any change syou make.

Source: https://wordpress.org/support/topic/get-rid-of-ver-on-the-end-of-cssjs-files

Pebble Time Long Term Review

So I have had my Pebble Time Steel a few months now so I thought that it would be good to revisit it and put down my thoughts having had it a while. The first thing is that I have now received my steel band and the watch looks so much better for it than the “leather” strap it was originally sent with. I’m really pleased with the look and it is a step up from the original plastic version (original Pebble) I had.

The Good and the not so Good

The watch interface has had an overhaul and there are a couple of new features that I really like. The first is the timeline. Pressing the top or bottom buttons on the right hand side of the watch cycles back or forwards in time through your calendar. Pressing the centre button then drills down to give more information on the calendar entry. This is so much more useful than just being able to change watch faces which was the option before.

The other useful change is to the left hand button. Previously this was just there to cancel any action or to move back a level. Now long pressing this turns on or off quiet time (or do not disturb as it is more commonly known elsewhere) which is great for when you are just about to go into a meeting. For some strange reason the image displayed on screen has been changed in the latest firmware from a bell to a mouse.

One thing that hasn’t change from the original Pebble is the font size. I really struggle to read the font on even the largest setting. With wearables becoming much more mainstream and more oldies such as myself wearing them this is something that is definitely going to have to change.

Battery Life and Steps

A year ago I published an article on the battery life and step counter in the original Pebble watch. Over the period of a few weeks I monitored both the battery life and number of steps that the watch recorded. This enabled me to see how close I got to Pebble’s suggestion of battery life and whether it could replace my Fitbit.

My original Pebble was getting about four days of life before I had to recharge it. The quoted battery life for the Pebble Time Steel is “up to 10 days on a single charge” but I never got close to that. With my usage I averaged out at comfortably seven days between charges. When you compare this to the competitors it is pretty decent I think. I was slightly disappointed that the old charger didn’t fit the new watch and that additional chargers are £15 a pop.

The graph below shows the battery remaining at the end of each day. You can clearly see that battery reduces in a very consistent way. I should point out that this is with a brand new battery in the watch. When I ran the tests before the watch was a year old so there would have been some reduction in the capacity of the battery.

Picture1

Since step counting was introduced to the Pebble I have long wondered whether I could use my watch instead of the Fitbit Flex I also wear. As the end of my last review of the original Pebble I concluded “unfortunately I just didn’t get the sense that the accuracy of the Pebble step tracker was as accurate as the Fitbit” so I was keen to run the same tests as before to see if this situation had improved at all.

As before I ran the tests both over a holiday period, when we tend to walk much greater distances each day, and when I was at work. Working from home means that I don’t get as much opportunity to do the steps as I would like (or probably should). This meant that I got a good mix of long and short distances to compare.

As before the Fitbit almost always recorded more steps than the Pebble (via the Misfit app) which was, on average, out by about 8.5%. This is actually down from last year when the average was 12.5%. However, as before I was put off by the fact that I need many more steps with the Pebble in order to do my 10,000.

The ultimate test, however, is to try and work out which is correct compared to the steps I actually take.

Picture2

There was one issue that I discovered that is currently being investigated by Pebble support. One of the new features of the Pebble Time is that it automatically manages watch apps for you unload and loading them as and when needed, much like Windows does with applications when it runs out of memory. This change gets round the old limit of eight apps but has an issue if the step tracking app is unloaded the steps are no longer counted. As I say this is being investigated by support presently and I hope that I can update you when they get back to me.

Conclusion

Since the original Pebble watch was launched the smartwatch landscape has changed significantly with many players entering the market, most obviously Apple. Being a big Apple fan many assume that I will have an Apple Watch and are somewhat surprised when I show them my Pebble Time. Had Pebble not released the Time then I would have switched to Apple but the Pebble wins out in a number of ways: price, battery life, cross platform compatibility and good third party support.

As I said before the Steel looks good and feels great on the wrist. I would have had to pay £650 more for an equivalent Apple Watch and I really don’t think it is that much better. In fact it’s not so I’ll be sticking with the Pebble.