Colin's Journal: A place for thoughts about politics, software, and daily life.
December 3rd, 2016
Freedom of movement and the single market
So, the Brexit referendum was a disaster with a slim majority voting to leave the EU. While leaving will significantly weaken Britain, there still remains the option of staying in the single market and retaining freedom of movement.
Why is this important? Most descriptions I’ve seen of the benefits of the single market completely focus on the benefits to businesses. While it is certainly a matter of great concern to businesses, I think there is a strong argument for individuals to be concerned about membership.
Here are my top ten reasons why the average person benefits from freedom of movement and the single market:
- Go on holiday anywhere in Europe, bring back anything you want to buy.
- Retire to your favourite place in Europe.
- Work in Dublin, Berlin or anywhere else in the EU for a day, a week, a month or years with no restrictions.
- Have the freedom to live anywhere you want in the EU.
- Study anywhere in the EU on the same terms as local students – often for free!
- Whether an eBay entrepreneur or small business sell to customers all over Europe with no barriers.
- Buy the best from wherever it’s made, in Europe or any of the 50+ countries with free trade agreements.
- Make your business a success – employ the best workers from across the continent.
- Keep your costs low with no import duties from Europe and beyond.
- Be safe selling and buying in the EU knowing that everyone is protected by common EU laws enforced by the ECJ.
June 21st, 2016
Voting to Remain in the EU
Chateaux reflected in lake
- Britain is part of Europe physically – the London Central Line is over twice as long as the narrow stretch of water separating Britain from France. The EU as an institution provides the framework for agreeing and enforcing the rules that let us live together as friendly neighbours. We need something to allow us to collaborate, cooperate and look after our shared neighbourhood – the EU provides that. If there are particular rules or aspects of the EU that we don’t like, being a member provides the means for us to change it for the better.
- I believe in freedom of movement. Being able to visit, work and live anywhere in the other 27 member states is a great privilege that I do not want to lose – I want to live where I choose in the European neighbourhood. I also want my fellow Europeans to have the right to visit, do business and live in the UK – it makes the UK a better place to live.
- An exit from the EU will adversely impact my family, my friends and colleagues. There are no meaningful benefits to leaving – indeed we’ll have many additional challenges. The only clear policy the Leave campaign have put forward is the restriction on freedom of movement, a policy I wholehearted disagree with.
January 23rd, 2016
Closures
I’ve found an increasing number of uses for closures in Go, to the point where they are one of my favourite features of the language. Closures are created when an anonymous function is declared and it references variables in the surrounding function, for example:
func main() {
a := 0
b := func() {
// a is shared between the main and anonymous function
a++
}
b()
// Value of a is 1
fmt.Printf("Value of a is %v\n", a)
}
Functions are first class types in Go, so a function can return a function that uses the variables and arguments of it’s parent. Such functions are a good alternative to producing a number of low value traditional struct{} types with a single method.
Closures also help in producing loosely coupled functions that serve as end points for a web service. Dependency Injection is used to provide the services required by the end point in a clearer way than defining end points as methods on a struct that holds all of the potential dependencies.
Closures are also good for creating iterators / generators, which don’t come up that often, but are a good way to encapsulate the complexity of navigating a complex data structure. By passing functions to functions that return functions, it’s possible to create simple micro-languages that make filtering and processing data records much easier to think about.
Finally closures enable in-line inversion of control in a very natural way. I’ve found this particular useful for iterating over SQL queries, significantly reducing boilerplate code and improving consistency of error handling.
While I’m clearly a big fan of using Closures in Go, you do have to be careful about exactly which variables you are capturing in a closure. For example:
func loop() {
var funcs []func() int
for i := 0; i < 5; i++ {
funcs = append(funcs, func() int {
return i
})
}
for _, f := range funcs {
fmt.Printf("Func value: %v\n", f())
}
}
The closures being created in this loop all capture the same variable i that is shared between them. As a result the output of the function is a series of 5s – the value of i at the end of the loop. If we create a new variable on each iteration of the loop and capture the value there, we get the desired behaviour of printing out 0,1,2,3,4:
func loop() {
var funcs []func() int
for i := 0; i < 5; i++ {
j := i
funcs = append(funcs, func() int {
return j
})
}
for _, f := range funcs {
fmt.Printf("Func value: %v\n", f())
}
}
October 4th, 2015
HTTP Push Notfications
Arran Beach
Owlauth allows users to register a device (e.g. a phone) and then use this device to confirm their identify when logging into a website or application. For this to work the Owlauth app running on the phone needs to be able to receive push notifications of authentication requests. There are many centralised ways to push notifications to a phone, but as each domain owner can run their own Owlauth server, a centralised solution isn’t a good fit.
By taking advantage of Go’s cheap handling of concurrent processing with Goroutines, a simple HTTP based push approach can be used. Clients connect to the server:
- Passing a Bearer token that authorizes them as a registered device.
- Passing an If-None-Match ETag if it has one.
- The HTTP GET then blocks until either:
- The HTTP Keep Alive timeout is approaching – in which case the server returns nothing.
- A new authentication request is available or an existing request goes away.
- When the GET returns, an ETag header reflecting the current state.
- The client loops, issuing a new HTTP GET request with the latest ETag.
The server returns before the Keep Alive timeout, which means that the client will reuse the same TCP (and SSL) session as for the first request. This makes this timeout and GET operation an effective ping that proves the connection is still established. If the TCP connection becomes invalid, the client’s HTTP library will open a new connection to the server, giving the desired reconnect behaviour.
The server implementation in Go is straightforward:
var requestedEtag string
requestedEtag = r.Header.Get("If-None-Match")
authChannel := reqStore.RegisterListener(dev.DeviceOwner)
defer reqStore.UnregisterListener(dev.DeviceOwner, authChannel)
clientGone := w.(http.CloseNotifier).CloseNotify()
var request *reqdb.PendingRequests
timeout := time.NewTimer(DeviceRequestGetTimeout)
for {
select {
case req := <-authChannel:
if req.GetEtag() != requestedEtag {
request = &req
// Return the ETag
w.Header().Add("ETag", req.GetEtag())
return request, nil
}
// Just loop and try again.
case <-clientGone:
// Nothing to be done - just return
log.Printf("Device gone - returning\n")
w.Header().Add("ETag", requestedEtag)
return nil, nil
case <-timeout.C:
log.Printf("Device request get timeout reached.\n")
w.Header().Add("ETag", requestedEtag)
return nil, nil
}
}
The reqStore object keeps track of outstanding authorisation requests and provides the details on a channel to all registered clients that are sat waiting in this GET.
The only other element of the equation on the client side is adopting a suitable retry strategy when http connections are not working. For desktops that could be just a simple back-off to a few seconds sleep. For the Android client it needs to taken into account the current network state to avoid excessive battery drain.
I’ve now got a basic version of this working on my phone. I’ll run it for a while and see how much battery impact it has.
September 8th, 2015
Killing Passwords
Passwords on websites and in apps are the bane of internet usage. Much has been written (recently in TechCrunch – Kill The Password) on how painful they are to generate and remember. Password managers help with the challenge, but are a cumbersome band aid.
When building a website application, adding username and passwords is also painful, requiring extensive work to get working well and securely. So, what are the alternatives? There are some good ideas out there, but they tend to be complicated (OpenID Connect), or put large identity providers in a special position (Fido, OpenID Connect) or are centralized and cost money (e.g. Clef).
I think it’s possible to build a simple, distributed, secure authentication mechanism that allows users to login to sites without generating passwords. The vision is that a user can authenticate themselves easily:
- Enter your email address
- See a pass phrase in the application / website
- Check your phone – if the same pass phrase is displayed, tap authenticate and you are in
I’ve been experimenting with how this could be done, documenting the specification on an Owlauth Github page, and writing an authentication server in Go that implements the specification. It’s not done yet – the current code sends email to the user with a link rather than a notification to a phone – but it’s close enough that it proves it can work.
In addition to the specification, I’ve also got a test application running (A larder app for tracking food best before dates) that authenticates using this method.
The challenge now is how to move this forward. Finishing the implementation to allow device based authentication is straightforward (I’ve got most of the code done), but it’s of no use if there isn’t a community of developers interested in deploying it.
June 13th, 2014
Golang
Venetian Boats
It’s been a long time since I set out to learn a new programming language. All of my development recently has been in Java (Android) and Python (Django), both of which I’ve known for years. Although Go hit version 1 in 2012, it is only in the last couple of weeks that I’ve started to pay attention to it.
The number of people talking about moving to Go from Python was a big influence to try it out. I’ve been using Python for a long time and it remains my favourite language at this point. After my first try at Go I can see the attraction to Python programmers. I started by doing a short script to load in some values from a CSV file, process them and write out to another CSV file.
This is the kind of thing that I find Python perfect for. It takes very little time to write thanks to the built in library, no effort to setup the environment as it just requires a single text file, is easy to debug thanks to the edit / run cycle being so short, and so gets you to a result really quickly. The experience with Go was somewhat similar. You do need to setup the environment for the first time, but the go run command means you can still have a short edit / run cycle. Like Python, the CSV library is built in and the resulting code is mostly free of boiler plate variable definitions, etc.
I’ve seen Go described as being 90% as expressive as Python. From my first encounters with it, that seems to hold true. There are a few tedious type casts required that Python avoids, but overall the feel is much closer to Python than to Java or C. The 10% given up buys a compiled, easy to deploy program that executes fast and can be more easily scaled than Python.
My initial experiments with goroutines and building a prototype TCP server resulted in compact code but with more type wrangling than I’d like. Having to think about when you are using values versus references (maps, slices and pointers) resulted in some subtle bugs only found in unit testing. This is probably down to too much time spent in Java and Python.
There are a lot of conveniences in writing Go. The source comments generated documentation system is trivial to use (far simpler than Javadoc) and the builtin unit testing framework is also very easy to use. The ability to have package level variables accessed from multiple files in the same package makes it very easy to build a modular set of code.
I haven’t tried any true OOO programming with Go yet. I’ve read much concern about the lack of generics and limited inheritance support. It’ll be interesting to see how much pain that causes in my code.
May 12th, 2013
Mileage in ExpenseClam
In previous releases of ExpenseClam I’ve introduced two major new features: mileage and attachments. Of the two, mileage proved to be the easier to do within the existing application structure.
When designing the mileage functionality I decided early on that I wanted to make mileage work the same way as other types of expense as closely as possible. Mileage expenses would appear in the same list of unclaimed expenses, be exported in the same spreadsheet file and be moved into claims in the same fashion.
The easiest way to do this would be to have a mileage specific screen that generated expense records using exactly the same format as normal expenses. This approach is used by a number of competitors and is very lightweight to implement.
The downside to this design is that once a mileage record has been entered it loses mileage specific fields (such as distance, mileage rate, etc). I decided on a slightly more involved way of solving the problem: extending the expenses records to have mileage specific fields and creating a specific set of screens for creating and editing mileage records.
The code that lists expenses required only minor tweaking in order to distinguish between the two record types. The export code required additional columns in the spreadsheet output for mileage records, and code for managing expenses in claims was unchanged.
When a new or existing expense is opened ExpenseClam determines which UI is required and launches the appropriate Activity. Both sets of screens have a common super class that contains all the common logic for managing the life cycle of creating, discarding and saving expenses. Code for handling receipts is restricted to the expense handling class, while distance calculation logic is restricted to the mileage class, hopefully making maintenance easy.
If I had dropped support for Android versions earlier than Honeycomb I could probably have maintained a common activity and used separate fragments to achieve the same effect. However to support Gingerbread and earlier I have most of the Actionbar logic in the Activities, along with code for coordinating interactions between other Activities.
May 5th, 2013
SwiftKey and trailing spaces
During testing of the latest version of ExpenseClam (released just last week), I stumbled across a small issue that I’ve not seen before.
Until recently I was using a back port of the Jelly Bean keyboard on my phone. I really like the swiping of characters implementation and find the auto correct to be rather good, although the prediction is rather limited.
Since SwiftKey released their version of character swiping (called SwiftKey Flow), I’ve been using it on my phone, and it’s generally good enough that I’ve not switched back to the Jelly Bean keyboard.
Now on to the issue: trailing spaces. Having swiped across letters to form a word, SwiftKey populates the word, and adds a space ready for entering the next word. Unfortunately this space isn’t just for show, it is populated in the widget value, so when the contents of the field are saved it includes the trailing space.
In general this is mostly harmless, however if you spot the trailing space and delete it, you can end up with two database records that look the same, but are actually different. This makes auto complete on fields less useful and potentially confusing.
The fix is easy enough, I just trim the string before saving to the database, but it would have been better if SwiftKey had used the same approach as Google and inserted the space at the start of a swipe rather than at the end.
November 26th, 2012
Android device compatibility
It took me two days of digging around to finally figure out why ExpenseClam was not working properly on the HTC One X. The symptoms were straightforward enough: taking a picture would show a blank result, and the expense record showed a warning icon saying that the image file could not be loaded.
By adding some debug cpde, and with the help of a friend who owns the device, I was able to confirm that the camera code was creating an image file and that it was successfully saved. By signing up to testdroid.com I could get access to the logcat output of a real device on the cloud, which gave me the next breakthrough. My call to BitmapFactory.decodeFile was failing to return a bitmap, in turn caused by SkImageDecoder returning null.
Searching online sent me chasing a few red herrings, but eventually I had the idea of removing the code which sets the thumbnail size before taking the photo. Suddenly it worked, the bitmap correctly loads. So what went wrong?
The API documentation for setJpegThumbnailSize is fairly unambiguous “If applications set both width and height to 0, EXIF will not contain thumbnail.” Further to this the method getSupportedJpegThumbnailSizes states “This method will always return a list with at least two elements. Size 0,0 (no thumbnail) is always supported.” Based on this I had chosen to disable thumbnails in order to reduce image file sizes.
It turns out that only most devices support this. The HTC One X accepts setting the thumbnail size to 0, but then produces a file that the BitmapFactory cannot read. Leaving the thumbnail settings to the device defaults works.
Hopefully this post will save someone else a couple of days investigation.
August 12th, 2012
Nexus 7
It’s been a long time since my last post, with the birth of our daughter taking up all of my nonworking time. I have however purchased a Nexus 7, now being able to see how the 7″ tablet running android compares to the playbook.
I haven’t yet tried travelling with the tablet, but for use at home, it blows away the playbook for usefulness. The experience is not perfect by any stretch. The BBC are yet to provide an iPlayer app that works without flash, so you have to use workarounds to watch TV. The Economist, fantastic publication though it is, have produced a rubbish android app that doesn’t support ICS, never mind Jelly Bean.
Putting to one side these niggles, most of the important apps are there and work well. The chrome web browser provides a good experience, with rendering much faster than the playbook. Games play fluidly on the Tegra 3 powered machine, with the vibrant screen providing an almost SciFi feel in such a slim form factor.
The keyboard is good, certainly the best soft keyboard I’ve used. It is a little sluggish at times, with some key taps not registering as well as you would hope. Strangely tapping quickly seems to work better than trying for accuracy.
While the bundled Transformers movie is a terrible film, it does show off the tablet to good effect, with the movie streamed in HD, and looking very crisp on the high resolution screen. It also further justifies having an unlimited broadband package as the one film weighed in at over 5GB of streamed data. Uploading my own DVDs was fiddly from Linux, but should be straightforward from windows or Mac OS.
In conclusion, while the Nexus 7 may not be a substitute for a laptop in the way that an Asus Transformer could be, it makes a great content consumption device. If the alternative for content creation is your phone, the 7 inch screen works far better, while retaining a lot of portability that a large device sacrifices.
The killer feature for me is the weight. Holding the tablet for extended periods is as comfortable as holding a paperback book, possibly more so. The Nexus is the first device for which I reach for most online tasks, with the desktop relegated to development and photo editing.
Software
The full list of my published Software
Email: colin at owlfish.com