Alex Bligh's blog

Alex Bligh's personal blog

A year or so ago, Nominet‘s board commissioned Sir Michael Lyons to perform an independent external review of Nominet’s operating model and governance arrangements. Sir Michael reported to the board in October, and the board have now released both Sir Michael’s report, and their response.

I spoke to Sir Michael briefly at the last AGM, and at his request had a longer telephone conversation. I have to say I was impressed – he seemed to have got to the heart of the issues pretty quickly. And he appears to have produced a very sensible report.

Sir Michael’s recommendations are to be found at the end of his report. In summary they are level-headed and reasonable. None are particularly radical, and I think he is probably correct that radical surgery is not needed. Nominet appear to have accepted most of them, although I do find it a little strange that they don’t accept the need for finance director on the board of a company Nominet’s size.

Given I agree with almost everything Sir Michael has written, I’m not going to pick the report apart in full here. But I will mention two details.


Firstly, Sir Michael suggests (page 19):

Introduce clear KPIs for cost control and return on Research & Development

I think introducing KPIs is a valuable strategy, particularly regarding cost control. However, I question the extent of the usefulness of introducing KPIs for ‘return on Research and Development’. Long term R&D produces returns only over the long term; by then it’s too late to control the cost of that R&D. I think maintaining a close eye on what is researched is probably even more important than what it costs. Moreover, in any R&D environment it should be an expected result that whilst some projects will produce transformative commercial successes, some (perhaps most) projects do not come to commercial fruition; accepting that this is an inevitability and not a failure is vital, not least as otherwise staff have the perverse incentive to carry on with such projects.

In his recommendations section, this has been tempered to:

Recommendation 18: Nominet should make public the KPIs by which it holds the executive to account reflecting at the minimum registry costs and progress with diversification

which I believe is a better view of things.

The Nominet Trust and Nominet’s public purpose

Sir Michael makes a number of wise remarks about the Nominet Trust and Nominet’s public purpose. Here are a couple of quotes:

From page 9:

It is not enough to argue that Nominet fulfils its wider public purposes by making a profit or that the bigger that profit, the bigger the social benefit. Nor, for that matter, that it meets its social responsibilities by donating some, or all, of its profit to charitable purposes.

From page 12:

However, there is one point that I would like to underline and that is the importance of taking a wide view of social benefit and so avoid focusing solely on welfare benefits. There may be a danger that this has marked the early days of the Nominet Trust, where the board appears to have put an emphasis on separation and independence for the new Trust (both important issues for charitable status to be secured) but, perhaps, inadequate consideration of purpose.
Much of what the Trust has undertaken appears to be valued by the beneficiaries and other commentators but does not appear to be widely understood, or valued, by the membership. In part, this may be remedied by clearer communications in the future, and that is certainly on the agenda, but I believe it may also offer some lessons for the definition of the company’s wider purposes. Lessons, in terms of both the importance of clearly-defined purposes but also of ensuring that they are based on a wide view of social benefit. Most crucially, the interest of the original founders in establishing Nominet as a company capable of contributing to the further development of the internet was, itself, a clear purpose of social benefit. Whilst I believe that objective now needs to be revisited and, perhaps, broken down with a set of purposes reflecting the company’s current understanding of the internet and the wider digital economy, I strongly encourage the board to give weight to objectives which offer economic as well as social benefits. Not least, because these are likely to be more appealing to the membership.

I think Sir Michael has these points exactly correct, though as they did not find their way into a recommendation, they board did not respond to them. Donating money to the Nominet Trust is laudable, but does not mean that by doing so Nominet has automatically achieved its public purpose solely by doing this; public purpose should run through its operations. Similarly, Nominet sometimes appears to want to wash its hands of the money once donated (perhaps in order to ensure the Nominet Trust appears to be independent); whilst I agree that Nominet should not involve itself in day to day decisions of the Nominet Trust it should ensure that the Nominet Trust is applying its funds in a manner consistent with Nominet’s own public purpose. Funding more (charitable) projects directly related to internet infrastructure, for instance, would not go amiss.

Nominet has announced that it is to increase its prices for UK domain names.

The announcement states in essence that prices will rise from a minimum of GBP 2.50 per year per domain (i.e. GBP 5.00 for two years – the same per annum for longer periods) to a minimum of GPB 3.75 per year per domain, which is a 50% price rise (assuming one was previously renewing each two years). Nominet note that the price hasn’t changed since 1999, so this is equivalent by my calculation to a (compound) 6% per year price rise. The cost increase is then potentially reduced by new co-marketing programmes. Note that the one year registration price was already GPB 3.50 per year, but that’s a relatively new introduction; if you were renewing domains this way, the price increase is smaller.

I’ve been asked what I think about this, and specifically I’ve been asked to sign this petition, which (as far as I can tell) is calling for an EGM of the company to vote on the price changes and some form of consultation. I’m against the former, but in favour in principle of the latter (for the reasons set out below), but as such I won’t be signing the petition.

Those hardest hit by the price rise are those maintaining large portfolios of domain names where the domain names fees are a high percentage of their cost base. Most ‘normal’ domain name registrants won’t give two hoots if the price of their domain name increases by GPB 1.25 a year, or even five times that. But those whose business relies on keeping these portfolios in order to speculatively sell a fraction of them, or to attract traffic (and thus ad revenue), are going to be affected significantly. Let’s call this group of people “domainers” (although some don’t like that title). The EGM petition appears to have been started by domainers, and signed by many domainers. In many quarters of the industry, domainers are not a popular group. My personal view is that it’s a legitimate business model (if not one I want to be involved in) provided IPR is not infringed, no consumers are deliberately confused, no animals hurt during filming etc.; but others have different views.

Nominet’s handling of this issue has been a mess. However, so poor has Nominet’s handling of this issue been that it has succeeded in getting several people to sign this petition who normally would have nothing to do with domainers.

Here’s what I think and why (skip to the end for a summary):

  1. Prima facie, Nominet should have the right (somehow) to change its prices. It’s not reasonable to expect a supplier to maintain the same prices ad infinitum. The question is how.

  2. Those who construct business models which rely on a single supplier for a huge percentage of their cost base, where that supplier has the freedom to change its prices, need to educate themselves on business risk. In this instance, they can renew at the old price for up to ten years (until the new prices come into effect), which will clearly have a cash cost. However, this was a risk that should have been evident from the point Nominet began (certainly since 1999). I’m afraid I have no special sympathy here.

  3. All of the justification for this price rise appears to have looked at supply-side issues, i.e. how much it costs Nominet to register a domain. Let’s briefly look into that. As far as I can tell (and as I raised at their last AGM) their average cost and marginal cost per domain appear both to have risen reasonable substantially since I was on the board many years ago. Whilst I accept that there must have been some inflation pressure (e.g. in wages), and the need to maintain an infrastructure handling more load, technology prices have fallen and processes should have been automated. The latter point is why the average wage at Nominet should have (and has) risen; because it should be employing fewer (relatively highly paid) people designing automated systems, not an army of (relatively low paid) administrators doing things manually.

  4. However, despite Nominet’s emphasis on the above, I suspect the real issue is that buried within the accounts are the costs of doing lots of things that do not directly involve .uk registrations. Nominet is attempting to diversify. This might be good, or it might be bad. But Nominet should have been clear as to how much of the increased cost is going towards expenditure in servicing .uk domain names, and how much for other purposes such as increased costs elsewhere (e.g. diversification), or building up reserves (increased revenue without increased costs). Nominet hasn’t published any figures, so we don’t know.

  5. The unexamined side of the equation is demand-side. As far as I can tell, Verisign’s wholesale price is $7.85 per annum (GBP 5.85 per year), and that’s for a thin registry (where the registry provides far fewer services, and the registrar far more). Clearly on this basis Nominet’s prices are and will remain well below market level. It would thus seem that Nominet is providing a fuller product (perhaps a better product) at a far keener price than its main competitor, despite have fewer economies of scale. And it is a product generally loved in its target market (the UK). Why on earth Nominet didn’t use this as the centre-point of its argument, I don’t know.

  6. I don’t think as a general principle price changes should have to be put to a vote of members. This is how things were (for a while) whilst Clause 19A (the ‘Hutty Clause’) was incorporated into Nominet’s articles. I was and remain in favour of its removal. Having members vote on every price change encourages the perception that Nominet is some form of cartel, and fetters the discretion of the directors. It also makes changing prices an unnecessarily difficult business, meaning it is hard for Nominet to respond to changes in the market place (arguably this can’t have been too much of a worry given the number of years without a price change since it was removed). But Nominet is a commercial organisation, not a golf club, and therefore its pricing should be set by its management.

  7. However, there is the question of how the management should set the prices, i.e. what objective are they attempting to achieve? Verisign is a public company, and its directors set prices to maximise profit in the long term. Nominet cannot distribute its profit to shareholders, so how should it set its prices? Should it too maximise profits? For many years the principles were long term cashflow neutrality, long term P&L neutrality, and maintaining a sufficient reserve for legal challenges and market downturn; these were called ‘the Bligh principles’, because (cough) I came up with them, and they seem to have survived a long while, for better or for worse. Prices were then meant to be set to accord with these principles. Some would argue the principles are still relevant, some would argue they have problems (I have a foot in both camps). But the point is that there were transparent principles that everyone knew about, and if they didn’t agree on them, well, they didn’t in general have a better suggestion.

  8. I am of the view that any change to these guiding principles should be carefully and transparently consulted upon; this is not because I’m particularly attached to the principles above, but because deciding which principles drive Nominet’s behaviour is a key matter of governance. Note this is a different matter to a change in prices (following the guiding principles); I’m happy to leave that to management provided they explain how the change better satisfies the guiding principles. If Nominet don’t publish these principles, or an explanation of how a price change better satisfies them, there is no way members can hold them to account. And whilst I recognise members can occasionally be a pain, there is no one else who can hold Nominet’s management to account. Quite apart from that, transparency is in itself a good thing. As is avoiding the appearance of something that might be problematic to the competition authorities.

  9. What appears to have happened now is that there are no guiding principles, or at least none that we know about. The suggestion that prices are set according to cost recovery principles (never particularly felicitously worded) is simultaneously being removed from the terms and conditions. Is the principle now profit maximisation? If so, please come out and say it. Is the principle now ‘whatever the management feel like’? That is not in my view acceptable. But the principles seem to have disappeared. Prices appear to be being set on the basis that ‘Nominet think they should be higher’. If this is not in fact the case, then Nominet has a communications problem.

  10. Lastly, there seems to have been some bizarre criticism of the co-marketing programmes proposed. The objection is that those who register the most domains get the most co-marketing, and that this is unfair. It seems to be me self-evident that those who register the most domains should get the most co-marketing funds, as they are meant to be put towards registering domains. Rather, my problem with them is that the co-marketing funds for the larger registrars are too small. How do I work that out? From the site calling for an EGM: ‘Registrars with over 250,000 domains under management can now claim up to £80,000 per registrar. Smaller registrars with under 5000 domains can only claim £2000.’ This completely misses the point. For an organisation with 250,001 domains, Nominet’s providing GBP 0.32 per domain back, reducing the price for that year to GBP 3.43. For an organisation with 4,999 domain names, Nominet’s providing GBP 1.25 per domain back, reducing the price for that year down to GBP 1.25. Or to put it another way, if you have 4,999 domain names as a registrar, and claim your full co-marketing allowance, you will be far better off than before (even if the number of domain names stays the same); if you have 250,000 domain names, you will be worse off than before, unless you increase the number of domain names you sell quite substantially. Every co-marketing program I’ve seen before scales in the other direction – i.e. the larger you are, the better deal you get per item sold. Rather than a bulk discount, Nominet is applying a bulk penalty! Whilst I am sure it has its reasons for this, I have no idea why smaller registrars are complaining it’s unfair on them. Of course not all registrars may be eligible to apply, but that’s not dependent on the size of the registrar. And the co-marketing is presumably directed at generating new registrations rather than renewals (this is co-marketing, presumably meaning it is dependent on Nominet related marketing spend from the registrar); whilst that may hit those with domain portfolios they are not growing harder, that’s not dependent on size either, and is presumably a desired result (encouraging people to grow the number of domains under management as opposed to merely renew an existing portfolio).

So, back to that petition:

  • Yes, Nominet should have (and should now) consult on any change to its pricing principles, and not change the prices until it has done so; but
  • No, Nominet need not consult (let alone have a vote) on the price change itself

But I think it unsurprising that people are annoyed.

Golang complained today:

RSA modulus is not a positive number

I wondered whether this was a fault with our cert(s) or whether golang was loading it properly. As there was no canonical resource on the net for how to diagnose it, her’s what I did (some anonymity applied to the cert):


openssl asn1parse -i -dump -in /path/to/certificate | egrep -A6 -B1 :rsaEncryption

which gives these lines:

289:d=3 hl=2 l= 11 cons: SEQUENCE
291:d=4 hl=2 l= 9 prim: OBJECT :rsaEncryption
302:d=3 hl=4 l= 270 prim: BIT STRING
0000 - 00 30 82 01 09 02 82 01-00 cb 4a 13 93 bf 3e 91 .0........J...>.
0010 - 89 08 4e 85 fb e0 2a b8-9e 1c 7f d3 1a eb 34 77 ..N...*.......4w
0020 - 6d 8a 0c 1d d9 13 70 40-ba be 5f 77 2f a1 88 66 m.....p@.._w/..f
0030 - fd 2e ef 14 3f 1d 36 ff-df 23 1c 6a a5 f1 ae fb ....?.6..#.j....
0040 - e6 9a 0a 8b dc 53 5d f4-7a be 0c 27 38 76 2f 27 .....S].z..'8v/'

That tells me that the rsaEncryption bit string starts at byte offset 302. So now I can look at it like this:

$ openssl asn1parse -i -dump -in /path/to/certificate -strparse 302
0:d=0 hl=4 l= 265 cons: SEQUENCE
4:d=1 hl=4 l= 256 prim: INTEGER :-4189AC6C40C16E76F7B17A041FD54761E3802CE514CB889275F3E226EC8FBF4541A088D05E779902D110EBC0E2C90020DCE3955A0E51041965F57423ACA20B8541F3D8C789D0D812AD2ED8006D99EE533A7000AB4FAFA61F46CE80E0CE46D1B4DB88383511FFCE3727D09EEDBE9D479C5FD9AD858580E44A0C9606483B1C7A248B30A7AE1E8CA7F679949AAAB961611EE677CA7DFF59EA8FAF7A6849FC28695B0CE6547CDB48633BC81EC419BB30DC53928C84946BBE5E8E8506971A184C4956E4FC8B4DCA53936B2F1E19937856673E9FF8D29746F41CA6F3A56E15A34F8C46785A06EC77773CD8A8E43EF5EC9EA6887530E4790DEDBED534033B0D1379E9DA
264:d=1 hl=2 l= 3 prim: INTEGER :010001

The two INTEGER values displayed are the RSA modulus and the exponent (65537 in this case).

But the modulus shown is negative (see initial - sign on the first line representing an INTEGER), which is contrary to RFC3447, which means golang complains, even if openssl s_client does not.

Note that if you attempt to display this with openssl x509 -modulus, it won’t display the fault (i.e. it will display the two’s complement hex value rather than the negative number, so you have no idea of the problem).

Conclusion: the certificate is broken, and golang is (technically) right to complain. Apparently the issue was caused by it being generated by a buggy version of gnutls. Solution: Regenerate the certificate.

A quick note on how to do this, partly so I don’t forget.

I used to think (and Google says) that the answer is to compare the output of the following two commands:

cat /path/to/public/key | openssl x509 -noout -modulus
cat /path/to/private/key | openssl rsa -noout -modulus

Links may suggest md5sum on the output to make the moduli easier to compare.

Well, this is all very well as far as it goes, but it won’t work with ECDSA certificates. Here’s a much more reliable way that works with all certificates, assuming you’re using a modern OpenSSL:

cat /path/to/public/key | openssl x509 -noout -pubkey
cat /path/to/private/key | openssl pkey -pubout

The openssl pkey command actually extracts the public key from the private key. You can then compare them (or their MD5 if you pass them through md5sum).

My Three Home Signal device has now been down for eight days. Despite their twitter support people (@ThreeUKSupport) saying periodically that they think the problem is fixed, my understanding is that this is pretty much a network-wide outage. Ringing the support number for 3 Home Signal (0800 358 4828) leads to a recorded message saying “We know you’re having problems with your Home Signal device …” and if you are prepared to wait for 20 minutes on hold for their call centre, they admit this is a network-wide problem.

Despite living reasonably centrally in London, I have no signal at home (well, occasionally one bar from the top of the house). So I rely on the Home Signal device to get any signal at all, as do the three other people (yes, glutton for punishment, I have four accounts with Three).

Now, I’ve worked in reasonably senior positions at telcos, and the only network-wide faults I have known that have taken eight days to resolve are serious physical problems, such as cable breaks, sensible telcos put in redundant cable systems to prevent these affecting service. If I had a twenty-four hour problem on my network, I’d be seriously embarrassed. But embarrassment aside, I’d be communicating to my customers what the problem was, and what my plan is to fix it. And that’s what Three are not doing.

Rather than admitting they have a problem, they are telling their customers that the problem does not exist or has been fixed:

  • Their support team is telling people the issue is fixed.
  • Their PR team is telling people the issue is fixed (see update on story, followed swiftly by another update)

However, as the rapidly flashing green light on my Home Signal box will attest, it hasn’t been fixed.

Calling the Home Signal support line results in a 20 minute wait to get through, followed by an admission (if pressed) that they have a network wide outage, that they have no idea of the cause (at least that they are prepared to reveal), and that they have no idea of an estimated time to fix. Calling the normal support line results in a suggestion you call the Home Signal support line.

Attempting to escalate the complaint led to the following comic list of events:

  • Long pause whilst transferred to supervisor
  • Supervisor asks whether she can put me on hold to find out more
  • I say yes, which results in the call being dropped
  • Supervisor calls back (on the mobile, which doesn’t work, obviously)
  • I receive a text message asking me to call 0800 358 4916 to talk through my complaint
  • That number is no longer in service (perhaps it has worn out), and leads to a recorded message, suggesting I call the normal support line

Oh, and complaining to @ThreeUKSupport merely leads you to be asked to fill in a form. Someone then rings you back, or attempts to (no luck given I, um, have no signal), who leaves you a voice message and tells you to ring the support line.

If you complain hard enough, they will refund your bill pro-rata – which on my larger tarriff (£45 a month) was a whole £8. Given I’ve spent at least a couple of hours on the phone, that hardly compensates for my time complaining. Notwithstanding, I suggest you do this, not because it’s worth getting the few quid back but because the administrative hassle of processing it might make them pull their finger out.

This is not how successful companies communicate. Put something up on your website. Proactively communicate that you have a problem, and what you are doing to fix it, and people will love your customer service. Pretend you have fixed the problem, send people around and around endless chains of telephone numbers, and they will think you are a bunch of incompetent muppets. And on balance, it seems they’d be right.

In the event, I got more information from Wireshark than I could get from Three. A little playing with packet capture establishes that the boxes (white labelled Ubiquisys, now part of Cisco) successfully boot, make NTP and DNS queries, download their config (when hard reset), successfully negotiated TLS to a three server, then set up and maintain an IPSEC tunnel to somewhere in Three. And indeed one engineer successfully remotely logged into my box and rebooted it. So the signs are they aren’t suffering from a DoS attack or anything similar. My understanding is that they then run SIP over this tunnel. Presumably this layer or upwards is not working.

So, Three

  • Fix your network
  • Fix your communication failures
  • Give the customers who are suffering some gesture beyond a couple of miserly quid.

Epilogue #1

Today Three finally fixed it (11 days later). Here’s how I got them to do it.

  • I rung 0800 358 4916 on a weekday, which is the complaint line, raised merry hell and asked for the fault to be escalated. They credited me 2 weeks’ line rental on each account at this point.
  • A few hours later, they rung me back on a landline and I got put through to their Home Signal team (without waiting 20 minutes on hold).
  • They asked me to reset the box which unsurprisingly did nothing.
  • An hour later they rung back to see if it was working (no), and said they were going to delete the registration from their system and set it up again from scratch. I should then reset the box.
  • Ten minutes later it was working. They rang back to check after a bit.

Why this simple reset could not have been done at any point during the 11 day outage remains a mystery.

Epilogue #2

Obviously that was two easy. It worked for an entire evening, but only on my phone (there are 3 more on the box). This morning it was trapped in a fugue state of apparent rebooting (red light, rapidly flashing green light, no light, red light, rinse and repeat).

Now Three are replacing my white home signal box with a black one.

One of the interesting aspects of the 2015 UK election is the disparity between votes cast and seats gained. The table below shows (for the parties that actually gained seats) the number of votes cast, the seats gained, and the votes per seat. It then goes on to show how, if those seats had been distributed amongst those parties* proportionate to their vote, how many seats they would have got, and what change that would produce.



Two things are immediately apparent.

First, some parties had to work much harder (i.e. gain more votes) for each seat than others. UKIP needed 3.8 million votes for each seat, the Greens 1.2 million, and the Liberal Democrats 302,000. Looking at the two parties that did really well, the Conservatives needed 34,000 votes for each seat, and the SNP a paltry 26,000.

Secondly, the make up of a parliament elected proportionately would be very different. The Conservatives would still comfortably be the largest party, but not have a majority. Labour would have done still worse. UKIP would have a third as many seats as the Conservatives. The Lib Dems would not be wiped out. The SNP would have around half as many seats as they do today.

It’s hard to understand the democratic legitimacy in a party polling 1.4 million votes (the SNP) getting 56 seats, whereas a party polling 3.8 million (UKIP) gets 1. Or of the Lib Dems and the DUP gaining the same number of seats (8) when the Lib Dems polled 2.4 million votes and the DUP 184,000.

The main beneficiaries of a move to proportional representation would be UKIP. Whilst they would be last on my ballot paper, it’s difficult to maintain that the current voting system is fair.

* I redistributed the seats amongst those parties that won at least one seat under the current rules; in a true proportionate system, the results might be slightly different.

I had an interesting situation today where qemu-img create performed oddly when using one particular NFS filer. The symptoms were:

  • with -f qcow2 it worked as expected, and a 500G image is approximately 1MB
  • with -f qcow2 -o preallocation=metadata, the image took hundreds of gigabytes

This is not meant to happen. The files are meant to be sparse (i.e. have holes in them). The metadata preallocated is pretty small. On any other NFS filer I’ve tried, and on any local filing system I’ve tried, this works as expected.

I therefore needed to narrow the problem down, and armed with an strace of what qemu-img was actually doing, I built sparsetest which creates a sparse file.

The results are pretty interesting. Here’s a normal ext4 volume.

$ ./sparsetest -b 4K -s100M -w1M test
  Intended logical size:       104857600 bytes;             100 M;           25600 blocks of 4096 bytes
  Optimum physical size:          409600 bytes;               0 M;             100 blocks of 4096 bytes
   Actual physical size:          409600 bytes;               0 M;             100 blocks of 4096 bytes

Used 100 writes of 4096 bytes every 1048576 bytes in ascending order
Created 800 512 byte blocks on disk
Density as % of actual physical size over logical size: 0.390625 %
Efficiency as % of optimum physical size over actual: 100.000000 %

What I’ve asked it to do there is write a 100MB (logical size) file, and every 1MB of data, write 4K of random junk. At the end of writing the file, it uses ftruncate to set the logical extent to exactly 100MB.

So the 100 4K sections of random junk are the only content in the file. The file is gratifyingly 409,600 bytes long – exactly as it should be, so 100% efficient at encoding the sparse nature of the file. And it’s 0.39% dense, i.e. 0.39% of the logical space is reflected in physical space.

I have a flag to set the logical extent of the file using ftruncate at the start rather than at the end. Unsurprisingly, this makes no difference here.

$ ./sparsetest -i -b 4K -s100M -w1M test
  Intended logical size:       104857600 bytes;             100 M;           25600 blocks of 4096 bytes
  Optimum physical size:          409600 bytes;               0 M;             100 blocks of 4096 bytes
   Actual physical size:          409600 bytes;               0 M;             100 blocks of 4096 bytes

Used 100 writes of 4096 bytes every 1048576 bytes in ascending order
Created 800 512 byte blocks on disk
Density as % of actual physical size over logical size: 0.390625 %
Efficiency as % of optimum physical size over actual: 100.000000 %

So, let’s see what happens on the filer in question:

$ ./sparsetest -b 4K -s100M -w1M /path/to/test
  Intended logical size:      104857600 bytes;            100 M;          25600 blocks of 4096 bytes
  Optimum physical size:         409600 bytes;              0 M;            100 blocks of 4096 bytes
   Actual physical size:      131252224 bytes;            125 M;          32044 blocks of 4096 bytes

Used 100 writes of 4096 bytes every 1048576 bytes in ascending order
Created 256352 512 byte blocks on disk
Density as % of actual physical size over logical size: 125.171875 %
Efficiency as % of optimum physical size over actual: 0.312071 %

Eek! My 100MB sparse file is no longer sparse. In fact it’s negatively sparse! It uses 125MB on disk (a density of 125%). And the efficiency is tiny (0.3%).

So out of interest, let’s run it calling ftruncate before writing the data to the file, so that the writing itself never expands the file.

$ ./sparsetest -i -b 4K -s100M -w1M /path/to/test
  Intended logical size:      104857600 bytes;            100 M;          25600 blocks of 4096 bytes
  Optimum physical size:          409600 bytes;             0 M;            100 blocks of 4096 bytes
   Actual physical size:          413696 bytes;             0 M;            101 blocks of 4096 bytes

Used 100 writes of 4096 bytes every 1048576 bytes in ascending order
Created 808 512 byte blocks on disk
Density as % of actual physical size over logical size: 0.394531 %
Efficiency as % of optimum physical size over actual: 99.009901 %

Well that’s pretty much normal.

So what’s happening here is that when a sparse file is expanded using ftruncate, the filer is fine. I the sparse file is expanded using pwrite at an offset beyond the end of the file, bad things happen; it would appear the amount by which the file is extended (or indeed more) is allocated to the file.

I suspect this may not be the filer vendor (who I am not naming unless they want me to), but rather a product of the (Linux based) underlying filesystem that the filer vendor uses (I don’t know what that is yet). I suspect this has something to do with treatment of how the write is journaled.

I’ve never seen this before. But if you want to test your filesystem’s treatment of sparse files, here‘s some GPL code that will let you do it.

I was so excited to receive my five new 46cm/18″ Jala Beech spoons from Amazon today. Well, I was so excited to receive the five massive boxes each containing one spoon, that I thought you’d all enjoy a step-by-step account of the unboxing.

Let’s first have a look at how they looked when they arrived. You’ll notice Amazon have thoughtfully ensured that each spoon is separately packaged in its own voluminous cardboard casket, to avoid any inter-spoon crosstalk during the delivery process. Well done Amazon here, as I bought these as an add-on item, so might have expected them to share some of the copious amounts of packaging one of the other ten things in my order took up, but no, Amazon saw fit to do the right thing and ensure each valuable spoon was carefully cosseted in its own cardbox box.

Spoons in boxes
Spoons in boxes

We couldn’t wait to unpack them. Note the well cushioned packaging protecting each valuable spoon (cost: £3.67) from any vibration damage in transit, plus the copious amount of space around each spoon cleverly inserted to allow each spoon to breath:

Spoon cheekily peeking out of box
Spoon cheekily peeking out of box

And here they are in their full unpacked wooden glory (with associated packaging) on my kitchen table:
Full frontal naked spoons

Full frontal naked spoons

And yes, I’m on Prime, so paid no delivery charges for this. I’ll always order my spoons this way in future.

I don’t normally blog about Flexiant related stuff here, but this is a fun open-source skunkworks project, so I’m making an exception.

We’ve released, an apache licensed tool that allows you to use the Docker command line you are familiar with across multiple clouds.

The announcement is below, but you’ll get a better idea from the website or by reading the source code. The website has a really useful tutorial that explains things better than I can here.

Date: Tue, 11 Nov 2014 04:25:40 -0800 (PST)
From: Javi Perez-Griffo <javi@...>
Message-Id: <>

For the last months I have been working on a side project within my company for the management of Docker instances across multiple clouds. Today I would like to annouce the first alpha version of Krane (

Krane is built on the Docker code base, so supports the existing Docker command-line in a multi-cloud environment. This allows Docker users to use their existing workflows to launch apps transparently in multiple clouds, saving time and further reducing the friction of moving workloads between development, QA and production.

Eventually we’d like to get upstream Krane into the Docker code base. The patch itself is relatively small (a few hundred lines of code). It’s written 100% in Go, and is released under the Apache 2.0 licence.

In Krane, a ship is a virtual machine (VM) running a Docker instance into which Krane will place Docker containers. Krane (running on the developer’s command line) communicates with Docker running on the ship via its API (you can run Krane on the ship if you like, but it’s not necessary). Using Krane, you can execute your Docker command (such as starting or stopping a Docker container) on any ship. Ships are built as standard with a cadvisor container, which monitors performance of the ship and its containers.

A Krane shipyard is a driver for Krane that is capable of building, destroying and controlling ships. We’ve initially provided two drivers. Firstly, a driver for AWS. Secondly, a driver for Flexiant Concerto, a multi-cloud manager that supports AWS, Digital Ocean, Rackspace, Joyent and of course Flexiant Cloud Orchestrator (we’re giving away free accounts). We would appreciate and encourage contributions of further shipyard drivers.

Krane can currently do all commands displayed in the tutorial. We have intention of adding more commands to the list over time.

We are aware that there has recently been an announcement around multicloud management in Docker – see:

We’re keen to work with the Docker community to integrate the two ideas or give back our code to them.

You can learn more about Krane here:
The code is on Github here:

Have fun! Javi

I had a connection with BeThere with my BeBox in bridged mode, with a Mikrotik router behind it. My connection got migrated to Sky when Sky bought the Be business. I lost all but one of my static IPs, but more importantly, bridged mode stopped working. I rung Sky and their helpful advice was that they don’t support multiple static IPs, and they don’t support bridged mode. They said I’d find instructions on how to run bridged mode in the forums, but those instructions simply aren’t there (not to any working degree anyway). Given they’d effectively broken my entire DSL connection and had no solutions to proposed, I asked them whether I should simply take a MAC code and my business elsewhere. The guy on the other end of the line was clearly used to this from Be customers and had no issue with this. However, first I decided to see whether I could fix it.

The problem here turns out to be that Sky only do PPPoA (not PPPoE), and don’t support what used to be bridged mode in the BeBox. The BeBox is in fact (in my case anyway) a Technicolor TG582n, and it is (just about) possible to get this working in a sort of bridged mode, by using a little known feature of the TG582n to bridge from PPPoA to PPTP.

For the benefit of Sky customer service: I know supporting PPPoE as well as PPPoA might have been a minor bother. However, you could relatively easily have worked out how to do this bit for Be customers. And no doubt you could also easily have worked out a way of putting the Sky box you shipped me into some form of bridged mode too. How many customers were you intending to lose?

Anyway, here’s how you do it.

Step 1:

Connect your TG582n direct to your computer and remove the ADSL connection.

Reflash your TG582n with a sensible firmware image. Some helpful person might supply you with a copy of the software. If you have Windows, there is a software update tool. If you don’t, it will tftp boot. Mine doesn’t on power up, but the ‘software upgrade‘ command works from the administrator CLI. I have a DANT-1 board. Apparently there are different board versions needing different firmware. The main thing is to remove the Be firmware.

Step 2:

Restore the factory settings on your TG582n. Do this by ‘telnet‘ (username ‘Administrator‘), then type:

:system reset factory=yes

This will reset your password to the default for the firmware (possible to an empty password, possibly to your serial number).

Step 3:

Now put your TG582n into PPTP bridge mode. To do this, follow the instructions here. In short, log back in and type:

:ppp flush
:eth flush
:atm flush
:atm phonebook flush
:atm phonebook add name=vpi0vci38 addr=0*38
:service system modify name=PPTP state=enabled
:pptp flush
:pptp profadd name=vpi0vci38 encaps=vcmux
:ipqos config dest=vpi0vci38
:wireless ifconfig state=disabled

Note: I am told that previous editions of the firmware may have had an issue where on reboot the ADSL interface is not correctly initialized. Hence whilst the config will work, it won’t survive a reboot. The workaround is to put in a fake routed connection to a non-existent VPI/VCI combination. I’m running and it does not suffer from this issue.

Step 4:

Connect the TG582n to the Mikrotik and the ADSL line. Number the raw ethernet interface (I used port 2 but I don’t think it matters) to be on the same subnet as (the Mikrotik).

Step 5:

In order for the TG582n PPTP connection to connect successfully, the session must have a phone number entry of ‘vpi0vci38‘. It is not obvious how to do this with a Mikrotik, but here’s how. Go to the PPP menu, click ‘Profiles’ and add a new profile (I couldn’t get anything but a name of ‘Profile1‘), and under ‘Address List’, enter ‘vpi0vci38‘.

It should look like this:
Mikrotik PPP panel
Mikrotik PPP panel

Step 6:

Create a new PPTP client interface (I called it ‘pptpsky‘). Make it ‘connect to’, and use the above profile (where it says ‘default‘ select ‘profile1‘ instead). Do not select ‘Add default route’ as Sky sends a bogus default route (or more accurately seems to negotiate a bogus endpoint), just to be unhelpful. Use the credentials of your ISP, i.e. username ‘‘ and ‘password‘.

As the PPTP IP connection runs between the Mikrotik and the TG852n only, you can set the MTU and MSS to 1500 bytes quite safely. Though you’ll get fragments over the half metre of ethernet cable, you will be able to move full unfragmented ethernet frames over the WAN.

It should look like this:
Mikrotik Interface panel
Mikrotik Interface panel

Hopefully the connection should now go into a ‘running’ state.

Step 7:

Insert a manual static default route, using route ‘‘ and for gateway enter the interface name of the PPTP interface. In my case that was ‘pptp-sky‘. No, it’s not obvious you can do this, and yes I thought it wanted an IP address too.

It should look like this:
Mikrotik Route panel
Mikrotik Route panel

That was sufficient to get my connection running again. Good luck!