My Three Home Signal device has now been down for eight days. Despite their twitter support people (@ThreeUKSupport) saying periodically that they think the problem is fixed, my understanding is that this is pretty much a network-wide outage. Ringing the support number for 3 Home Signal (0800 358 4828) leads to a recorded message saying “We know you’re having problems with your Home Signal device …” and if you are prepared to wait for 20 minutes on hold for their call centre, they admit this is a network-wide problem.

Despite living reasonably centrally in London, I have no signal at home (well, occasionally one bar from the top of the house). So I rely on the Home Signal device to get any signal at all, as do the three other people (yes, glutton for punishment, I have four accounts with Three).

Now, I’ve worked in reasonably senior positions at telcos, and the only network-wide faults I have known that have taken eight days to resolve are serious physical problems, such as cable breaks, sensible telcos put in redundant cable systems to prevent these affecting service. If I had a twenty-four hour problem on my network, I’d be seriously embarrassed. But embarrassment aside, I’d be communicating to my customers what the problem was, and what my plan is to fix it. And that’s what Three are not doing.

Rather than admitting they have a problem, they are telling their customers that the problem does not exist or has been fixed:

  • Their support team is telling people the issue is fixed.
  • Their PR team is telling people the issue is fixed (see update on story, followed swiftly by another update)

However, as the rapidly flashing green light on my Home Signal box will attest, it hasn’t been fixed.

Calling the Home Signal support line results in a 20 minute wait to get through, followed by an admission (if pressed) that they have a network wide outage, that they have no idea of the cause (at least that they are prepared to reveal), and that they have no idea of an estimated time to fix. Calling the normal support line results in a suggestion you call the Home Signal support line.

Attempting to escalate the complaint led to the following comic list of events:

  • Long pause whilst transferred to supervisor
  • Supervisor asks whether she can put me on hold to find out more
  • I say yes, which results in the call being dropped
  • Supervisor calls back (on the mobile, which doesn’t work, obviously)
  • I receive a text message asking me to call 0800 358 4916 to talk through my complaint
  • That number is no longer in service (perhaps it has worn out), and leads to a recorded message, suggesting I call the normal support line

Oh, and complaining to @ThreeUKSupport merely leads you to be asked to fill in a form. Someone then rings you back, or attempts to (no luck given I, um, have no signal), who leaves you a voice message and tells you to ring the support line.

If you complain hard enough, they will refund your bill pro-rata – which on my larger tarriff (£45 a month) was a whole £8. Given I’ve spent at least a couple of hours on the phone, that hardly compensates for my time complaining. Notwithstanding, I suggest you do this, not because it’s worth getting the few quid back but because the administrative hassle of processing it might make them pull their finger out.

This is not how successful companies communicate. Put something up on your website. Proactively communicate that you have a problem, and what you are doing to fix it, and people will love your customer service. Pretend you have fixed the problem, send people around and around endless chains of telephone numbers, and they will think you are a bunch of incompetent muppets. And on balance, it seems they’d be right.

In the event, I got more information from Wireshark than I could get from Three. A little playing with packet capture establishes that the boxes (white labelled Ubiquisys, now part of Cisco) successfully boot, make NTP and DNS queries, download their config (when hard reset), successfully negotiated TLS to a three server, then set up and maintain an IPSEC tunnel to somewhere in Three. And indeed one engineer successfully remotely logged into my box and rebooted it. So the signs are they aren’t suffering from a DoS attack or anything similar. My understanding is that they then run SIP over this tunnel. Presumably this layer or upwards is not working.

So, Three

  • Fix your network
  • Fix your communication failures
  • Give the customers who are suffering some gesture beyond a couple of miserly quid.

Epilogue #1

Today Three finally fixed it (11 days later). Here’s how I got them to do it.

  • I rung 0800 358 4916 on a weekday, which is the complaint line, raised merry hell and asked for the fault to be escalated. They credited me 2 weeks’ line rental on each account at this point.
  • A few hours later, they rung me back on a landline and I got put through to their Home Signal team (without waiting 20 minutes on hold).
  • They asked me to reset the box which unsurprisingly did nothing.
  • An hour later they rung back to see if it was working (no), and said they were going to delete the registration from their system and set it up again from scratch. I should then reset the box.
  • Ten minutes later it was working. They rang back to check after a bit.

Why this simple reset could not have been done at any point during the 11 day outage remains a mystery.

Epilogue #2

Obviously that was two easy. It worked for an entire evening, but only on my phone (there are 3 more on the box). This morning it was trapped in a fugue state of apparent rebooting (red light, rapidly flashing green light, no light, red light, rinse and repeat).

Now Three are replacing my white home signal box with a black one.