Light The Blue Touch Paper: Your Handy Guide To Conducting Useful Tests

Light The Blue Touch Paper: Your Handy Guide To Conducting Useful Tests

In Part One:Explore More we looked at how to be specific in your scoping, giving you the best foundation from which to guide your project. Part Two: Don’t get it right, get it moving considered generating the best solutions to the problem, which could well be liberally stealing the ideas of others around you. It covered how, once you’ve selected and appropriately documented these ideas, it’s time to build a test plan for each prototype.

The next phase is the actual testing, the outcome of which could, and should vary depending on what stage of thinking you’re at.  In the Bromford Lab I’m breaking down tests into three distinct types:

  1. Tests that help you build and improve your prototype. Your very earliest designs will have problems – undoubtedly. Maybe the systems you were reliant on are actually full of rubbish data or maybe you’ve got some really great feedback from customers on something you didn’t expect – either way you’ll want to make changes to your current build. If you want to ‘learn by doing’ be sure to set frequent milestones where you can conduct ‘health checks’. Set an ultimate deadline to avoid descending into an unfocused glob of a project.
  2. Tests that validate your design. When you think you’ve got something that fits with your culture, adds value and works without falling to pieces when you look away – stop. Start afresh with a new test. Stop making incremental changes and stick to this working prototype. Install measures of success, formed from your initial brief to validate how well it’s working and whether it’s still fit for purpose…
  3. Pilots. While still constrained to either a demography or geography the scope of pilots is typically larger – we’re talking about evaluating hundreds of interactions, not just tens. Earlier tests will have identified issues with implementation, but pilots will identify issues with ‘scalability’. Project roll out is fraught with its own challenges, which is why the prototype should be solid before being piloted. You don’t want to sink a good idea just because your team struggled to get it started at scale. You don’t want to float a bad idea because a handful of positive cases can be used to gloss-over a sour core.

You can find out more info about tests v. pilots here

The progression through these test types should seem clear and logical. Not working through these stages successively causes fundamental problems that could derail your testing. Take a look at the big four challenges in the infographic below – how could you protect the test against them?

 

 

The key to small scale initial testing is to be honest with yourself. Unless the new concept is simply branching off an existing service, you probably have no idea how you’ll fare in a new area. Don’t worry about it! Knowledge and understanding take time to build, if only a few weeks.

How do you facilitate good testing in your business?

Advertisements

Don’t Get It Right – Get It Moving: How To Build Your Prototypes

Don’t Get It Right – Get It Moving: How To Build Your Prototypes

Prototyping can be one of the most exciting parts of concept development. Yep, getting those pent up ideas finally out into the open can be pretty cathartic, a sweet/salty release.

But wait, is your original idea cut from the exactly the same cloth as a million others in the sector? Is your idea recycled from something you yourself were part of 5, 10 or 15 years ago?  These ideas tend to be the ‘go-to’ options for a reason, they are tried and tested.  I reached this conclusion for some ideas in our latest wicked problem:fuel poverty – the only thing is, if it was a truly good solution why is everyone talking still about it and not doing it.

Why have you got to revisit it years after your first attempt?

Without dedicated time to research innovative solutions, your solutions will never be innovative – you’ll never break the mold and never solve the problem for good. If you’re on easy street or if you’re just looking for a stopgap then OK, but I’ll see you here again in another 5 years…


Collecting inspiration, the first step of prototyping, means casting the net as wide as you dare. Plunder the influential others for directly relevant material, sure, but true skill lies in stealing ideas from industries you’ve barely heard of….

You’ll need a solid grasp of the fundamental problem, something you should have pinned down in the earlier exploratory phase of concept development. Conduct a root cause analysis, deep dive, collaborative questioning – whatever you have to do to get to the crux. The further you dive the more issues and ideas become transferable. Take a look at the example below I’ve paraphrased from Sticky Wisdom (Allen et al.):

A printer company is sending out lots of engineers because it’s machines keep getting blocked. Now think about this – what else gets blocked?

Noses. Noses get blocked. And if getting a blocked nose is critical, we’d all be dead. Fortunately both nostrils don’t tend to get blocked at the same time and even if they did we’ve usually got a mouth available to pick up the slack. Redundancy. Could we add another feed tray?

But what else? Pipes. Pipes get blocked. We overcome this by increasing the demand on the pipes, using a plunger or object to move the blockage through the machine. Force. Can we load enough paper in one go to push the blockage out?

Tunnels. Accidents happen and tunnels get blocked. Operators may set up a system that allows the other lane to be used, managing traffic each way. Intelligent design. Can we set the out-tray to accept paper until the blockage is cleared up?

Biology, plumbing and road traffic management – all to fix a printer jam. Some of the ideas don’t actually look too bad – though I’ve blown my intellectual property rights to the out-tray in-tray idea… Allen and co. call this technique the ‘lateral step’ and it works by taking our minds out of the paradigm we’re all used to thinking in – ‘social’ (bleurgh). Think instead from the perspective of someone who’s already solved the problem, just in a different context. It’s brilliant, and it also takes care of step two, generate/transfer ideas.

Being social, lots of housing associations face people problems, that is colleagues or customers not behaving as you’d like them to, for whatever reason. Fortunately for us, pretty much every industry has people – so the possibilities are endless. Though this isn’t an invitation to start replacing everyone with robots just yet!


Obviously, some other companies would seize an opportunity in exactly the wrong way for your business, so the ideas still need some filtering. It may be that there aren’t any directly useful solutions out there. But what you can do now is spruce up your ‘sector thinking’. Take lessons from those other fields and apply them to a ‘run of the mill’ original idea.

 

pipeline-prototype.png

 

Finally – you’ll need to draft up a proposal for each idea you want to keep. I call them test plans and include a list of draft measures (again, the explore phase will help you identify what success looks like). How to design a test plan is a post in itself – but for now I’d suggest keeping it purely academic. If you can’t write the pertinent points up in a few lines, you’re not doing it quite right.

Don’t bulk it up, don’t try and include everything else under the sun. Keep it simple, keep it light and add features incrementally as and where it makes sense. But more on this in the next pipeline installment: testing.

Tom

Explore More – How to give your innovation efforts the best start

Explore More – How to give your innovation efforts the best start

Usually performance reviews are good for two things:

  1. Creating more work, usually in the way of forms to fill in and…
  2. Personal development (yuck!)

That said Vicky and I also took the opportunity to look at our innovation pipeline in a bit more detail and, considering the successes and failures over the last year, seek to build on it…

Involving customers early on in the engineer visits concept was an overwhelming success, providing the team with exactly the sort of feedback we needed to redevelop our service offer before sharing the next iteration. Customer feedback and colleague insight can easily get us to this first ‘service design’ – with the added benefit of not approaching customers with a blank piece of paper.

We also identified that keeping control of high profile concepts is an ongoing battle. I’ve witnessed stakeholders trying to accelerate new-born concepts straight through to pilot if the concepts have received lots of hype. If you’re not sure what’s wrong with that, click here for my post on tests v. pilots post. Innovation is as much about winning hearts and minds as following a pipeline, guys.

Vicky and I also talked about how, while concepts do need a certain level of individual design, without a physical reminder of the pipeline I’ll start winging it for sure.  No Bueno. If concepts worked we wouldn’t know why and couldn’t repeat it. If they didn’t we couldn’t be certain about what we’ve learned…

So without further ado here is the first installment of our innovation pipeline – the six stages of exploration needed before you start mocking up ideas and solutions.

Got any comments, praise or just want a shoulder to cry on? Hit me up on the particularly inspiring account@ThomasHartland or chat to all of us @BromfordLab. More to come!

pipeline-explore2

What’s The Difference Between A Test And A Pilot?

What’s The Difference Between A Test And A Pilot?

I get it – looking down the barrel of any new project might seem especially daunting, but your first steps will determine whether it develops into a beauty or a beast…

You’ve been handed the baton and a million different possible combinations stretch out before you. Where to begin, who to involve, what to do first? Then there’s the constant”I want it yesterday” battle with the project sponsor. How do you possibly start to navigate through all that?

The temptation to just pick up an idea and run with it is often overwhelming. 

In the first year of the lab I spent most of my time trying to get projects like these back on track, or stop new ones falling into the same routine. Services were being bought to the lab because the sponsor either didn’t know or wasn’t happy with the progress being made. The teams had gone through the right process – piloting out the new service at a local level. Measures had been set so that an evaluation could take place once the pilot was finished, making recommendations for whether the service continued.

Unfortunately, the service leaders/managers often tasked with implementing shiny and new services didn’t have the right skill-set, mind-set or time to design their offering properly. When projects were evaluated the usual conclusion was that the service changed so frequently during the pilot that most of the measures were now redundant… The word ‘pilot’ became synonymous with last minute scrambles to evidence whether the service worked or not.

How people envisage innovation works – sort of a ‘one-stage’ model
What actually happens when you rush to implement your ideas…

For the project teams immediacy and performance might as well be mutually exclusive – with immediacy winning out every time. ‘Design’ was too time consuming to be an option. We watched frustrated, wondering whether people knew they were scaling services that never worked in the first place..


In truth, good design (i.e. not only drawn up on the back of a cigarette packet) doesn’t just make things better and more fit for purpose, it could actually be faster.  Our innovation pipeline is designed to produce a minimum viable product that we can release in a safe, controlled test environment – not holding ‘creative sessions’ for weeks on end. Our pipeline gives us a focus and a reason to just get the ball moving.

We redefined our pipeline around ‘tests’ – our way of getting quick bursts of feedback about new concepts in the simplest way. Read the definitions below, taken from my earlier post on the Bromford Lab website.

Testing

Tests tend to focus explicitly on the building blocks of a new service. They are time-limited, closed off experiments that help us evaluate design components in isolation, without any of the noise that ‘real life’ might generate. Not that this real noise isn’t relevant – it just muddles things early on.

Light, fast and ‘dirty’ tests come with relatively low risk so we can afford to do lots of them – indeed we can even fail lots of them without worrying too much. Much better to fail quick, fail cheap, right? More often than not it’s not the idea that fails, just the way we’ve chosen to implement it or that colleagues have gone ‘off-piste’ with the delivery. Early testing has usually highlighted a new problem – e.g. engagement problems – that could have presented significant start up lag-time if we’d jumped straight to pilot.

Tests are also a great way of identifying weaknesses in the data you’re trying to collect. Data requirements should be small and focus solely on what effects you’re trying to observe, rather than exploring relationships between a new and existing service.

It’s important to document a test so we prep all concepts with a test framework that outlines all the aims/objectives, the methodology and all the data we’re trying to collect and whether the test is exploratory or aiming to answer a specific question or hypothesis.  When you’ve finished testing you should be able to pull together all the information and produce an up to date, field-tested service offer document that outlines your idea in depth, with any changes as a result of your testing included.

Piloting

Pilots evaluate the whole, assembled service and usually take place over a protracted time-frame so you can spot the interactions you might’ve missed in testing stages. This is ‘adding the noise’ back in to see if your idea holds up.

testsandpilots
Because of the resourcing, duration, risks, costs and difficulty in mobilizing – you don’t really want to fail pilots. Better to fail a pilot than have a rubbish service implemented, granted. Then again, much better to drop an idea as a result of a couple of failed tests too…

A whole swathe of measures will likely be drawn up for data-hungry pilots, which drastically limits your ability to adapt and change the way the pilot is ran… or if you do, you can’t trust your before/after measures any more.

Pilots are the only way you can test your idea out in real life situations, so are probably important or whatever… but don’t get caught in the trap of fetishizing about how many pilots you have on the go at any one time. Calving more unwieldy pilots into existence is not a badge of honour, it’s a badge of not valuing your own time.

Finally, Pilots should never be implemented or scaled into the business without being evaluated. If you’re not going to let the idea fail, there’s no point piloting it in the first place.


testpilotlist.jpg

So – the lab has become the driving force for testing out new ideas. I like to use the following metaphors to illustrate how this mindset differs from the default “let’s go to pilot!” setting.

Test Driven Design

Adam wants to know if his fort can withstand an enemy invasion. Adam’s not wearing a dress – it’s a kilt, he’s Scottish. (Plus I can’t draw trousers that don’t look crap yet…)

His fort needs a little work, so Adam thinks about all the possible options and what ramifications they’d have on ‘invasion-proofness’. Does he make it 10ft taller or 50ft taller, use traditional bricks or spherical bricks, fill the moat with fire or water? Adam decides to build little towers to prototype each element.

Bigger is better, a water-moat is easier to maintain than a fire-moat and in hindsight spherical bricks were probably never going to work. The results of these tests are evaluated and Adam builds his fort accordingly. He still doesn’t actually know whether it’ll hold up to an invasion – he’d probably get his mates round for a dummy attack run, see how everything holds up. Maybe he’ll even let them in for a beer afterwards…

Pilot Driven Design

Dave wants to know if his fort is invasion proof too, but the cunning fox has spotted a funding opportunity that might help him make his fort an ass-kicking machine. He successfully negotiates a tender with his local NHS clinical commissioning group, on the premise he can demonstrate the fort improves the well-being and psychological resilience of his local community.

Dave embarks on weeks of interviews with his local community, collecting baseline data and identifying individual needs. Soon enough he’s had to change his designs and make a number of compromises all around.

Unexpectedly, accessibility was an issue so he had it converted into a bungalow. People also asked for columns, and they seemed to show an improvement in well-being scores so were incorporated. An invasion was due, so Dave quickly spent the rest of the money on anti-aircraft guns to deter an onslaught.

Unfortunately, most of the preparation time was spent on delivering the NHS targets and the objective of ‘building an impenetrable fort’ sort of fell by the wayside. When the invasion came, it did so on foot, immune to the weaponry. As it happens, the quality and craftsmanship of the pillars diffused the invaders, who went inside to drink tea rather than pillage. Aside from a few stress related deaths, the local community were also showing higher well being scores.

Confused? Dave is too..

 


Take Home Message

High performing products and services aren’t by accident, they’re the outcome of good design. If a project is truly worthwhile, why not invest time in the design of it rather than just winging it? Invest in the right people to guide the idea, people who can maintain professional distance and keep egos out of the equation – ultimately people who are willing to tell you if your idea sucks.

testmore.png