R&D Turnaround at Switchgear & Instrumentation

From struggling development to the start of a long-term success

A new product line

Switchgear & Instrumentation (S&I) were in the midst of the in-house development of software and hardware for a new motor-control solution which they had committed to deliver to ExxonMobil for two of their oil rigs destined for service in the North Sea.

They had won the contract based on the benefits of this new intelligent product. Successful delivery was critical to both the relationship with ExxonMobil and to S&I’s own future, differentiating them from their competitors.

A need for clarity

With the date that ExxonMobil were due to arrive and begin acceptance testing looming, S&I’s Managing Director, Ray Lacy, recognised that the project needed additional engineering leadership support if it was going to make the date.

Progress had slowed, features were incomplete, there were hardware issues and the small development team was locked into an endless battle, trying to make existing features work.

Ray was concerned that there was no real understanding of the state of the project and that any such understanding would not paint a pretty picture.

The referral

One of S&I’s hardware suppliers knew of SQC’s experience and work, and recommended that Ray consider engaging SQC to review the development activity, clarify the position and prognosis, and make recommendations for change.

Initial findings

SQC completed the review in two weeks. It explored the state of the hardware, the progress made with the writing of the software, the quality of the software, the ways of working within the team, the resourcing of the team and engineering practices and standards.

Over ambition, a lack of resources (both people and tools), inappropriate working practices, a lack of tracking, a lack of prioritisation and an over reliance on one engineer, were amongst the critical factors that had doomed the project from the start.

It was clear that without radical change there was no hope of meeting the ExxonMobil deadline and there was every possibility that the delivery would fail completely. Recognising the jeopardy, Ray acted decisively. SQC were asked if they could reset, replan and lead a recovery exercise, backed by a commitment to bring to bear whatever resources turned out to be necessary.

The essentials

SQC’s first step was to focus on the essentials. Features of the solution were removed or simplified until what remained was what was actually needed to meet ExxonMobil’s requirements. This was a tough ask for the lead player in the S&I team. His vision for a feature rich and generally applicable product was being reduced to a bare bones ExxonMobil specific system. It was hard to accept, but he recognised the necessity of it.

Gathering resources

The project was understaffed and under equipped. More people and better tooling were desperately needed. At SQC’s behest, priorities across the organisation were changed with experienced resources joining the project.

People came in from the IT department and from the project and professional services teams. A project workspace was secured. Equipment was upgraded, new equipment types purchased and centralised services set up to hold project assets. The recruitment of additional software engineers began.

Engineering discipline

Under SQC’s direction, software engineering disciplines were brought to bear, and practices were established that covered:

• Defining behaviour in advance, rigorously
• Technical impact assessment and designs
• Peer reviews & design authority reviews
• Systematic defect and issue management
• Strict source code change control and immutable system builds

Stability and progress

Greater discipline led to code stability with less breakages, fewer defects in new features and an increase in right first-time fixes of both old and new defects. Steady progress towards a feature complete state began.

Testing to the limits

A rigorous test regime was a fundamental foundation of the recovery. There was to be no illusion of progress and improvements by claiming things had been coded. Nothing was counted as ‘done’ until it had been passed by the dedicated test team.

Not that developers were allowed to rely on this testing. The ethos was clear, developers had to test their own work, thoroughly, no short cuts, before handing changes over to the test team. Change design reviews focussed as much on a developer’s testing proposal at is did on the planned code changes.

S&I’s test team, coached by SQC, developed an ethos akin to IBM’s ‘Black Team’ ethos. Their role was to break whatever came their way and to find at least one catastrophic bug in each change before it was signed out; they often found many. They knew the system was a critical component of an oil rig, the cost of failure would be vast, so it had survive under all foreseeable circumstances.

Healthy competition

Competition between the development team and the test team was fierce, but healthy. Developers were determined to find the bugs in their code before the test team did and the test team were determined to find something development had missed or not thought of. Quality improved dramatically. People thought more and coded more carefully. Some things took longer, but the time was well spent with the system becoming more complete and reliable.

A win for S&I

After one last momentous effort from the team, all of the critical issues affecting the ExxonMobil solution were cleared. The entire intelligent motor control centre then passed factory acceptance testing by ExxonMobil and was shipped off to be installed in the first oil rig. So began a new phase for the industry, using intelligent, not hardwired, motor control centres.

The project spawned S&I’s Intelligent Systems Division, a business unit that went on to develop the initial CMAC product, a widely used and successful product line, followed by the Motor Manager 3, 5 and 6 products, and a number of desktop applications for power systems monitoring and supervision.

A win for SQC

This work started a decade long partnership between S&I and SQC. A partnership that saw SQC act for them to:

• Assess the safety implications of control system development work
• Produce the functional specification for their first intelligent motor protection relay
• Provide design leadership, hardware and software development management, test leadership and establish manufacturing and distribution of their Motor Manager 3 and 5 products
• Lead on the adoption of the Motor Manager products into first off commercial use
• Test desktop control software products and control system implementations

Switchgear & Instrumentation Limited (S&I) was a leading supplier of medium and low voltage switchgear, intelligent motor control systems and power distributions solutions, to a wide range of process industries, including oil, gas, petrochemical, power generation and distribution. S&I was acquired by Powell Industries, Inc., a leading manufacturer of equipment and systems for the management and control of electrical energy. S&I rebranded and today are known as Powell UK Limited.

• Turnover £33 million (2024)
• 195 employees (2024)