Test and verification: lessons learned

  • 13-Jul-2009 11:56 EDT
The modular Vector VT system was developed for functional testing of ECUs from the early development phase onward. It is placed between the ECU and the actuators and sensors.

In theory, a supplier should only deliver fully tested and debugged electronic control units (ECUs) to a vehicle manufacturer. However, reality is sometimes different: Take BMW Group. Up to a year ago, no one at BMW had the explicit responsibility to make sure that ECU software is not just implemented by the supplier but is also systematically tested and verified. Now, eight BMW experts do nothing else—full-time—says Dr. Maximilian Fuchs, who has been heading the verification strategy for the E/E process chain plus EMC at BMW Group since 2008.

The initiative to improve product verification was driven by a massive frustration at the premium carmaker as Fuchs explained during the 2009 Vector Consulting Forum at Stuttgart: “There were suppliers that used BMW as an external debug-loop instead of handing in properly tested ECUs. We are addressing this now. Test and verification are henceforth given the same priority that implementation has,” explained Fuchs.

The new rule of the game is: Bugs are fixed by the supplier. To make sure that OEM and supplier fully understand what is necessary to achieve this target, BMW closely defines the organization and methods of testing. At the beginning of a development project, both parties sign a written agreement that minutely defines the interfaces between the supplier’s responsibilities and the OEM’s tasks. “This is a mutual understanding of the things that need to be done.” As a consequence, the whole test process is formalized.

During the Vector Forum, focusing on the collaboration between OEM and supplier, Peter Feulner, head of passenger car driveline electronics at ZF Friedrichshafen argued along the same lines: “The most important virtue in working with suppliers is to precisely define who is responsible for what. Also, working with suppliers requires very stable processes and a maximum of standardization. The OEMs, for instance, should push standardization via AUTOSAR much more to build up pressure.” Yet, Feulner also agrees that while standardization may be the silver bullet to control development cost, “flexibility is also very important.”

Defining responsibilities does start early at BMW now. Four months before the first system is handed in to BMW, the supplier has to define what variants will need to be tested. “Three months prior to A-sample delivery, the test team has to be in place; test cases, methods, and formats have to be agreed on. The test specification must be done, and the reuse of test cases has to be documented. We want to know how the supplier plans his testing resources and exactly what is going to be tested when, where, and in what way,” Fuchs says, summing up the strict approach.

While all this may sound quite straightforward and should by all means be a part of software development, experience shows that it is not always so. Says Fuchs: “When we start to ask whether the testing capacity and schedule are in fact sufficient to warrant not only the testing but also the time for a bug-fix loop plus subsequent verifying of the debugging activities, we sometimes encounter doubtful faces.”

Following the actual software testing phase, BMW expects a test and quality report, which specifies which functions have been successfully tested, which functions are not available or not fully functional, and which bugs have been successfully fixed. Probably the most important bit of information is a list of remaining problems with the software. “Thus, we avoid having to find out about bugs ourselves.”

So how is the split done between the testing responsibilities of the supplier and the testing responsibility of BMW? Fuchs says: “We follow what we perceive as a logical split. The supplier is expected to test what he can test. What the supplier cannot test is our job.” To help the suppliers and improve the test quality, BMW provides them with various testing tools, such as Flash-Admiral, and automated test cases via a server.

It comes as a surprise to see how deeply and in how much detail BMW gets involved in optimizing test procedures. “We help our suppliers to develop detailed planning,” said Fuchs. Often, when an OEM “helps” a supplier, this support is not always met with enthusiasm. However, supplier reactions are far more positive than could be expected, according to Fuchs. “They are quite relieved when they realize that we do not let them alone with a long list of strict expectations. Instead, we sit down with them and develop the testing strategy hands-on. If necessary, we will drill down as deep as the C code.”

While BMW is tackling previous supplier quality problems by making sure that products are properly tested before they are delivered to the OEM, Dr. Christof Ebert, Managing Director of Vector Consulting Services, highlighted requirements engineering as one viable strategy to improve the collaboration between OEM and supplier. Requirements engineering can limit additional cost, said Ebert: “Systematic requirements engineering can reduce life-cycle cost by between 20 and 40%. This compares to doubling the much lower 5% cost for requirements engineering.”
HTML for Linking to Page
Page URL
Rate It
0.00 Avg. Rating

Related Items

Technical Paper / Journal Article
Training / Education
Training / Education
Training / Education