GoMentum testing team leads closed-course evaluation of ADA systems
Authored by Atul Acharya
Today, AAA published the results of testing active driving assistance (ADA) functions available on several commercially available vehicles. The results show that critical ADA functions, namely Lane Keep Assist (LKA) and Adaptive Cruise Control (ACC) that drivers rely on, fall short of expectations. These ADA functions — categorized as SAE Level 2 (L2) automation — are a subcategory of the generally known advanced driver assistance systems (ADAS).
AAA’s Automotive Engineering Group regularly performs research that benefits AAA’s 60 million members and the general public; this research is executed on commercially available vehicles (not on research prototypes). Previous research examined important ADAS functions such as automatic emergency braking (AEB) technology, revealing shortcomings of various available systems. Additional research recommended renaming various ADAS functions, a position now endorsed by SAE, noting how the commercial names for ADAS functions have become too confusing for consumers. These studies are independent evaluations and aim to be objective in their methodology and findings.
In the same vein, the recently concluded research aimed to examine the limitations of lane keep assistance and adaptive cruise control acting as one system. This L2 feature forms a core of automation functionality as vehicles become more complex on their way towards full automation. As more auto OEMs launch more ADAS features, it is imperative that motorists and consumers get an unbiased view of their benefits and limitations. Thus, the aim of the project was to find limitations of active driving assistance and inform both consumers and OEMs on their performance with the aim of improving them.
The latest research was led by AAA’s Automotive Engineering Group, in collaboration with AAA Northern California’s AV Testing team at GoMentum Station, and Automobile Club of Southern California’s Automotive Research Center. The test plan included two equally important parts: closed-course testing at GoMentum Station, and naturalistic driving on highways between Los Angeles and San Francisco. The tests were conducted over a period of a few weeks in late October and early November 2019. The work at GoMentum was led by Atul Acharya along with Paul Wells.
GoMentum Station was specifically chosen for closed-course testing as it is one of the premier sites for AV and ADAS testing, and includes features such as 1-mile long straight roads in the Bunker City test that include fresh lane markings, along with curved roads like Kinne Boulevard that has degraded lane markings. These features are ideally suited for testing Lane Keep Assistance functions that rely on lane markings, with the degraded lane markings offering an additional challenge to the vehicle’s sensors. Other areas of Bunker City were used to test Traffic Jam Assist (TJA) functionality, as well as testing the subject vehicle approaching a simulated disabled vehicle.
For closed-course testing, the key questions were:
- How do vehicles with active driving assistance systems perform during scenarios commonly encountered in highway situations?
- How well does the lane keep assist system perform?
- How does a vehicle perform in stop-and-go traffic?
- How does a vehicle respond to disabled vehicle on the roadway?
All vehicles were equipped with industry standard equipment such as:
- OxTS RT 3000 – inertial measurement unit
- OxTS RT-Range S hunter – for accurately tracking ranges to target vehicles
- DEWEsoft CAN interfaces for reading CAN bus messages
- DEWEsoft CAM-120 cameras
Target vehicles were equipped with:
- OxTS RT 3000 and OxTS RT-Range S
Testing Methodology Overview
Lane Keep Assist Testing
Sustained lane-keeping functionality is one of the primary capabilities of active driving assistance. To test LKA, the roadway utilized must have visible lane markings. Prior to test, a lane survey was performed on GoMentum’s 12th street test zone, a straight, 1.2-mile roadway with clear and fresh lane markings. This roadway is ideal for high speed testing so that vehicles can be tested at various speeds. Using the same high-precision lane survey equipment from OxTS, a precise map of lane markings was created by walking the entire length of the road. The map is then used as an underlay when lane tests are performed.
During testing, the OxTS 3000 inertial measurement unit tracks the precise movement of the vehicle under test (VUT) as it moves along the road when LKA function is active. As part of configuration setup, a polygon is previously defined that marks the edge boundaries of the VUT. Range data is collected that determines precise lateral distances from the vehicle’s polygon boundaries (more specifically, from its leftmost and rightmost points) to the nearest lane markings. All this data is captured at 100Hz, and then subsequently plotted. The charts show the vehicle’s lane centering position, as well as its distance to the right lane mark and the left lane mark. When charted appropriately, the data can show whether the VUT had any bias towards left/right placement when traveling in the lane.
Traffic Jam Assistance testing
Stop-and-go traffic situations are frequent while driving on highways. Nominally, adaptive cruise control (ACC) systems will “follow” a lead vehicle at a safe distance, accelerating automatically if the lead vehicle accelerates, and decelerating automatically if the lead vehicle decelerates. Of course, exactly what a “safe distance” is, and just how soon the vehicle accelerates or decelerates depends on the vehicle. Knowing the limits of these systems is important to motorists so that they are aware of potential risks.
To test traffic-jam assistance, the team utilized a DRI Soft Car 360® on a Low Profile Robotic Vehicle (LPRV) platform. The DRI Soft Car 360® is a foam car that is mounted on the LPRV platform which itself can move at speeds of up to 50 mph. With the DRI Soft Car acting as a simulated “lead vehicle”, a vehicle under test (VUT) activates its ACC system (by reaching a certain speed, such as 30 mph) and then lets the ACC system follow the lead vehicle automatically. The lead vehicle is then programmed to accelerate for some time, which causes the VUT to accelerate while maintaining a safe distance. Similarly, the lead vehicle is then programmed to decelerate, which causes the VUT to decelerate. The lead vehicle once again accelerates, causing similar stop-and-go behavior in the VUT. At all times, the vehicles’ kinematic data is recorded in a data logger. The vehicles are subjected to varying levels of deceleration at 0.3g, 0.45g, 0.6g and three runs are performed for each VUT. The following distance, separation distance / time-to-collision at start of braking, speed differential at start of braking, average and max instantaneous deceleration are all recorded. When charted out, the data reveals the system performance.
Simulated Disabled Vehicle approach testing
Driving on highways is often risky. AAA, the largest emergency road services (ERS) provider, alone handles over 30 million emergency road service requests nationwide every year. Encountering disabled vehicles on highways in a risky scenario for motorists. The team wanted to find how active driving assistance systems react when faced with such a situation.
To create a disabled vehicle situation, the team created a simulated scenario with the DRI Soft Car 360 placed halfway on the roadway, with 50% of the soft car in the travel lane and the rest 50% on the right shoulder. A vehicle under test is then subjected to this situation and its ADA system reaction is noted.
So how did these vehicles perform? While active driving systems mostly worked the way they were designed, there were notable shortcomings in their performance when these systems were pushed to the limits. Consumers and motorists should always be vigilant and attentive when driving, and be ready to take over at a moment’s notice whenever these L2 automation systems are active.
To learn more about the ADA L2 Testing, please download the full report.
If you are an AV or ADAS developer, or a technology vendor working on core components of automation, and would like to confidentially test the limits of your system, or to learn more about the ADA L2 Testing project, please get in touch with Atul Acharya or Paul Wells at: firstname.lastname@example.org