Page tree

Transmitted by the expert from EC for subgroup 2a

 

Document: VMAD-05-05

Concept paper on Audit/virtual testing/in use data reporting

 

This paper was drafted by the EC with subgroup 2a to describe the concept of the “Audit/virtual testing/in use data reporting” pillar in accordance with the layout proposed by VMAD at the last meeting (Why, What and How). It also includes as an annex the main building blocks identified at this stage for possible requirements under this pillar. Another paper proposes a draft Annex to the new UN Regulation on automated Lane Keeping Systems (ALKS)

--Why--

With automated driving the amount of possible (traffic) scenarios to be compared to current driver assistant functions such as Anti-lock Braking System (ABS) or Advanced Emergency Braking Systems (AEBS). The complexity of the situations to be managed by automated driving systems will also increase, as the systems will take care of the entire dynamic driving tasks, in particular environment monitoring and interaction with other road users, which are managed by the driver in case of driver assistant systems.

Consequently, the “classical approach” based mostly on a set of pre-defined physical tests is no longer sufficient. It will be impossible to demonstrate the performance of the system solely on the basis of a limited “representative” set of tests on a test track.

--What--

As a result, it proposed that manufacturers will be required, in addition to pre-defined physical tests, to demonstrate that an acceptable thorough consideration of functional safety (safety issues due to possible system failures) and operational safety (safety issues due to other causes than failure, e.g. perception problems) has been implemented during the design and development processes of the automated driving system and will continue to be done throughout the vehicle lifecycle (development phase, production, postproduction phase, operation, decommissioning).

Industry standards already exist for functional safety (ISO 26262) or are under development for operational safety (ISO PAS ISO/PAS 21448 and UL 4600). Similar process-based standards are being developed for cyber security (ISO/SAE DIS 21434).

--How--

The manufacturer will be required to demonstrate through documentation, physical tests (on track and on road) and/or virtual testing that:

  1.                    Hazard and risks relevant for the system have been identified and a consistent safety- by-design concept has been put in place to mitigate these risks
  2.                    A robust validation process of the design (virtual testing, track tests, real driving tests) confirmed that the system meets the legal performance requirements, traffic rules and is free of unreasonable safety risks to the driver, passengers and other road users before the vehicle is placed on the road.
  3.                    Robust processes/mechanisms/strategies (safety management system) are in place to ensure continuous safety through the vehicle lifecycle including development phase, production, but also operation on the road and decommissioning (e.g. the manufacturer shall have processes to monitor incident/accidents with their vehicles and to react appropriately in case of incident).

On the basis of the evidence provided by the manufacturer, authorities will be able to audit/assess whether the design, validation and the safety culture of the manufacturer are robust enough with regard functional and operational safety. Authorities will also be able to check through targeted physical tests (on track and on road) and virtual testing whether the safety by design concept is implemented on the vehicle.

If virtual testing is used, manufacturers will have to demonstrate the scope of the virtual testing, its validity for the scenarios concerned as well as the validation performed for the simulation tool chain (correlation of the outcome with physical tests).

Further discussion is needed in VMAD to clarify the difference between physical tests under the audit pillar (mainly to confirm the safety concept and the validity of simulation) and physical tests as discussed in subgroup 2b (to check some minimum driving capabilities by the system). Connections also exist with subgroup 1 to determine which critical scenarios should be considered.

The audit approach places a higher emphasis on the format of the documentation to be provided by the manufacturer /made open for inspection, the transparency of the appropriate information to all Contracting Parties, pass/failed criteria and the competence of the auditor.

Possible building blocks for Audit/assessment/simulation pillar/in use have been discussed in subgroup 2a and are proposed in the annex to this paper. An example of the implementation of this concept is also provided in the draft Annex to the UN Regulation on Automated lane keeping systems in a separate paper.


Annex : Building blocks Audit/Assessment/simulation/in use reporting’ pillar

Subgroup 2 agreed to split between the requirements that shall apply:

- for the audit of the manufacturer processes/mechanisms/strategies (audit of the safety management system) in place to ensure continuous safety through the vehicle lifecycle that are not necessary linked to a specific AD system.

-and the assessment of the AD functional and operational safety that is specific to each AD system.

 

The tables hereafter give the main elements (building blocks) identified at this stage for the requirements Audit/assessments of AD systems. They are not necessarily final but give a good overview of the main elements discussed in subgroup 2a. They also served as a basis to draft the Audit Annex to the new Regulation on Automated lane keeping systems (ALKS).

Building blocks for the audit of the manufacturer safety management system

Possible requirements

A safety management system shall be established, documented and implemented by the manufacturer and shall demonstrate a clear safety culture through adequate processes within the manufacturer in accordance with ISO 26262

It shall include a clear process for: Safety by design (high level safety rules), hazard and risk analysis, verification and validation through simulation, test track and on road tests.

The organization shall institute and maintain effective communication channels between functional/operational safety , cybersecurity and any other disciplines related to the achievement of  vehicle safety.

They shall have processes to monitor incident/accidents with their vehicles overtime and react appropriately (closed loop of field monitoring).

Manufacturers shall document and demonstrate how they ensure a safety culture and the capability to manage safety during the lifecycle of the vehicle (development phase, production, postproduction phase, operation, respecting/updating traffic rules, , decommissioning).

Manufacturer process shall include continuous independent internal audit to ensure that the process is implemented consistently throughout the different phases of vehicle lifetime (development phase, postproduction phase, respecting/updating traffic rules, etc).

Manufacturers shall put in place suitable arrangements (e.g. contractual arrangements, clear interfaces, quality management system) with suppliers to ensure that  to ensure that the manufacturer safety management scheme is implemented by suppliers

 

Same arrangements as today in the CEL?? Interface between OEM and suppliers and what are the terms of these interfaces. Responsibilities, safety plan, internal audits, supplier selection criteria (focus on safety aspects and quality management systems, etc. )

 

-Action item: system/component level by suppliers? Black box safely integrated. Safety and process objective. Similar to ISO 26262 (look at the wording).

 

Failed/pass audit (Reference to Annexes B and C of ISO 26262-2 for criteria??) :

2 levels: Major/minor

Major: process not documented, no demonstration that there is an appropriate level of management, process not implemented in a consistent manner

Minor: the demonstration exists but is not convincing enough

Both Major/minor issues have to be close before any vehicle is placed on the market.

No grading in Annex 6 in CEL at this stage.

Both parts are covered by Annex 6.

 

Competences/skills and criteria for the independent auditor from the technical service (in accordance with ISO 26262?):

Renewal of the audit:

Initial assessment and monitoring by authorities. Validity of the assessment. Impact on granted approvals?

The audit of the safety management system may be reused for different vehicle assessments done by the same technical service/type approval authority. But there is no obligation of other type approval authorities to recognize an audit if the safety management system if this audit is not linked to a specific type-approval.

References

ISO 26262/SOTIF 21448

UL 4600

Conformity of Production provisions in the 1958 agreement

Annex 6 of Regulation 79

Cyber security UN regulation

 

 

Assessment of AD functional and operational safety – Comparison with UN R79-CEL

This table show the main elements identified for the assessment of the AD functional and operational safety and the comparison with the latest version of the Annex on complex electronic systems (CEL) used in different UN regulations.

A Safety  report by manufacturers

Gap analysis made by Industry with CEL-step2 Annex of R79 (Complex electronic system)

1)      Description of the system (ODD, function, type)

 

The manufacturer shall declare to the type-approval authority the scope of the automated driving mode (so-called operational design domain(s) (ODD): Road conditions, Geographical area, Speed range, Other conditions that must be fulfilled for the safe operation in the automated driving mode or will otherwise trigger a transition demand/minimum risk maneuver.

 

NOTE: For ALKS: description of the ODD within the ALKS definition. However a smaller ODD is allowed by the regulation (e.g. traffic jam conditions only)

 

Action : Model information document and information package/safety report tbc

Limits of the system or ODD?

 

 

  • 1) Description of the system is covered by paragraph 3. Documentation.
  • The safety report would be the information document which the manufacturer provides to the Technical Service/Authority. The information document becomes de facto a public document as attachment of the Test Report/approval.
  • Description of the functions is covered by 3.2
  • The term ODD is not explicitly mentioned, but covered by a different wording in 3.2.3 “Limits defining the boundaries of functional operation…”. Check: How to clearly add ODD to the existing text
  • 3.2 requires description of the system layout and schematics (which is another word for architecture). Explanation of the system “architecture” is also covered by 3.4.2

 

The manufacturer shall declare to the main functionality of the system (functional architecture), its dependencies on, and interaction with, other vehicle systems, the driver, the environment and other road users (see section 1 of the Annex 1- See also main sections of the framework document).

 

Action for ALKS : Check interaction with the core text of ALKS

 

  • Description of the functions including control strategies is covered by 3.2. This also includes a list of all input and output variables (relation to other systems)

“Interaction with the driver” for fault-conditions is covered as 3.4.3 requires warning signals or message displays to the driver in case of failures. And 3.4.4.1 that corresponding warning signals shall be given to the driver. In addition 4.1.2 requires the verification tests under influence of failures including HMI aspects Check: How to add “interaction with the driver” in non-fault cases to the existing text

Interaction with other road users not covered
Note: In case of ALKS, interaction with the driver is addressed in the current draft with performance requirements in 2.4 (Activation, Deactivation and Driver Input), 2.7 (Transition Demand and System Operation during Transition) and 2.8 (Information to the Driver)

 

 

The manufacturer shall declare the overall system architecture (see generic architecture “sense, plan, act” in the safety first white paper from industry) as well as the safety concept put in place to meet the safety requirements.
Follow the principle of RXSWIN vs Software number.

 

It shall provide an identification of safety critical components and software in particular for the following subsystems:

-Perception and objects detection including mapping and positioning

- Characterization of the decision-making safety

- Documented data processing in case of continuous learning implemented.

- Human-machine interactions including the driver but also other road users 

- Supervision and remote monitoring (if applicable).

 

Covered by 3.4.3

Type definition??

 

Today’s wording of Annex 6 is very much focused on the vehicle. We could focus on the system.

Used on a practical basis

 

Action item for ALKS

1)      Hazard and Risk analysis (functional and operational safety, link with cyber risk as well), Safety concept and safety by design by manufacturers

NOTE : Avoid duplication (cross reference to cyber regulation?): Important is the link between cyber analysis and the safety of the function. CSMS in application.

 

Paragraph 3.4.4. covers HARA for functional and operational safety. This should not be part of a public
Cyber Security is not included in today’s annex CEL and not necessary to add here for ALKS as a separate CS/OTA group is currently developing the requirements. Check: How to add a reference to this activity?

General safety requirements are at this stage set by the framework document and future more detailed requirements development in FRVA and ACSF (prevent accident, traffic rules, duty of care, failsafe response, HMI, OEDR, cyber/software updates, etc.) .

 

For ALKS : main safety requirements shall be established in the core text of the Regulation.

See 1. General: The annex itself does not specify the performance criteria for "The System" but covers the methodology applied to the design process and the information which must be disclosed. The objective is to demonstrate that "The System" respects, under non-fault and fault conditions , all the appropriate performance requirements specified elsewhere in the ALKS Regulation and that it is designed to operate in such a way that it does not induce safety critical risks.

The manufacturer shall declare the high-level safety rules and the safety concept used for the design and described how this concept is implemented in the vehicle design.

Covered by 3.4.3

Check: Wording also include non-fault conditions in terms of external (environmental) disturbances and interaction with other road users

The manufacturer shall in particular demonstrate that it has conducted a hazard and risk analysis for the automated system, its integration in the overall vehicle design and the broader transportation ecosystem and put in place adequate design and redundancy to address these risk and hazards (safety concept) and that the design leads to an acceptable residual risk for the ODD concerns).

 

Covered by 3.4.4.

 

Systems shall in particular be designed to adresss risks that could impact critical functions of the system (for the driver and other road users) due to cyber-attacks and failure (functional safety) but also potential inadequate control, undesirable control actions, reasonably foreseeable misuse by the driver or other road users(to be further defined), tampering and inadequate interaction with other road users (operational safety) during the lifetime of the vehicle. Relevant demonstration methods include ISO 26262 for functional safety and a system-theoretic process analysis (STPA) for operational safety or an equivalent method such as draft ISO PAS 21448.

 

  • For cyber-attacks (incl. tampering) see above not included
  • Control/interactions with other systems covered by 3.4.4.
  • Failures/malfunction covered by 3.4.4
  • Reasonably foreseeable misuse covered by 3.4.4.

“interaction with other road users (operational safety) not explicitly covered Check: How to address operational safety?

The manufacturer shall describe how the current responses of the ADS to non-critical driving situations, expressing carefulness/duty of care of the ADS (to be made more precise).

For ALKS

Note: Challenging/critical test cases for track testing will be defined in the ALKS regulation

 

Description of ADS-driving behavior in non-critical driving situations not explicitly included  in today’s annex Check: How to integrate into the existing text

 

 

The manufacturer should identify critical scenarios in the ODD and responses to these critical scenarios. This includes traffic challenging scenarios for the ODD with other road users, emergency situation, severe failures).

Critical scenario identification not explicitly mentioned in today’s annex indirectly addressed by “safety concept” which includes operation both under fault- and non-fault conditions Check: How to integrate into the existing text

 

 

Traffic rules/lifetime :

 

The manufacturer shall describe how the ODD will be managed during the lifetime of the vehicle and how the function will be prevented from being activated in case the conditions required for the ODD cannot be guaranteed anymore by the manufacturer.

 

The manufacturer shall describe how the compliance with traffic rules will be managed through the lifetime of the vehicle.

 

Process regarding implementation of traffic rules not covered yet Check: How to integrate?

 

 

2)      Verification and Validation  by manufacturer

 

Need to include definition of verification/validation

 

The manufacturer shall demonstrate that all design solutions have been verified and validated (through simulation and physical testing including real world testing) by the manufacturer as individual subsystem and as part of the entire vehicle architecture and that the residual risk is acceptable for the driver and other road users.

 

In case the manufacturer rely on sub-contractors for subsystem, for the demonstration, the manufacturer may use documentation on  the verification, validation carried out by its suppliers but remains responsible of the overall safety of the ADS (from the type approval point of view).

Verification of functions by the manufacturer covered by paragraph 4.1.1.

Safety verification by the manufacturer covered by paragraph 4.1.2.

 

The acceptable residual risk is not defined yet and cannot be confirmed by the manufacturer Check : How to integrate?

 

Vehicle level (manufacturer) / component level (supplier) not explicitly mentioned in the current annex Check: How could the wording be improved? Add overview table with components supplied by manufacturers?

The manufacturer shall declare the scenarios taken into account for the ADS design, and declare the scenario management method used to select scenarios and choose a validation tool.

 

NOTE: After market retrofitting

 

Not covered yet Check: How to integrate
 

The manufacturer shall demonstrate how the performance of the following subsystems have been validated

 

- Perception and objects detection including mapping and positioning

- Characterization of the decision-making safety

- Documented data processing in case of continuous learning implemented.

- Human-machine interactions including the driver but also other road users 

- Supervision and remote monitoring (if applicable)

Verification perception and object detection including mapping and positioning is part of paragraph 4.1.1 and 4.1.2

 

Characterization of the decision-making safety  is part of paragraph 4.1.1 and 4.1.2

 

Driver HMI is part of 4.1.1 and 4.1.2
Not covered yet: interactions with other road users Check: How to integrate? 

The safety demonstration shall combine

- a quantitative pre-validation target (e.g., using validation acceptance criteria), documented by the manufacturer, demonstrating that the introduction of the ADS will not increase the overall level of risk for the driver and other road users.

- a scenario-based approach, taking into account that the most relevant critical scenario have been addressed and that the residual risk is acceptable for the driver and other road users.

 

Acceptable risk level for AD function are not defined for now. OEM shall explain why the risk is acceptable (free from unreasonable risk).

 

See above:

Covered by paragraph 4.1.1. Verification of functions by the manufacturer

Covered by 4.1.2 safety verification by the manufacturer

 

The acceptable residual risk is not defined yet and cannot be confirmed by the manufacturer Check : How to integrate?

Scenario based approach not explicitly mentioned Check: How to integrate? Update of 1. General necessary?

B) Evaluation by authorities

 

1) Evaluation of the design and its verification/validation by the manufacturer

 

Before assessing the manufacturer’s safety concept” the type-approval authority or the technical services acting on its behalf (hereafter referred to as type approval Authority) shall check that a valid safety management scheme is implemented within the manufacturer (See paper on the audit of the safety management scheme).

Covered by paragraph 1. General “methodology applied to the design process”

Covered by paragraph 3 Documentation “development process/method declared by the manufacturer”

Paragraph 3.4.2 for SW development “the design methods and tools” shall be identified”

The type-approval authority  shall make a finding of at least safety equivalence compared to at least to manual driving including driver assistant systems within the ODD (i.e. quantitative and qualitative target above)) based on the manufacturer’s safety evaluation report documenting scenarios taken into account, testing, validation methods listed above).

Action : check consistency ‘free from unreasonable risks, safety equivalence,etc)

Safety equivalence compared to at least manual driving not covered yet Check: How to integrate?

They shall verify that the hazard and risk analysis (i.e. hazards, occurrence and criticality) covers the scenarios of the system that are relevant to the ODD concerned.(link with subgroup 1).

Scenario-coverage with respect to the ODD not explicitly mentioned under 3.4.3 and 3.4.4 (assessment of the analytical approaches to cover the ODD-relevant scenarios) Check: How to integrate?
 

They shall assess that the logical chart of responses to risk (e.g. redundancy, manoeuvers) covers the range of identified scenarios. They shall check that the safety concept is implemented consistently in the design of the different functions of the AD system

Covered by 3.4.3 (“provide explanation of the design provisions”)
Link to identified scenarios is not mentioned yet Check: How to integrate?

 

They shall check that the verification/validation process is robust enough (simulation, track test, in use data) and manufacturer has mitigated risks as reasonably possible and meet the minimum performance requirements. . The validation shall in particular show that the human – machine interactions (including misuses by the driver and other road users) have been properly assessed, based on a relevant set of tests and users.

Verification of functions by the manufacturer covered by paragraph 4.1.1.

Safety verification by the manufacturer covered by paragraph 4.1.2.

 

They shall ensure that there is a transparent method of measuring the operational/run-time performance of the system.

Verification of functions by the manufacturer covered by paragraph 4.1.1.

Safety verification by the manufacturer covered by paragraph 4.1.2.

The assessments is to be concluded in several steps: audit of the processes in place in the manufacturer organization, assessment once the safety concept is established and assessment of the validation of the manufacturer, monitoring of the manufacturer once the product is on the market.

Different steps are not defined by today’ annex Check: Necessary?

 

 

 

2) Evaluation tests by authorities

 

The type-approval authorities or the technical services acting on their behalf shall carry out a minimum number of track tests to verify that the vehicle operates safely from the functional and operational safety point of view.

 

They shall check that the safety concept is implemented by the manufacturer. They shall confirm that the safety concept is valid to the vehicle type variant version to be covered by the assessment

 

Verification of functions by the technical service is covered by paragraph 4.1.1.

 

Safety verification by the technical service is covered by paragraph 4.1.2.

 

 

They shall in particular check the basic functionality as well as critical failure and driving scenarios through (track) testing. However, simulation can be used in case some critical scenarios/failure may prove to be difficult to be tested on track.

Type-Approval Authorities or Technical Services should keep track testing to the minimum number to verify safe operations. Simulations through validated tools should be used for a wider (and comprehensive) scenario coverage.

See comment above

They shall carry out  carry out test drives in real-world traffic to verify the carefulness and understandability of operation by other road users in non-critical scenarios and the respect of basic traffic rules.

Not covered by today’s annex Check: Update necessary for ALKS?
 

 

The minimum number of tests should include false negative and false positive test scenarios.

 

NOTE:  Goal : Inject perception problem.

 

Question: What does this technically mean? A perfect system would not have FNs and FPs…

How to test false positive and especially false negative on the road??

Even for a track test, how can one induce a system to fail by influencing its objects detection capabilities? The assessment of false or true is due to the system reaction.

 

Covered by 3.4.4.2 you do not test false negative and false positive but rather conditions that are likely to lead to false negative- and false positive-reaction
 

The type-approval authority carrying out the tests shall have access to the system that is necessary  to carry out the test under this section.

 

OICA comments: What is meant by “shall have access to the system”?

For type-approval, manufacturers will make all the relevant documents and test results open for inspection (accessible). However, it must be clear that neither a source code review nor a direct copy of the verification toolchain (databases, simulation toolchains, sensor models, reprocessing data) will be practicable due to Intellectual Property, time constraint and intrinsic complexity.

  Such wording is not mentioned in today’s annex CEL Update not necessary

 

 

3) Simulation evaluation by authorities (see Regulation 858/2018)

 

 

 

Simulation method may be used by manufacturers to demonstrate safety, subject to their validation by the approval authorities/technical services in accordance with the procedure for virtual testing in revision 3 of the 1958 Agreement. See Annex 2

OICA comments:

The purpose for simulation as described in the EU Regulation 2018/858 and Directive 2007/46/EC is different from what is needed for AVs verification. While the general simulation tool validation concept can be transferred, it needs some additional adaptation.

 

Simulation is not explicitly mentioned in today’s annex, but it is also not explicitly excluded from the scope Check: How to integrate? 

Manufacturer shall demonstrate the validation of the simulation tool and its scope (e.g. vehicle model, sensor model, recognition model or environmental model)

Not covered yet Check: How to integrate?

 

Authorities shall verify the validation done by the manufacturer to demonstrate the correlation of the expected results with track tests/on road tests (tool give representative results), scope, traceability, etc. as well as the validity of the simulation tool for the system concerned(applicability to the ODD concerned).

 

Not covered yet Check: How to integrate?

 

 

 

3 different axis have to be analysed : (1)definition of dedicated tests for CAE methodology validation (validity area), (2)evaluation of the CAE process with correlation criteria, application of the validated CAE methodology for approval virtual testing.

OICA question : What means CAE?
Unclear, what is technically required by this paragraph

 

  1. Set of tests to “optimize” and “validate” the simulation tool and understand its boundaries;
  2. Correlation between simulation tool and physical tests

 

The activities are complementary, why is there a need to differentiate them in 2 different stages?

Not covered yet Check: Necessary to integrate?

 

 

4) Audit and assessment to be conducted by a qualified independent 3rd party: criteria to be applied.

 

The type-approval authority or/and the technical services acting on its behalf shall have the necessary competences, certifications and training to carry out the vehicle safety assessment and tests listed above.

Reference to ISO standards?

Not covered yet Check: How to integrate?

???

 

 

 

 

5) Outcome of the evaluation by authorities/ Failed evaluation/Follow up:

 

Clear Failed evaluation: the hazard/risk analysis does not cover the hazard risks of the use case, the safety concept does not address the hazard/risk analysis identified, the verification/validation does not guarantee an acceptable level of risk or the scenarios taken into account are not transparent ,test failed.

Covered partly by paragraph 4.1.1.1. and 4.1.2.1 verification test results must correspond to the description in the safety concept

More elaborated common rating scheme needed? (e.g. observations, recommendations for the different level of criticality).

Not covered by today’s annex Check: Necessary to add?
 

What about follow up in case of failed evaluation?

 

What about monitoring after a successful evaluation? On road monitoring.==> Covered by the pillar on road monitoring

Not covered by today’s annex Check: Necessary to add?

 

 

 

6) Transparency/information sharing amongst type-approval authorities on the assessment carried out.

 

As the future new assessment, method will rely more on the assessment by authorities than standardized technical requirements, to continue to ensure mutual recognition more transparency will be needed on what was done by the authorities while at the same time guaranteeing the protection of intellectual property for vehicle manufacturers.

 

Where do we put the limit between the two objectives? See example part 1 of Annex 1+ report on testing done by TAA.

 

Minimum set of information required to understand the system (functionality, operation) can be shared

 

Description of the function (ODD) focusing on the functions available to the driver and other road users + test report done by the TAA

Check: How to integrate information document? 

Action item

Systematic transmission or on request?

 

Shall some information be available to some stakeholders beyond traditional type-approval authorities and technical services (e.g. consumer, registration authorities, insurers, police)

 

OICA comments:

Consumer: Educated and trained as requested by the provisions of the UNECE Framework Document on AVs. 

Registration authorities : Detailed Type Approval information out of their scope. They need to be aware of the vehicle being “capable of automation” and “system being enabled in the vehicle”.

Insurers: No rational behind sharing technical information with insurers.

Police: DSSAD / EDR provisions will be applied.

 

 

 

 

References :

 

UN Framework document on automated vehicles

FRVA functional requirements

VMAD subgroup work on scenarios.

ISO 26262/SOTIF 21448

UL 4600

EU guidelines: https://ec.europa.eu/growth/content/guidelines-exemption-procedure-eu-approval-automated-vehicles_en

Annex 6 o UN Regulation 79.

EU Regulation 858/2018

Others??

 

OICA comments :   UL4600 is not a reference in this document; it is not a standard rather a compendium. Should be deleted here.

 

 

 

 

 

Annex 1: Information to be provided by manufacturer (draft- to be reviewed )

 

1.OVERALL DESCRIPTION OF THE SYSTEM

 

a) Automated System Type Definition

b) Operational Domain (Speed, road type, country, Environment, Road Conditions

c) Main conditions for Minimum risk manoeuvres and transition demands

d) Main automated Driving Functions (functional architecture)

e) Basic Performance (e.g. max. lateral acceleration, OEDR …)

f) Tasks other than driving technically enabled by the system

 

2. SYSTEM PERFORMANCE IN THE AUTOMATED DRIVING MODE

 

A. ENVIRONMENT PERCEPTION

 

a) With respect to operation domain

b) Lanes / Objects

c) Redundancy (with respect to system performance)

d) Sensor monitoring:

    1. Plausibility check with respect to misuse

    2. Implemented monitoring system or degradation considered.

e) Connectivity

f) Maps

 

Question OICA : What does “Connectivity” in terms of Environmental Perception mean? Communication vehicle with backend?

 

B. DYNAMIC DRIVING TASK AND INTERACTION WITH OTHER ROAD USERS

 

a) Have a predictable and careful behaviour:

1. Driving in accordance to the speed limits (explicit and implicit)

2. Obeying passing restrictions

3. Adapting the speed of the vehicle to environmental conditions (e.g. rain, fog, curves, hilltops, sun glaring) affecting:

   • Adhesion of the road

   • Viewing distance of the system

4. Keeping the required minimum distance to other road users

5. Rules regarding the preferred lane of travel (“Drive on the rightmost lane”)

6. Compliance with relevant country specific traffic rules (respecting road markings and road signs)

 

 

 

 

b) React to:

1. Other vehicles within the ego lane or in the  neighbouring lanes (e.g. other vehicle cutting into the ego lane, neighbouring vehicle driving too close or across the lane marking)

2. Vulnerable road users (if applicable in the ODD)

3. Police and Emergency Vehicles

4. Law enforcement injunctions (police control, compliance with officers' regulations)

 

2. DRIVER INTERACTION

 

a) Activation / Deactivation / Modes (on / off / standby)

b) Overriding / Human driver priority

c) Human Machine Interface (HMI):

   1. Driver Information (Operation Status, Failure)

   2. Optical Warning Signal (type and operation mode)

   3. Acoustic / Haptic Warning Signals (type and operation mode)

d) Driver Presence and Responsiveness Recognition System

e) Extract of the relevant part of the owner`s manual

f) Means to prevent misuse and manipulation

 

3. TRANSITION OF THE DRIVING TASK

 

a) Planned:

   1. Boundary conditions

   2. System behavior

   3. System performance

b) Unplanned (incl. mayor system failure):

   1. Boundary conditions

   2. System behaviour

   3. System degradation

   4. System performance

 

4. MINIMUM RISK MANOEUVRE

 

c) Description  of the different risk manoeuvres for the different scenarios (e.g. planned and unplanned events)

d) Emergency (only in case of imminent danger of a collision):

   1. Boundary conditions

   2. System behaviour

   3. System performance

 

5. EDR/ DATA STORAGE SYSTEM

 

a) Type of Data stored

b) Storage location

c) Storage duration

d) Means to ensure data security and data protection

e) Access to the data

 

Question OICA: Why part of this assessment? Separate regulatory activity covers EDR/DSSAD (IWG DSSAD/EDR)

 

6. CYBER SECURITY

 

Description of the cyber security and software update management scheme

Description of the different risks and measures put in place to mitigate these risks.

Description of the update procedure.

Question OICA : Why part of this assessment? Separate regulatory activity covers CS/OTA (IWG CS/OTA)

 

7. SAFETY BY DESIGN, VERIFICATION AND VALIDATION

 

Design and validation process:

-Assessment of the functional and operational safety for the automated system design.

Risk analysis:

List of critical scenarios identified for the ODD.

Safety by design principle

-Verification/validation used by the manufacturer (simulation, track test, on road tests).

   Test of the functionality in nominal conditions

   Tests in case of system failure/critical situations:

      1. Measurement equipment used

      2. Test conducted by the technical service/type-approval authority

      3. Description of in-use tests

-Evaluation of the residual risk

-Process used for field monitoring

 

Question OICA : What is technically meant by “in-use-tests”?

What is the accepted “residual risk”?

 

 

 

 

 

 

 

 

 

 

 

 

 

8. INFORMATION PROVISIONS TO USERS

 

Model of the information provided to users.

 

 

Annex 2: : General conditions for virtual testing methods (draft)

 

  1. Virtual test pattern

 

The following scheme shall be used as a basic structure for describing and conducting virtual testing:

(a) Purpose;

(b) Structure model;

(c) Boundary conditions;

(d) Load assumptions;

(e) Calculation;

(f) Assessment;

(g) Documentation.

 

2.1. Mathematical model

The mathematical model shall be supplied by the manufacturer. It shall reflect the complexity of the structure of the wheeled vehicles, equipment and parts to be tested in accordance with the requirements of the UN Regulations concerned and its boundary conditions.

 

The same provisions shall apply, mutatis mutandis, for testing components independent of the vehicle.

 

Question OICA: Does this mathematical model refer to the vehicle simulation model or to the complete simulation environment?

 

 

 

 

Remark OICA: Simulation in the context of vehicle certification should focus on the whole vehicle reaction and not on single components. Why this reference to components?

 

2.2. Validation process of the mathematical model

The mathematical model shall be validated in comparison with the actual test conditions.

 

To that effect, physical testing shall be conducted as appropriate for the purposes of comparing the results obtained when using the mathematical model with the results of a physical test. Comparability of the test results shall be proven. A validation report shall be drafted by the manufacturer or by the technical service and submitted to the approval authority.

 

Any change made to the mathematical model or to the software likely to invalidate the validation report shall be brought to the attention of the approval authority which may require a new validation process to be conducted.

 

2.3. Documentation

The data and auxiliary tools used for the simulation and calculation shall be made available by the manufacturer and be documented in a way suitable for the technical service.

The wording “shall be made available” is misleading. It could be understand that manufacturers have to provide all tools including hardware/software and licenses to the authority and that these tools are operational at the authorities facilities OICA/CLEPA proposal: “documentation/data/tools shall be made open for inspection…”

 

3.Tools and support

 

At the request of the approval authority or the technical service, the manufacturer shall supply or provide access to the necessary tools including appropriate software.

 

In addition the manufacturer shall provide appropriate support to the approval authority or the technical service.

 

Providing access and support to a technical service does not remove any obligation of the technical service regarding the skills of its personnel, the payment of licence rights and respect of confidentiality.

 

The wording “supply or provide access” is misleading (see above) “manufacturer shall make open for inspection the necessary tools…”

 

 

The sentence “In addition the manufacturer shall provide appropriate support to the approval authority or the technical service.” is self-evident and can be deleted.

 

It would be too complex within a certification process to require manufacturers to provide all tools including hardware/software and licenses to the technical service or to the approval authority and that these tools would be operating at the authority’s facilities. What manufacturers can in the available timeframe realistically do  is to make such tools and related documents open for inspection.