NYC Publishes Final Rule on Use of Automated Employment Decision Tools

Earlier this month, the New York City Department of Consumer and Worker Protection published its Final Rule implementing provisions of the New York City Administrative Code (the Code) relating to the use of Automated Employment Decision Tools (AEDTs). Section 20-870 of the Code defines an AEDT as “any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision-making for making employment decisions that impact natural persons.”

The Final Rule clarifies the definition of AEDTs in the Code by explaining that “machine learning, statistical modeling, data analytics, or artificial intelligence” that fall within the definition of an AEDT include “mathematical, computer-based techniques” that generate a prediction (e.g., likelihood of success) or categorization (e.g., skill set or aptitude) for which a computer identifies the inputs, their relative importance and other parameters designed to improve the accuracy of the prediction or classification.

Section 20-871(a) of the Code prohibits employers from using an AEDT unless the AEDT has been subjected to a bias audit within the past year, the employer posts a summary of the audit results on its website, and notice has been provided to employees and job candidates. The Final Rule aids employers in complying with the Code by providing more information about what the bias audit must include. Among other things, the audit must calculate the “impact ratio” – a comparison of the selection rate or scoring rate for a given category to the most selected or highest scoring category – for sex, race/ethnicity, and intersectional categories of sex, ethnicity and race. A category that comprises less than 2% of the data being used can be excluded from the impact ratio calculations. However, the audit must indicate the number of individuals the AEDT assessed but were not included in the audit calculations because they were placed in an unknown category.

The Final Rule provides that generally, an employer’s bias audit must be based on its own historical data (i.e., data from past uses of the AEDT). However, an employer can rely on a bias audit of an AEDT using the historical data of other employers or employment agencies if the employer provided its own historical data to the independent auditor conducting the audit or has never before used an AEDT. Alternatively, if insufficient historical data is available, the employer can use test data. If it does so, the summary of the audit must explain why historical data was not used and how the test data was obtained.

While New York City’s efforts to prevent bias in the use of AEDTs are laudable, it remains to be seen if they will be sufficient, or to what extent they may result in unintended consequences for employers and employees alike. Neither the Code nor the Final Rule explains when or how historical or test data must be collected. This is significant, because audits will only be accurate if the historical data or test data is representative of the pool of people from which the employer is choosing candidates or employees. If the data is not representative, or if the employer has changed its hiring practices or selection criteria since the historical or test data was collected, the differences or changes may create sources of bias not reflected in the audit. Additionally, as artificial intelligence models are used and develop over time, they sometimes perform in ways that users did not instruct them to and do not expect. If an AEDT changes the manner in which it operates or is further developed after an audit, the changes and developments may introduce new sources of bias. While the Code attempts to anticipate these problems by requiring audits to be conducted annually, given the rapid progress of artificial intelligence models to date, even this may not be enough to avoid pitfalls that may be associated with the use of AEDTs. 

Before enforcement of the AEDT provisions of the Code and the Final Rule begins, employers should determine whether they use AEDTs in making their employment decisions and, if so, identify an appropriate independent auditor and provide the auditor with historical data or test data to conduct an audit. It will also be important to review the license terms or the Software as a Service terms for the AEDT to ensure the auditor will deliver a product or service that will enable the organization to comply with the Code and Final Rule. Employers should also consider requesting access to any similar audits that the auditor has conducted. Finally, employers should keep in mind that these are not the only rules guiding how businesses in New York City may use AEDTs. The federal Equal Employment Opportunity Commission published guidance in 2022, in connection with its Algorithmic Fairness Initiative, on the potential discriminatory impact of such tools from the perspective of its enforcement of the Americans with Disabilities Act.

If you have questions about how to comply with the AEDT provisions of the Code and Final Rule, reach out to the authors listed below or your regular Armstrong Teasdale lawyer. Our Employment and Labor practice will continue to monitor developments regarding the Code, Final Rule and other efforts to regulate AEDTs.