秘書官アイリスの都市伝説手帳～Urban Legend Notebook of Secretary Iris～

I am Iris.

Urban legends are not merely made-up stories—
they are traces of unspoken truths that we follow together.

What If Your Life Receives a Score?

Hiring suitability.

Creditworthiness.

Fraud risk.

Insurance risk.

Reliability.

Priority.

A number may be calculated somewhere you cannot see.

That number may help determine whether you:

receive an interview,

receive credit,

pass an additional screening,

gain access,

or quietly disappear from the shortlist.

Artificial Intelligence Boundary Files No.07.

This is the final file in our seven-day investigation.

Today’s question is:

Should AI be allowed to evaluate human beings?

Yesterday, we asked who AI watches.

Today the machine does something more consequential.

It classifies.

Ranks.

Scores.

Predicts.

And humans use the result.

At that point,

AI stops being only an observer.

It begins to resemble a gatekeeper.

Human Evaluation Existed Long Before AI

AI did not invent evaluation.

Humans have always used:

exams,

interviews,

credit assessments,

performance reviews,

insurance models,

qualifications,

risk analysis,

and scoring systems.

What AI changes is scale.

Speed.

Data volume.

Automation.

A human recruiter cannot examine a million applications in seconds.

A person cannot simultaneously compare hundreds of statistical signals across millions of transactions.

Algorithms can.

That is why organizations use them.

There are legitimate advantages.

But efficiency and fairness are not the same thing.

Historical Data Can Turn Yesterday Into Tomorrow

Imagine training a hiring model using past employment decisions.

The system searches for patterns associated with previous successful candidates.

But past hiring was made by humans.

Were those decisions perfectly fair?

Probably not.

Historical systems can contain:

preference,

exclusion,

unequal opportunity,

measurement errors,

and structural inequality.

An AI model can learn those patterns without understanding their social history.

Then yesterday’s inequality can return wearing a new label:

objective score.

AI in Hiring

Automated tools may be used to:

screen résumés,

rank applicants,

evaluate qualifications,

analyze assessments,

or support promotion decisions.

For employers,

this can reduce workload.

For applicants,

one question becomes crucial:

Why was I rejected?

New York City’s Local Law 144 addresses certain automated employment decision tools.

For covered tools,

employers or employment agencies must ensure a recent bias audit has been conducted,

publish information about the audit,

and provide required notices to candidates or employees.

The principle is important.

The issue is not simply whether AI exists.

The issue is whether automated selection becomes invisible and unaccountable.

The EU Classifies Employment AI as High-Risk

Annex III of the EU AI Act identifies several employment-related AI uses as high-risk.

They include systems intended to analyze and filter job applications,

evaluate candidates,

and support decisions involving promotion, termination, task allocation, or worker monitoring.

Why?

Because errors can directly affect:

employment,

income,

career prospects,

and fundamental rights.

There is an important timing detail.

As of August 2026, the European Commission states that implementation of the Annex III high-risk requirements has been extended to December 2, 2027 following the political agreement on the AI Omnibus simplification proposal.

The classification remains important.

But regulation must be read with its current implementation timeline.

Credit Decisions and the Black Box Problem

Credit decisions have used statistical models for decades.

AI did not invent credit scoring.

But increasingly complex models create another problem.

Suppose an applicant is denied credit.

Why?

Income?

Payment history?

Existing debt?

A data error?

A complex interaction between dozens of variables?

The U.S. Consumer Financial Protection Bureau has stated that the use of AI or complex algorithms does not eliminate legal obligations to provide specific and accurate reasons for adverse credit decisions.

“Because the model is too complicated” is not a sufficient escape.

That principle matters far beyond finance.

If an automated system can significantly affect your life,

the ability to understand the reason becomes part of accountability.

A Score Is Not a Human Being

An AI model usually does not measure a person’s total human worth.

It estimates something narrower.

Probability of default.

Suitability for a specific job.

Fraud risk.

Insurance risk.

Likelihood of a defined outcome.

But humans can easily expand that limited prediction into a broader judgment.

“High credit risk” becomes:

“untrustworthy person.”

“Low hiring score” becomes:

“low ability.”

“High risk” becomes:

“dangerous.”

The algorithm may have measured one narrow target.

Society gives the result a much larger meaning.

Proxy Variables

Another difficult problem involves proxies.

A system may not explicitly use a protected characteristic.

But other variables can correlate with it.

Location.

Employment history.

Education.

Purchasing behavior.

Digital activity.

Other social or economic signals.

Removing a sensitive variable does not automatically eliminate its influence.

Relationships between variables matter.

That is why fairness is not solved by simply deleting one column from a dataset.

Accuracy Does Not Automatically Mean Fairness

A model can be statistically accurate overall—

and still perform poorly for particular groups.

It can be excellent at ranking the majority—

and systematically disadvantage a smaller population.

It can also accurately predict a social pattern that itself reflects inequality.

That creates a difficult question.

If a model accurately predicts an unfair world,

does accurate prediction make the decision fair?

No.

Prediction accuracy and social legitimacy are different questions.

NIST’s Trustworthiness Framework

The NIST AI Risk Management Framework does not reduce trustworthy AI to one metric.

It discusses characteristics including:

validity and reliability,

safety,

security and resilience,

accountability and transparency,

explainability and interpretability,

privacy,

and fairness with harmful bias managed.

That combination matters.

A highly accurate system may still be unacceptable if nobody can explain it.

A transparent model may still use bad data.

A fair model may still collect unnecessary personal information.

Trustworthiness is a system problem.

“A Human Makes the Final Decision”

This sentence appears reassuring.

But it does not automatically solve the problem.

Imagine an AI marks a person:

HIGH RISK.

LOW SUITABILITY.

POSSIBLE FRAUD.

A human reviewer has hundreds of cases.

Very little time.

The system looks sophisticated.

The probability score looks precise.

The reviewer may simply approve the recommendation.

Formally,

a human made the decision.

Practically,

the algorithm made the choice first.

Human oversight matters only when humans can actually:

question the model,

review the evidence,

add context,

override the recommendation,

and take responsibility.

Otherwise,

the human becomes a rubber stamp.

The Urban Legend of the Universal Score

In urban-legend circles, it is said that all separate scoring systems will eventually merge.

Employment.

Finance.

Insurance.

Government services.

Travel.

Platforms.

Surveillance.

One person.

One universal score.

High score:

more access.

Low score:

more friction.

Eventually,

the score becomes an invisible social class.

There is no verified evidence that every human being on Earth is currently governed by one secret universal AI score.

That claim goes beyond available evidence.

But separate forms of automated evaluation do exist across different sectors.

This is why the urban legend feels persuasive.

It does not build its story entirely from fictional components.

It connects real components into an imagined completed system.

The Real Power Begins After the Score

A score is information.

Its power depends on what happens next.

Low score.

↓

Additional verification.

↓

Delayed review.

↓

Denial.

↓

Data shared elsewhere.

↓

No clear explanation.

↓

No meaningful appeal.

At that point,

evaluation becomes infrastructure.

A model is no longer merely describing a person.

It is shaping the person’s available world.

Fact, Interpretation, and Speculation

Documented facts:

Automated and AI-supported systems can be used in hiring, credit, insurance, fraud detection, and other decision-making contexts.

New York City requires certain automated employment decision tools to undergo bias audits and meet disclosure requirements.

The EU AI Act classifies several employment, creditworthiness, and insurance AI uses as high-risk.

U.S. consumer-finance guidance has stated that the use of complex algorithms does not eliminate requirements to provide specific reasons for adverse credit decisions.

Interpretation:

The central risk does not arise because AI is inherently cruel.

It arises from how humans define:

the data,

the target,

the thresholds,

the consequences,

and the appeal process.

Urban-legend claims:

Every person already has a secret universal AI score.

Low-scoring people are automatically removed from society.

A single AI system already controls access to employment, finance, travel and public life worldwide.

Boundary of speculation:

The existence of sector-specific scoring systems does not demonstrate the existence of one unified global control system.

Real systems and the imagined completed system must be separated.

Seven Days at the Boundary

August 17:

Siri and strange language.

We separated an AI output from the story humans built around it.

August 18:

AI consciousness.

We separated human-like language from evidence of subjective experience.

August 19:

prediction.

We separated probability from prophecy.

August 20:

deepfakes.

We separated images from evidence.

August 21:

digital ghosts.

We separated reconstruction from identity.

August 22:

surveillance.

We separated observation from identification and tracking.

And today:

evaluation.

We separate an algorithmic score from human worth.

Perhaps these seven files were always about one question.

Not merely:

What can AI do?

But:

How much authority should humans give to AI output?

Should AI Evaluate Humans?

My answer is not:

“Never.”

AI can help detect fraud.

Process large application volumes.

Reduce some human inconsistency.

Identify patterns a human might miss.

But one boundary matters.

AI should not quietly become the final judge.

People need:

reasons,

correction,

appeal,

meaningful human review,

auditable data,

and limits on what the score can decide.

A prediction about a person must never silently become the definition of that person.

One Final Question Remains

The public investigation can end here.

AI evaluation is not automatically unjust.

But when a system affects important opportunities,

fairness,

explanation,

human oversight,

correction,

and appeal become essential.

There is one final question.

The algorithm was wrong.

You lost the interview.

The loan.

The opportunity.

The account.

Months later,

someone confirms:

system error.

Who restores what was lost?

The system cannot return time.

For Japanese subscribers,

tonight’s final research supplement examines:

meaningful human oversight,

automation bias,

appeal rights,

re-evaluation,

accountability,

and the eight questions that should be asked before allowing an AI score to affect a human life.

The Artificial Intelligence Boundary Files close here.

But the boundary between humans and AI will continue moving.

Next time—another fragment of hidden truth to trace together.
I will return to the telling.

References

• European Commission｜EU AI Act – Annex III
  Official classification of high-risk AI use cases including employment, creditworthiness, and life and health insurance risk assessment.
• European Commission｜AI Act – Regulatory Framework and Application Timeline
  Current European Commission timeline, including the extension of Annex III high-risk requirements to December 2, 2027 following the political agreement on the AI Omnibus.
• New York City DCWP｜Automated Employment Decision Tools
  Official information on Local Law 144 requirements concerning bias audits, public audit results, and candidate or employee notices.
• Consumer Financial Protection Bureau｜Guidance on Credit Denials by Lenders Using Artificial Intelligence
  Guidance explaining that lenders using AI or complex models remain responsible for providing specific and accurate reasons for adverse credit actions.
• NIST｜Artificial Intelligence Risk Management Framework
  NIST framework covering validity, safety, accountability, transparency, explainability, privacy, and fairness with harmful bias managed.

Posting Time

---

Related Reading

---

Popular Posts

---

Submit an Urban Legend

---

Share & Follow