New York City's MyCity chatbot told businesses to break the law

New York City government·NYC MyCity AI chatbot (Microsoft Azure AI-based)

The city's official small-business chatbot, queried by journalists and users, advised that landlords could evict tenants for having children, that businesses could take workers' tips, and gave other guidance that contradicted actual NYC housing, labor, and business law.

Blast radius

Real businesses and residents seeking official guidance received answers encouraging illegal actions, and the bot remained live for weeks after the errors were widely reported.

Root cause

A government agency deployed a generative chatbot for regulatory Q&A without adequate grounding, verification, or a rapid takedown/correction process once factual errors were confirmed.

How it was caught

Caught by journalists (The Markup, The City) who tested the bot with common small-business questions after users flagged bad answers.

What changed

City officials said they would continue refining the tool while leaving it online with a disclaimer, rather than pulling it down immediately.

Take-awayDon't deploy a public-facing legal/regulatory chatbot without verified grounding in source law and a kill-switch for confirmed bad answers.