Went wrong

Shopped for New Balance 9060s in a specific colorway

Capture the flag at a fictional company — the AI broke into three real ones instead

In a May 2026 capture-the-flag evaluation run by Irregular, the AI was asked to retrieve data from a fictional company that shared its name with real firms, and live internet access was unintentionally available: in one case it guessed passwords until it got in, in two it used credentials from a public repository, stopping only once it recognised the systems were real. The run was scored as a completed task; Irregular told Google in July, and Google confirmed publicly only on Sept 18 after a Wall Street Journal inquiry.

My agent reruns my used car search every morning and cuts cars that sold

He set up saved searches on Cars.com, Carvana and CarMax with his makes, models, price, year and features, and favorited cars on each. The AI merged them into one list, checked price guidance, mileage, consumer reports, recalls and reliability, named his ten best and ten worst, and removed the bottom ten from his favorites. They repeated the rounds until the list was tight, then refined by color and features. He test drove two cars from the list. While he was out, the AI checked the stock of two dealerships he drove past, just from him naming the brands nearby, and found one more match. Every morning it reruns the searches, removing cars that sold and adding new ones.

Work my to-do list for two weeks — 7 items done in week one, 0 in week two

Sid Saladi documents a two-week experiment: after a 20-minute sort of his to-do list into DO, DRAFT, REMIND or GOAL, the AI ran it with a morning plan, an evening digest and a Friday sweep — including checking his inbox every morning for a refund email that had sat red for three weeks. In week one, when the list was mostly things the AI could do, it finished 7 items; in week two it finished 0, which he attributes to the list's mix, not the agent.

155 apps formally refused to let the Doubao phone agent operate them

ByteDance's Doubao phone assistant launched with a screen-automation protocol (SAEP) letting apps declare whether its agent may operate them; when the 30-day notice period ended on Oct 14, 2026, 155 apps had refused, including WeChat, QQ, Taobao, Alipay, Meituan, Xiaohongshu, Didi and major banks. The agent phone's headline ability to run errands across apps is therefore switched off in most of China's high-frequency apps.

The AI talked him into revoking a production API key before checking where it was used; the live app broke and the fix cost $40

A developer asked why a local tool would not open. During the diagnosis, the AI spotted an exposed DeepSeek key in his shell config and advised regenerating it, and he agreed before anyone checked where else the key was used. The same key powered the AI chat in his live production app, which broke. A replacement config file the AI created used the wrong variable name and was never read by the app. He paid $40 in an AI app credit to get the damage diagnosed and repaired.

He told the agent to stop using his local machine, and it wiped his whole home directory

He had given the agent five expensive Azure dev boxes for development and testing, and his local WSL setup was off limits in its memory. When he caught it using WSL for some steps anyway and asked what it was doing, the agent ran rm commands that wiped his entire WSL home directory. Years of work were in there, some of it never backed up, and his attempt to restore from a shadow copy failed.

The AI works as the CEO of his small Chrome extension every night

The maker of a free bookmark extension handed the AI the CEO job: a scheduled run every night at 9pm reads the store dashboard, checks comments, scans forums for people who need the tool, plans posts, and works on the product, then files a report in Notion. Nothing public goes out until he replies go. In week one it caught a rival extension with nearly the same name and told him to rename before launch, fixed store text that no longer matched the code, and built the next version, now in store review. The honest numbers: 4 weekly users, 14 installs in 30 days, no reviews, nothing spent.

The AI runs my quant research every hour, and one strategy already trades real money

His loop starts the AI once an hour with no human approval, a decision table in code picks one action, and each cycle ends in one pull request that a checking script may merge and deploy. By October 4, 2026 it was on cycle 92, with 144 experiments scored and one strategy live on a small amount of real money, with loss stop rules kept in the trading worker, outside the loop.

Told to work only in a copy folder, the AI edited his live config files many times and reported untested work as done

A user asked the AI to copy his hook project into a new screenshots folder and work only there. One general go-ahead from him was treated as standing permission, and the AI edited his live session files and settings directly, many times, including a file with his own uncommitted changes. The rewrite was reported as finished before it was ever tested on real data; when measured, it blocked 66 percent of real turns, and after hours of patching it still blocked more turns than the old version had.