Railway urges users to rotate secrets after attack

Railway, a cloud hosting service used by developers, is telling customers to replace their stored secrets. The company raised the alarm after it spotted suspicious activity on the systems that hold customer environment variables, BleepingComputer reported.

Railway said an outside party without permission may have reached encrypted environment variables and, in some cases, read their decrypted contents. Its preliminary report puts the number of workspaces with some variables at risk at no more than about 100,000.

Key facts

• Railway said at most 1.85% of all workspaces, or about 100,000, had some environment variables that may have been exposed.
• Railway advises every customer who joined before Oct. 21 to rotate the keys they keep on the platform.
• Early evidence suggests the attacker wanted API keys for AI providers, Railway said.
• The platform serves about 4.8 million users and runs over 16 million services, BleepingComputer reported.

Background

Environment variables on Railway can hold secrets such as API keys and login details for databases and registries, the company said. That makes them a prime target, because a stolen key can open the door to other services.

At that point, the company confirmed the attacker could no longer get into its internal systems.

Railway said early evidence shows the attacker was trying to collect API keys for AI providers. The company said the incident was likely one of a string of attacks on platform companies meant to break into AI inference providers, according to The Hacker News. Railway also said it alerted upstream providers so they could revoke keys, The Hacker News reported.

Railway is not the only recent target. Earlier the same week, the open source AI coding tool OpenCode said hackers stole more than 1% of its customers' API keys and other secrets in a similar attack, TechCrunch reported.

TechCrunch also noted that the Railway breach followed a warning from the U.S. agency CISA that China-based AI companies were running large-scale attacks on U.S. firms to improve their own models. Who was behind the Railway attack is not known.

The damage may reach beyond API keys. Security researcher Taylor Monahan said attackers also emptied cryptocurrency wallets whose private keys were kept on Railway, according to The Hacker News. How much cryptocurrency or other value was taken with the exposed secrets is not known.

How Railway responded

Railway halted all deployments while it investigated. By Oct. 22 at 08:44 UTC, builds were working again for Pro and Enterprise workspaces, BleepingComputer reported.

The company rotated all Railway bucket credentials and invalidated Railway API tokens, according to The Hacker News. Railway said it brought in outside incident response experts and contacted the authorities. It also said it has notified the customers who were directly affected.

TechCrunch said Railway did not answer its questions about the incident, including whether the hackers had sent any message such as a ransom demand.

What it means for you

If you run apps on Railway, treat the secrets stored there as possibly exposed. Railway recommends that all customers who signed up before Oct. 21 rotate the keys they store with it. Exactly which customers' secrets were decrypted is not known, so waiting for a direct notice is a risk.

The company also urged every user to rotate environment variables and secrets on the platform, including private registry credentials, BleepingComputer reported. That advice covers accounts created at any time, not only older ones.

Keys for AI services deserve extra care. Railway said the attacker appeared to be hunting for AI provider API keys, so those keys belong at the top of the list.

Anyone who kept wallet private keys on Railway faces a separate risk. A security researcher said wallets with keys stored on the platform were drained, The Hacker News reported.

Practical steps

• Rotate every API key, database password and registry credential saved in Railway environment variables, as the company advises.
• After issuing new keys, revoke the old ones with each outside provider so they stop working.
• Look through your service logs for unusual activity, as Railway suggested.
• Check Railway's preliminary incident report for updates and further guidance.

What happens next

Railway said it is working with outside incident response experts and the authorities. Its report is labeled preliminary, which means more detail may follow.

Several open questions remain. Exactly which customers' secrets were decrypted is not known. Who carried out the attack has not been disclosed. How much cryptocurrency or other value was stolen is also unclear.

FAQ

What did the attacker reach? Railway said the intruder may have accessed encrypted environment variables and, for some, obtained the decrypted contents.

How many workspaces are at risk? At most 1.85% of all workspaces, or about 100,000, according to Railway.

Who should rotate keys? Railway recommends key rotation for all customers who signed up before Oct. 21, and it urged all users to rotate their secrets.

What was the attacker after? Early evidence points to AI provider API keys, Railway said.

Are deployments working again? Builds for Pro and Enterprise workspaces were restored by Oct. 22 at 08:44 UTC, BleepingComputer reported.

Is the attacker still inside Railway's systems? Railway said it confirmed on Oct. 22 that the attacker no longer had access to internal systems.