Shortly after the open on December 7, 2020, Interactive Brokers, a platform that is typically resilient and crash-proof, had major issues. Customers of Interactive Brokers started complaining on Twitter and posting reports to DownDetector very early in the morning, peaking at 9:30 AM Eastern Time, saying they were unable to log in to their accounts. The problems were traced to a data storage system operated by an unnamed third party vendor.
Interactive Brokers was not alone: other brokers listed on Downdetector reports the same day include Robinhood, Fidelity, and Charles Schwab. There were no outages reported at E*TRADE, Merrill Edge, or Vanguard.
KEY TAKEAWAYS
- Interactive Brokers, usually a very stable brokerage, suffered a major system outage the morning of December 7, 2020.
- The outage was traced to a data storage system operated by a third party.
- Customers continued to report problems logging in for several hours after IBKR stated the issue was resolved.
The IBKR Twitter feed acknowledged the problems at 9:00 AM Eastern, with a brief tweet saying, "We are experiencing technical difficulties and we are working to restore services." At 10:15 AM Eastern, a post detailing the issues declared, "We have experienced a significant failure in multiple segments of a highly resilient data storage system. We have engaged the vendor for analysis and recovery. Many services have already been restored and the remaining services are expected to be available promptly."
:max_bytes(150000):strip_icc()/IBKRtwitter12072020-21ef6d842dbe4f77b5b0aabbbd8b91a9.jpg)
In an email, IBKR's Director of Media Relations, Kalen Holliday, said, "We recognize the dislocation this has caused to our clients, and express our apologies." Clients were not mollified, however, continuing to complain about the inability to login and manage positions during a fast-moving market.
:max_bytes(150000):strip_icc()/IBKRdowndetector12072020-fb2ce505e1594a4982c5090bd84ab2a9.jpg)
The firm's system status page reported at 1:42 PM Eastern, "Systems are returning to normal functionality," but continued to indicate that there were problems. Downdetector continued to report outages and login issues as of 1:30 PM Eastern, but the number of issues is declining.
During the period that the systems kept many customers offline, Twitter was awash in complaints and threats to sue. The IBKR Customer Agreement, a 12-page document packed with legalese and signed without reading by most customers, includes a section describing the company's "As Is" clause. It's on page 8, so it's buried fairly deeply in the document (point 28 on the list). This clause is in all caps and bold type and states, "IB shall not be liable to customer by reason of delays or interruptions of service or transmissions, or failures of performance of the IB system, regardless of cause..." The following point, number 29 on the list, charges the customer with the responsibility of maintaining alternative trading arrangements in the event the IB system is unavailable. This sort of clause is standard in the online brokerage industry, so lawsuits are seldom settled in a client's favor.
At the end of the day, systems were all back to normal. Reports to Downdetector had dwindled significantly.
:max_bytes(150000):strip_icc()/IBKREOD12072020-8d944f9aece946bea2119e7c03dd7aec.jpg)
On December 8, Interactive Brokers' CEO Milan Galik sent an email to all clients, explaining that the data system, provided and serviced by a third party vendor, that is designed to minimize failures, "did not work as expected today." Malin says that IBKR worked with the vendor to figure out what had happened, and began restoring systems so that most clients were able to access most services by late morning. The remaining services were brought back online over the rest of the trading day.
Malin says, "Any technology can fail, and we take the quality and resiliency of our systems very seriously. We will incorporate the vendor’s analysis into our own independent assessment of today’s partial outage to make it even more unlikely that it could occur again in the future."