Public XRP ledger nodes fall out of sync - some XRP services down, others disrupted

TL;DR: the most-used XRPL nodes fell out of sync due to too many requests. Many services use these nodes as running a node is expensive, owing to XRP's huge ledger and high hardware requirements.

For more info, see this Twitter thread, this Twitter thread and this news item.

What seems to have happened is that two nodes run by Ripple (s1 and s2) fell out of sync with the network.

• The XRPL can't handle the current load well
• The amount of Trust Lines & tokens has exploded beyond numbers expected/tested
• XRPL public nodes, clusters are being scaled up, but that isn't ready yet

These are nodes being run by Ripple for public use, but apparently too many users rely on these "free-to-use" nodes rather than spinning up their own nodes. We're not just talking small users here - apparently many exchanges are running these free nodes as well and doing API calls on them.

What we need right now are Full History Nodes. That's really expensive hardware.

This seems to illustrate the issue. The reason that many use these public nodes is that running a full node is.. cumbersome, to say the least. XRP enthusiasts correct me if I'm wrong, but it seems like these are the requirements:

• Operating System: Ubuntu (LTS) or CentOS or RedHat Enterprise Linux (latest release)
• CPU: Intel Xeon 3+ GHz processor with 8+ cores and hyperthreading enabled
• Disk: SSD / NVMe (10,000 IOPS or better)
• RAM: 64 GB
• Network: Enterprise data center network with a gigabit network interface on the host

On top of that, there are huge storage requirements - roughly 18 TB that needs to be stored on SSDs, it appears, increasing by ~12 GB a day (year-old information). It takes literal months to bootstrap this, according to the XRPL website.

I'm no expert, but it seems to me like this might not be a sustainable method of doing things. XRP's idea is that with more businesses using XRP, there are incentives for them to run nodes themselves. Given that Nano uses the same incentive and many Bitcoin enthusiasts run full nodes, I don't blame them for that thinking and believe it could work.

However, there's one important difference here. A Bitcoin full node is ~360GB, and grows slowly. A Nano full node is ~81 GB, and grows slowly. In both cases though, ledger size is limited because both Bitcoin and Nano do not try to do it all. Bitcoin limits itself to ~7 TPS max, Nano limits itself to pure transactions only. XRP on the other hand seems to be trying to do it all - not just storing simple transactions. I can't find accurate figures for total transactions since start unfortunately, would love to be helped here.

Anyway, this is an interesting event to follow. Because the "primary" nodes are down, some services are switching to other nodes, increasing the API calls on those nodes, putting those nodes under pressure again, etc etc. It does not seem fixed as of yet that I can tell, but information is rather murky.

Lesson in all this, to me, is that it's vital to, if you want to be truly decentralized, keep running a node as lightweight as possible. People and services are naturally lazy, and the fact that many exchanges and XRP's most well known wallet (Xumm) seem to rely on public nodes rather than running their own seems to illustrate that running a node should be made easier and/or cheaper, and that having massive hardware requirements + massive storage needed might not be the way to go. This obviously holds for any cryptocurrency trying to "do it all", and is why I think we'll see many chains all specialised in doing one thing and doing it well.

Any corrections much welcome, I'm following this with interest but am not the most well-read up on XRP lately.