The Next Generation of PDS MOOver is here! While PDS MOOver started out as a very simple Index.html page to help users migrate to a new PDS it has evolved a bit beyond that. We still handle all your atproto events like migrating, finding missing blobs, etc client side and our server never sees your passwords. But, I had to build a whole backend XRPC service(like an AppView) to handle the biggest and newest feature. PDS MOOver can now handle backs up for your ATProto account and recovery in the event your PDS goes down, or adversarial an event. This is by far the biggest new feature, but it's not the only one!
Straight to the point, what's new?
The PDS MOOver kingdom has been united and all the one off tools are under one roof with a navbar for easy finding
You can now create rotation keys during migration and manually enter any you may already have
Backups! These happen every 24 hours on sign up. And are completely free to you as a user. Can sign in as well to see stats about your backups and if you have any missing blobs
On the backups sign up can also register a rotation key (which is needed for recovery/restore)
PDS MOOver also supports PDS Level backups. This will check the PDS everyday for new accounts, and back them up. There will be a future update as well to allow PDS Admin to restore accounts. If you're interested in this please let me know. All I ask for is a small monthly sponsorship to help cover server costs. Most cases this is $2, but please let me know how big your blob directory/S3 is for pricing.
Restore/recovery, no longer need to fear if your PDS goes down or is beyond recovery. You can now use PDS MOOver to restore your account from our backup service and migrate to a new PDS with your rotation key
We now speak XRPC! You can view all supported lexicons. This includes getRepo and getBlob so you can use other popular atproto tools to access your backups
Don't trust me? Self host your own PDS MOOver. Along with the source code being open source I have also provided a single docker compose file to allow you to run your own PDS MOOver stack to handle your backups.
With these changes I hope that it provides a somewhat user friendly ways to keep their atproto identity protected, and peace of mind knowing they have daily backups to restore their social life if it's ever needed.
Alright, what's the details?
This will be a somewhat deeper dive of the new features along with talking a bit about the tech behind each of them
Migration changes
There's a couple of new things to migrations to help keep an account safe. The first big one is you can now select to create a rotation key on signup, along with signing up for backups. The rotation key is an important part to account recovery. Without you having your own if your PDS ever disappears or crashes you cannot recover your account. This is a very important thing to have if you ever move to another PDS, and to understand.
When you download this key file it will contain three things. Your did that you will enter if your PDS is down during a restore, your PublicKey if you ever want to verify it's registered with the PLC for your account, and the most important part, the private key. It is important to keep this key and file in a secure location if you ever need it. In the wrong hands this can also let someone take over your account, even if they do not know your password.
The second thing is the ability to add up to 4 rotation keys during the migration. 5 is the limit and you want your PDS to have one. PDS MOOver still does not merge these together to help keep it a bit simple, but the hope is this will cover most use cases and there may even be a more advance PLC tool in the future.
Backups
Backups are for sure the big headliner of this update and took the most time. Backups currently only cover your blobs(pictures/videos) and your repo(posts, following, likes, etc), but I am exploring the possibility of adding your preferences(holds things like your saved feeds), but since this is private data and requires authentication I need to think on it a bit more. Your backups are currently taken every 24 hours automatically. We are hoping as we onboard more users and see how the service works to move that down to shorter intervals. When we pull the backup we only pull the repo if it changes, and only pull new blobs.
Your backups are compressed via zstd and stored on a S3 bucket on UpCloud's Object Storage compressed, then when you restore or need something you can call the PDS MOOver's new XRPC endpoints com.atproto.sync.getRepo and com.atproto.sync.getBlob just like you would on a PDS, it decompresses it and streams the response much like the PDS.
This compression does not actually give us a lot of space savings on blobs, but surprisingly it gave us over 50% compression on CAR exports. You can see in the picture below of my .car export at 22 MiB uncompressed and with zstd compression it is sitting at 8.88 MiB
I made sure to build out everything where it can scale as well. The web front end is behind a load balancer allowing us to scale it out to multiple instances, all the actual backup logic is handled by workers using the Rust crate apalis allowing us to scale to multiple worker nodes during busy hours or on boarding a larger PDS. The hope is with this any surge in traffic or backups can be easily handled with adding a couple of servers here and there to future proof the project for some time to come. Can check out the network diagram for a better idea on how it all works togather.
Special thanks to Orual for the network drawing
You can also login after sign up and see some stats about your backups as well as requesting a manual backup, or to delete them completely and remove yourself from it. There's also a helpful count to show any missing blobs on your repo from a migration that may have missed them.
Restore/Recovery
Not much of a point to a backup if you can't use it is there? One major goal of this release was to make backups and account recovery something anyone can do from a web interface. It's not the most polished user experience, nor does it set out to be. But it brings this ability to everyday users allowing them to protect and backup their digital social identity. With PDS MOOver you can now recover your identity and restore a backup in the event your PDS goes down, or if a malcious actor tries and takes control of your identity on the PLC. To my knowledge this is currently the only web UI that does this, and it is currently tightly coupled with a restore from PDS MOOver, but I am planning on bringing BYOB(bring your own backups) to allow you to recover from any backup you may have. It is important to know you got to have a Rotation Key that you control. This can be setup during migration, or sign up for PDS MOOver backups. Your PDS Admin can also use web ui to restore your account on a new PDS as well with their rotation key in the event it is needed. The recovery process is based off of the Adversarial ATProto PDS Migration blog post.
This feature has been tested to the best of my ability and can see the first successful recovery with this account.
This was the A-Z test btw, and it was a success
I lived
We speak lexicon
PDS MOOver's backend is implemented as an XRPC service. This means you can do any action the front does, but via an XRPC request with your favorite atproto library. You can view all the current and future endpoints in our tangled repo.
This is also how you get your backups when restoring an account. We support both com.atproto.sync.getBlob and com.atproto.sync.getRepo with the hopes to make it easier for other migration tools to interface with a PDS MOOver instance
Can click this link to view a blob that was backed up with PDS MOOver.
Future Items
Rest, I'm probably not going to touch a atproto code base for at least a week (pending major bugs)
Clean up and refactor some of the code. I'm coming off of a week long crunch to get this out, So I want to look at the codebase with fresh eyes
Condense the JS logic into an external npm package so that any one can use it to migrate, create rotation keys, restore, or sign up for backups easily in their projects. What I consider a big cornerstone of PDS MOOver and sady broke that a bit with this release for speed of development.
Better documentation on running your own instance of PDS MOOver
Support backing up of preferences
PDS Host restores. This will let you run a restore of all the repos of a PDS in the event of a PDS goes down and you are the PDS Admin
PLC editor to change your rotation keys, services, etc.
BYOB (bring your own backup) to restores
PLC watcher to notify if any PLC operations happen against your account
Real time backups via subscribing to the fire hoses from a PDS. You create a new record or add a new blob, we see it and add it to the backup as soon as it happens.