The Future

The Future

Last updated on Oct 18 2015.

The list of defunct 4chan archivers has gained a new addition, archive.moe. Its death reminded us of how dependent we are on archives and has unsettled many about the state of popular archives. This unsettlement has led discussion about preventing this from happening again.

This is what we need to do:

  • 1. Recover

  • 2. Distribute

  • 3. Collaborate

  • 4. Prevent

1. Recover

We need to start off by picking up the pieces. Archive.moe has published their database to archive.moe/dump/ but unfortunately we are missing a hefty four months of data. Media from archive.moe has not been uploaded yet and we have not been able to check the status of it.

Thanks to the help of very generous anons, we are in the process of having /a/, /mlp/, and /qa/ completely recovered (text-wise -- images might be another story). The four months for the other boards are either lost or stored on another archive, waiting to be collected. If you know someone who has the data for a missing board, let us know. If it is for one of the boards Desuarchive provides an archive of, please get in touch with us (info is at the bottom of this page). If it is hosted on another archive, see if the owner would like to import that into their archive.

The recovery process will take time. Our infrastructure can only work so fast, and with the size of archive.moe's database, the process can easily take a few weeks.

2. Distribute

We need to spread load around. Smaller archives are more maintainable than larger ones and create a smaller point of failure. As evidenced by archive.moe, the monstrous archive required big servers, huge bills, and left a huge hole after its shutdown. We need to lessen the wound a dying archive has on users by creating more, smaller, archives.

It is for this reason that I will help anyone set up their own archive (granted you have the hardware to handle it). If you are interested, please contact me (info is below).

Private Archives can help!

If you want to run your own archive in your house just for you, go for it. You can end up saving a board in the future if an archive goes down. If it weren't for small archives, we wouldn't even have the chance of having /a/, /mlp/, and /qa/ complete. If you are interested in this, again, just let me know.

3. Collaborate

Finally, after archives have recovered and been distributed, we need to collaborate. What this means today and what this means in a few months may change, but one thing I can think of off the top of my head is searching across multiple archives.

In large archives, one search could show you results across majority of the boards all in one place. Since distributed archives will not be able to do this, we should create one hub where users can search across all boards on all archives. This is why I am starting the DesuSearch project, which will be an opt in by the owners of archives. What this will do is provide users (and userscripts) a single place to search and find threads across multiple boards and archives. For userscripts it will also return a list of archives which contain a certain thread (incase one archive is missing a thread due to a technical problem or deletion). If you are an archive owner or a developer and you are interested in joining this project, please contact me with the info below.

4. Prevent

We need to make sure this won't happen again. Period.

Now is the time to setup proper backups and assist other archives into doing the same. As for desuarchive, we will be making our weekly database backup available for anybody to download. As for media, we are working to find a backup solution to suit our needs. Desuarchive will also be running an offsite alternate archiver, which will continue to grab posts and media if the main server should ever experience a problem. In order to cut costs, this alternate archiver will not be publicly accessible and will not store media permanently. Instead it will check to see if the main site has that media file before deleting it. The database on this alternate archive will not be purged.

Contact:

IRC: #desustorage on irc.rizon.net

Email: [email protected]

Twitter: @Desuarchive

Discussion Board: >>>/desu/