The Bibliotheca Anonoma has built two drop in replacements in Python to replace the aging FoolFuuka/Asagi archival stack. https://wiki.bibanon.org/FoolFuuka
Our goal is to increase scraper efficiency, increase performance of the search engine, and significantly reduce the training time and setup necessary to operate a 4chan archive.
This new server setup allows us to generate backups without affecting scraping, public backups will be uploaded over the coming months.
If you'd like to help develop or plan to make a 4chan archive of your own, it is a tough road but we have the expertise to help you set things up. Post below or contact us at:
irc.rizon.net #bibanon
https://matrix.to/#/#bibanon-chat:matrix.org
https://discord.com/invite/3jxxGDC

* https://github.com/bibanon/neofuuka-scraper is in use at Desuarchive as part of this server migration. See the improved features over Asagi scraper in the repository.
* https://github.com/bibanon/ayase is a drop in replacement for the FoolFuuka frontend, check it out at our demonstration site: https://ayase.wakarimasen.moe. It is sufficient for non-ghostposting use, but we are still working on search and moderation tools and development help is appreciated.

Lastly, We hope to learn Rust to totally redesign the Asagi SQL schema. (currently dependent on Percona MariaDB TokuDB and has numerous conversion issues) Two efforts are underway:
* https://github.com/miyachan/torako is a drop in replacement for the Asagi scraper. It is in testing at https://ayase.wakarimasen.moe .
* https://github.com/reasv/mitsuba is a full stack archive stack replacement and PostgreSQL schema redesign, most likely the best path forward past the aging Asagi schema.

So is the test server located in Africa or is it just really fucking shitty? Images are loading from it at a snail's pace, and every threads' images are trying to load from it when they still load from s1 just fine.

For example if I had the board and 16351092417704.jpg how would I turn that into a search result?

As part of the server migration announced here: https://wiki.bibanon.org/News/2021-04-11_Desuarchive_Migration

The Bibliotheca Anonoma has built two drop in replacements in Python to replace the aging FoolFuuka/Asagi archival stack. https://wiki.bibanon.org/FoolFuuka

* https://github.com/bibanon/neofuuka-scraper is in use at Desuarchive as part of this server migration. See the improved features over Asagi scraper in the repository.
* https://github.com/bibanon/ayase is a drop in replacement for the FoolFuuka frontend, check it out at our demonstration site: https://ayase.wakarimasen.moe. It is sufficient for non-ghostposting use, but we are still working on search and moderation tools and development help is appreciated.

Lastly, We hope to learn Rust to totally redesign the Asagi SQL schema. Two efforts are underway:
* https://github.com/miyachan/torako is a drop in replacement for the Asagi scraper. It is in testing at https://ayase.wakarimasen.moe .
* https://github.com/reasv/mitsuba is a full stack archive stack replacement and PostgreSQL schema redesign, most likely the best path forward past the aging Asagi schema.

If you'd like to help develop or plan to make the next Yuki.la, it is a tough road but we have the expertise to help set things up. Post below or contact us at:
irc.rizon.net #bibanon
https://matrix.to/#/#bibanon-chat:matrix.org
https://discord.com/invite/3jxxGDC

Im curious to how difficult it is to manage a 4chan archive. I'm not interested in archiving myself, I'm just a 4chan poster, I am intrigued by the infrastructure needed to keep it running though

For example, how many posts per day do you think would be necessary to make archiving difficult? Do you think there will come a point where you can no longer sustain archiving due to the sheer number of posts archived or is it completely managable? Do you suspect that archiving images will become problematic?

It seems like it should be very hard to archive tens of millions of posts but then again Youtube hosts something stupid like thousands of hours of videos every hour with files easily hitting gigabytes a lot of the time. So maybe its just my ill understanding of archiving and the small size of posts that makes me misinformed.

TLDR Is archiving under threat due to the current stream of 4chan posts or is it alright for the long future? Does archiving become cheaper as data storage becomes cheaper or does the ever increasing posts make it around the same? Will desuarchive ever get too expensive to maintain? will it become more expensive to maintain? Thanks

why are all the webms broken?

Have you seen this thread?
https://boards.4chan.org/t/thread/1033260

Does desuarchive have any plans to import older posts like 4plebs does?

https://desuarchive.org/a/search/tnum/229371671/text/yuristacy/deleted/not-deleted/

This post only shows up in not-deleted and not in deleted search, but it is a deleted post.

https://desuarchive.org/a/search/tnum/229371671/deleted/deleted/

There are over 100 deleted posts in the thread but searching for deleted posts only turns up 30.

Please investigate and fix thanks.

where the fuck did /fa/ go?