Donations to the archive would be appreciated to help fund our server hardware & storage drives. We are looking for developers to help build new software and archives, discuss here.

New Archive Software Stack

## Developer No.4785 View ViewReplyLast 50ReportDelete
The Bibliotheca Anonoma has built two drop in replacements in Python to replace the aging FoolFuuka/Asagi archival stack. https://wiki.bibanon.org/FoolFuuka
Our goal is to increase scraper efficiency, increase performance of the search engine, and significantly reduce the training time and setup necessary to operate a 4chan archive.
This new server setup allows us to generate backups without affecting scraping, public backups will be uploaded over the coming months.
If you'd like to help develop or plan to make a 4chan archive of your own, it is a tough road but we have the expertise to help you set things up. Post below or contact us at:
irc.rizon.net #bibanon
https://matrix.to/#/#bibanon-chat:matrix.org
https://discord.com/invite/3jxxGDC

* https://github.com/bibanon/neofuuka-scraper is in use at Desuarchive as part of this server migration. See the improved features over Asagi scraper in the repository.
* https://github.com/bibanon/ayase is a drop in replacement for the FoolFuuka frontend, check it out at our demonstration site: https://ayase.wakarimasen.moe. It is sufficient for non-ghostposting use, but we are still working on search and moderation tools and development help is appreciated.

Lastly, We hope to learn Rust to totally redesign the Asagi SQL schema. (currently dependent on Percona MariaDB TokuDB and has numerous conversion issues) Two efforts are underway:
* https://github.com/miyachan/torako is a drop in replacement for the Asagi scraper. It is in testing at https://ayase.wakarimasen.moe .
* https://github.com/reasv/mitsuba is a full stack archive stack replacement and PostgreSQL schema redesign, most likely the best path forward past the aging Asagi schema.
77 posts and 3 images omitted

## Admin No.3026 View ViewReplyReportDelete
Welcome to /desu/. Use this board to report issues, request features, and for other discussions regarding desuarchive.org. Other posts will be removed.

When reporting a technical issue, be sure to include the full URL of the page/image.

Do not use this board for removal requests, which must be emailed to [email protected] Other rule violations can be reported by clicking the "Report" button on the post.

I can't search some ascii symbols

No.6120 View ViewReplyReportDelete
Symbols like < and / can't be searched on the archive. However, It doesn't have a problem with unicode. Am I doing something wrong?

It's bizarre that the archive has no problem searching characters that we generally regard as more complex such as https://desuarchive.org/_/search/text/ア/.

This is a general question about the FoolFuuka software btw. I asked this same question on https://archive.wakarimasen.moe/meta/thread/296/, where I wanted to find most instances of the "/tea/" general on the cooking board, but I didn't get an answer.

No.6053 View ViewReplyReportDelete
Is there anywhere I can find full /a/ images from 2014-2016? Your archive only seems to only go back to 2017.
Also Nyafuu's /c/ images go further back than yours do.
1 post omitted

No.6098 View ViewReplyReportDelete
So is the test server located in Africa or is it just really fucking shitty? Images are loading from it at a snail's pace, and every threads' images are trying to load from it when they still load from s1 just fine.

No.6090 View ViewReplyReportDelete
images, gifs, webms are consistently giving 404s, from s1, s2 and test urls. not sure why, putting this here incase admins werent aware.

https://test.desu-usergeneratedcontent.xyz/trash/image/1487/23/1487230274243.jpg

example of broken url

No.6085 View ViewReplyReportDelete
For example if I had the board and 16351092417704.jpg how would I turn that into a search result?

## Developer No.4548 View ViewReplyLast 50ReportDelete
As part of the server migration announced here: https://wiki.bibanon.org/News/2021-04-11_Desuarchive_Migration

The Bibliotheca Anonoma has built two drop in replacements in Python to replace the aging FoolFuuka/Asagi archival stack. https://wiki.bibanon.org/FoolFuuka

* https://github.com/bibanon/neofuuka-scraper is in use at Desuarchive as part of this server migration. See the improved features over Asagi scraper in the repository.
* https://github.com/bibanon/ayase is a drop in replacement for the FoolFuuka frontend, check it out at our demonstration site: https://ayase.wakarimasen.moe. It is sufficient for non-ghostposting use, but we are still working on search and moderation tools and development help is appreciated.

Lastly, We hope to learn Rust to totally redesign the Asagi SQL schema. Two efforts are underway:
* https://github.com/miyachan/torako is a drop in replacement for the Asagi scraper. It is in testing at https://ayase.wakarimasen.moe .
* https://github.com/reasv/mitsuba is a full stack archive stack replacement and PostgreSQL schema redesign, most likely the best path forward past the aging Asagi schema.

If you'd like to help develop or plan to make the next Yuki.la, it is a tough road but we have the expertise to help set things up. Post below or contact us at:
irc.rizon.net #bibanon
https://matrix.to/#/#bibanon-chat:matrix.org
https://discord.com/invite/3jxxGDC
194 posts and 18 images omitted

Precarity of archiving

No.6054 View ViewReplyReportDelete
Im curious to how difficult it is to manage a 4chan archive. I'm not interested in archiving myself, I'm just a 4chan poster, I am intrigued by the infrastructure needed to keep it running though

For example, how many posts per day do you think would be necessary to make archiving difficult? Do you think there will come a point where you can no longer sustain archiving due to the sheer number of posts archived or is it completely managable? Do you suspect that archiving images will become problematic?

It seems like it should be very hard to archive tens of millions of posts but then again Youtube hosts something stupid like thousands of hours of videos every hour with files easily hitting gigabytes a lot of the time. So maybe its just my ill understanding of archiving and the small size of posts that makes me misinformed.

TLDR Is archiving under threat due to the current stream of 4chan posts or is it alright for the long future? Does archiving become cheaper as data storage becomes cheaper or does the ever increasing posts make it around the same? Will desuarchive ever get too expensive to maintain? will it become more expensive to maintain? Thanks

No.5923 View ViewReplyReportDelete
Does desuarchive have any plans to import older posts like 4plebs does?

Deleted Search Failure

No.6038 View ViewReplyReportDelete
https://desuarchive.org/a/search/tnum/229371671/text/yuristacy/deleted/not-deleted/

This post only shows up in not-deleted and not in deleted search, but it is a deleted post.

https://desuarchive.org/a/search/tnum/229371671/deleted/deleted/

There are over 100 deleted posts in the thread but searching for deleted posts only turns up 30.

Please investigate and fix thanks.