/g/ - Technology » Thread #85357870

[327 / 53 / 106]

306KiB, 1000x1333, 2c2e2403cce595b9a26951e09fdbbfb7.jpg

/dhg/ - Data Hoarding General

Anonymous Sun 23 Jan 2022 20:32:52 No.85357870 View View Reply Original Report

Quoted By:

>Links
Rentry: https://rentry.org/dhg

>What is /dhg/
In this thread we discuss and create technology and software for data-hoarding, archiving, scripts, and more.

>gallery-dl - scrape images, manga, videos and more from many websites
https://github.com/mikf/gallery-dl

>Hydrus Network
https://hydrusnetwork.github.io/hydrus/

>Stash
https://github.com/stashapp/stash

>SmartImage
https://github.com/Decimation/SmartImage

Previous Thread: >>85259453

Anonymous

Anonymous Wed 26 Jan 2022 19:39:53 No.85400949 Report

Quoted By:

>>85400896
Golden boy is a classic.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1629624503754.jpg, 466KiB, 2048x2048

Anonymous Wed 26 Jan 2022 19:42:13 No.85400980 Report

Quoted By: >>85401924 >>85401952

Before I try and make one, are there any endchan downloaders? Gallery-dl doesn't support it

Anonymous

Anonymous Wed 26 Jan 2022 20:22:35 No.85401671 Report

Quoted By: >>85401749

>>85400185
>feature will take whatever image set it has been given and then a scriptable filter can be applied to it interactively?
Yes - to be sure, "whatever image set it has been given" means the images that sxiv was originally opened with. So if you call sxiv like so: sxiv -t a.png b.png c.png then a.png, b.png, c.png will be passed to the "filter" script.
>all my images tagged with "image"
>not sure I want to feed it such a huge set initially
What do you mean? That your images are all tagged with "image" shouldn't matter. When you call the filter script interactively, you can filter any tags you want.

Anonymous

Anonymous Wed 26 Jan 2022 20:27:23 No.85401749 Report

Quoted By:

>>85401671
>What do you mean?
I thought the initial list of files would be too long, and that it might negatively impact sxiv's performance, but it worked out ok.

Anonymous

Anonymous Wed 26 Jan 2022 20:39:23 No.85401924 Report

Quoted By:

>>85400980
gallery-dl doesn't have endchan support, but
it does have 8_k_u_n support.
With some very small modifications,
it can be made to support endchan too.
4chan is making it really hard to type a proper response to you due to word filters, but
go here:
https://github.com/mikf/gallery-dl/tree/master/gallery_dl/extractor
Look at the appropriate driver and notice how little you'd have to change to get the driver to work for endchan.

Anonymous

Anonymous Wed 26 Jan 2022 20:41:27 No.85401952 Report

Quoted By:

>>85400980
If you want to go the home grown route and not make another drivery for gallery-dl, be sure to use the JSON endpoints that endchan gives you.
https://endchan.net/cumg/res/69.json
https://endchan.net/cumg/res/69.html

Anonymous

Anonymous Wed 26 Jan 2022 21:02:32 No.85402254 Report

Quoted By: >>85402429

>>85390205
Use a custom Obsidian setup like me.

Anonymous

Anonymous Wed 26 Jan 2022 21:10:33 No.85402356 Report

Quoted By:

>>85390205
I'm trying to make tmsu work for me.
I'm finding and building little tools around it.

Anonymous

Anonymous Wed 26 Jan 2022 21:16:34 No.85402429 Report

Quoted By: >>85402517

>>85402254
What's Obsidian?

Anonymous

Anonymous Wed 26 Jan 2022 21:23:08 No.85402517 Report

Quoted By: >>85402592 >>85402605 >>85402779 >>85404390 >>85404436

>>85402429

It's a knowledge management system, but it can be heavily customized with relative ease. You create notes and link them together in various ways. The result is not unlike a personal Wiki, but more streamlined.

I use it to organize movies from Japan. Each actress has a note. Each video has a note. Etc. It's all interlinked, and links to the actual files are imbedded within. Here's a video of what I've been able to achieve (I've since improved a number of things). Beware, it's not safe for work.

https://files.catbox.moe/xzewmr.mp4

Anonymous

Anonymous Wed 26 Jan 2022 21:28:15 No.85402592 Report

Quoted By: >>85402661 >>85404436

>>85402517
I thought org-roam-ui was cool, but Obsidian just mogged tf out of it.
Pain.

Anonymous

Anonymous Wed 26 Jan 2022 21:29:26 No.85402605 Report

Quoted By: >>85402661

>>85402517
So it was that note taking thing I found. Holy shit anon that is very impressive.

Anonymous

View Same Google ImgOps iqdb SauceNAO home.jpg, 190KiB, 1920x1156

Anonymous Wed 26 Jan 2022 21:33:41 No.85402661 Report

Quoted By: >>85402746

>>85402592
>>85402605
>>85402592
It's fun to tinker with. I've been improving my personal home page lately. I added some of these link areas (not too sure what to call them).

Anonymous

View Same Google ImgOps iqdb SauceNAO home2.jpg, 146KiB, 1920x1162

Anonymous Wed 26 Jan 2022 21:40:04 No.85402746 Report

Quoted By:

>>85402661
Further down I have a few bands of relevant content (new releases, new additions, etc.) I was inspired by Plex here. (movie posters removed to keep things sfw) The work is never done though. I keep tweaking things.

Anonymous

Anonymous Wed 26 Jan 2022 21:41:57 No.85402779 Report

Quoted By: >>85402904

>>85402517
Interesting. I'll look into this later for myself. How well would this handle galleries, gravure sets, or mangas?

Anonymous

Anonymous Wed 26 Jan 2022 21:50:57 No.85402904 Report

Quoted By:

>>85402779
Image support isn't great out of the box, but a number of the better themes take care of that. There are also plugins and css snippets that can help.

It wouldn't be very difficult to create a page with a thumbnail gallery and some way of displaying the full-sized image. Others have done this very thing, and their work can be replicated.

It gets time-consuming once you have lots of them, but templates can automate a lot.

Anonymous

View Same Google ImgOps iqdb SauceNAO file.png, 105KiB, 1920x1080

Anonymous Wed 26 Jan 2022 23:35:32 No.85404034 Report

Quoted By: >>85404075 >>85404356

>tfw a bunch of unknown, non-public chink codecs BTFO the open-sores implementations of the west
Look at Trencent VAV1. It has 28% better compression and is 25% faster at the same time than libaom (yes, both encode to the same format.)
Same for H.265 (x265 vs TrencentV265). Trencent has literally the best implementations for all formats (AV1 < H.265 < H.266). And you cannot get them, for any price.
I'm not even memeing. x264 was best in class for H.264. The new codecs have dogshit free implementations, all of them. What the fuck is going on?

Anonymous

Anonymous Wed 26 Jan 2022 23:35:49 No.85404037 Report

Quoted By:

bump

Anonymous

Anonymous Wed 26 Jan 2022 23:38:37 No.85404075 Report

Quoted By:

>>85404034
For reference:
https://compression.ru/compression.ru/video/
https://compression.ru/video/codec_comparison/2021/main_report.html

Anonymous

Anonymous Thu 27 Jan 2022 00:07:18 No.85404356 Report

Quoted By:

>>85404034
It's over

Anonymous

Anonymous Thu 27 Jan 2022 00:10:30 No.85404390 Report

Quoted By: >>85404799 >>85405116

>>85402517
Holy... how much time did you invest? And what Meta Data scrapper did you use?

Anonymous

Anonymous Thu 27 Jan 2022 00:15:46 No.85404436 Report

Quoted By:

>>85402517
>>85402592
One thing org-roam-ui is missing from this is a grid view of notes.

Anonymous

Anonymous Thu 27 Jan 2022 00:48:11 No.85404799 Report

Quoted By: >>85404878

>>85404390
A hundred hours maybe? I'm not sure.
Embarrassingly, I haven't figured out scraping yet, so most things have been entered by hand. It's not so bad though. I can enter a new movie in less than 5 minutes now. More like 2 or 3 if high quality cover art is easy to find. I just plop down a template with a few keyboard shortcuts and copy and paste the info from a few sites.

Anonymous

Anonymous Thu 27 Jan 2022 00:57:14 No.85404878 Report

Quoted By: >>85405035

>>85404799
Mind sharing some of your templates then?

Anonymous

View Same Google ImgOps iqdb SauceNAO temp.jpg, 332KiB, 1964x2040

Anonymous Thu 27 Jan 2022 01:15:52 No.85405035 Report

Quoted By: >>85405119

>>85404878
They're mostly straightforward.
Each video note uses the video code for its title. (example: HMN-022). Covers and the videos themselves are named the same way allowing me to make certain assumptions.

Here are the jav, series, and actress templates to give you a sense of how they work.

Jesus

Jesus Thu 27 Jan 2022 01:19:21 No.85405065 Report

Quoted By: >>85405077

what SMART Attributes should i keep an eye on?

Anonymous

Anonymous Thu 27 Jan 2022 01:21:10 No.85405077 Report

Quoted By:

>>85405065
pending sector count
uncorrectable sector count
G-sense (this tells you if hdd is getting affected from outside enviroment like vibrations....etc)

Anonymous

View Same Google ImgOps iqdb SauceNAO actresses.jpg, 544KiB, 2560x1400

Anonymous Thu 27 Jan 2022 01:24:46 No.85405116 Report

Quoted By: >>85405381 >>85405575 >>85406098 >>85406139 >>85406228

>>85404390
>Holy... how much time did you invest?
Filling out actress info is the next big project.

Anonymous

Anonymous Thu 27 Jan 2022 01:24:57 No.85405119 Report

Quoted By: >>85405162

>>85405035
I was just having a look through Templater's documentations. You using any other plugins?

Anonymous

View Same Google ImgOps iqdb SauceNAO addons.jpg, 277KiB, 2232x1178

Anonymous Thu 27 Jan 2022 01:28:53 No.85405162 Report

Quoted By: >>85405387

>>85405119
I wasn't using that many at first. It kind of grew over time. Some are more important than others.

Anonymous

Anonymous Thu 27 Jan 2022 01:55:48 No.85405381 Report

Quoted By: >>85405417

>>85405116
What is the み link go to?

Anonymous

Anonymous Thu 27 Jan 2022 01:56:50 No.85405387 Report

Quoted By:

>>85405162
I had no idea Obsidian had an add-on ecosystem.

Anonymous

Anonymous Thu 27 Jan 2022 02:00:00 No.85405417 Report

Quoted By: >>85405575

>>85405381
Actress profile on a Japanese website.
Example: http://www.minnano-av.com/actress604746.html

It's where I get their measurements and Twitter links, among other things. I've also been defaulting to that website's name spellings to avoid accidental duplicates.

Anonymous

Anonymous Thu 27 Jan 2022 02:13:41 No.85405575 Report

Quoted By:

>>85405116
>Filling out actress info is the next big project.
This should be automated, at least partially.

>>85405417
>Actress profile on a Japanese website.
>Example: http://www.minnano-av.com/actress604746.html
What a qt.

Anonymous

Anonymous Thu 27 Jan 2022 03:14:53 No.85406098 Report

Quoted By: >>85406107 >>85406139 >>85406221 >>85406228

>>85405116
>Filling out actress info is the next big project.

Let me help you out.

// ==UserScript==
// @name Actress Metadata - minnano-av.com
// @namespace Violentmonkey Scripts
// @match http://www.minnano-av.com/actress*.html
// @grant none
// @version 1.0
// @author -
// @description 1/26/2022, 6:22:28 PM
// ==/UserScript==

const div = document.querySelector('div.act-profile')
const trs = div.querySelectorAll('tr')
const r0 = trs[0].innerText.match(/^\s*(.*?)\s+\uff08(.*?)\s+\/\s+(.*)\uff09/)
const name = {
 kanji: r0[1],
 hiragana: r0[2],
 romaji: r0[3]
}

const r2 = trs[2].innerText.match(/(\d+)\u5e74(\d+)\u6708(\d+)\u65e5/)
const birthdate = `${r2[1]}-${r2[2]}-${r2[3]}`

const r3 = trs[3].innerText.match(/T(\d+).*?B(\d+).*?(\w)\u30ab.*?W(\d+).*?H(\d+)/)
const height = r3[1]
const bust = r3[2]
const cup = r3[3]
const waist = r3[4]
const hip = r3[5]

const r8 = trs[8].innerText.match(/(\d+)/g)
const career = {
 begin: r8[0],
 end: r8[1] || ''
}

const r10 = trs[10].innerText.match(/(http.*)/)
const links = {
 twitter: r10.length ? r10[1] : '#',
 mi: document.location.href
}

const a = { name, birthdate, height, bust, cup, waist, hip, career }
unsafeWindow.a = a
unsafeWindow.aa = `NameJP: ${name.kanji}

Alias: -

Birthdate: ${birthdate}

Height: ${height}

Bust: ${bust} (${cup} cup)

Waist: ${waist}

Hip: ${hip}

Career: ${career.begin} - ${career.end}

Link: [み](${links.mi}), [TWTR](${links.twitter})
`

When you use this userscript, it'll put two variables in your browser's JS console.

a - an object representing the actress
aa - a string that should drop right in to your actress template

Anonymous

Anonymous Thu 27 Jan 2022 03:16:01 No.85406107 Report

Quoted By: >>85406221

>>85406098
a

{
 "name": {
 "kanji": "初愛ねんね",
 "hiragana": "ういねんね",
 "romaji": "Ui Nenne"
 },
 "birthdate": "2001-10-01",
 "height": "155",
 "bust": "99",
 "cup": "H",
 "waist": "60",
 "hip": "90",
 "career": {
 "begin": "2020",
 "end": ""
 }
}

NameJP: 初愛ねんね

Alias: -

Birthdate: 2001-10-01

Height: 155

Bust: 99 (H cup)

Waist: 60

Hip: 90

Career: 2020 - 

Link: [み](http://www.minnano-av.com/actress604746.html), [TWTR](http://twitter.com/ui_nenne)

While you're in the console, I would do this:
copy(aa)
Then you can paste that into your actress notes.

Anonymous

Anonymous Thu 27 Jan 2022 03:21:01 No.85406139 Report

Quoted By: >>85406221

>>85405116
>>85406098
Even if you don't use this userscript, the regexps can be used in whatever you build to eventually automate JAV actress metadata input.

Anonymous

Anonymous Thu 27 Jan 2022 03:31:45 No.85406221 Report

Quoted By: >>85406958

>>85406098
>>85406107
>>85406139
Holy Moly. Thanks!

Anonymous

Anonymous Thu 27 Jan 2022 03:32:18 No.85406228 Report

Quoted By: >>85406270

>>85405116
>>85406098
I ran into a little problem.
Actresses that have had many name changes break some assumptions I made.
Example:
http://www.minnano-av.com/actress689323.html
The userscript needs some debugging.
Sorry about that.

Anonymous

Anonymous Thu 27 Jan 2022 03:36:44 No.85406270 Report

Quoted By: >>85406563

>>85406228
No problem. You've given me something to start with.

Anonymous

Anonymous Thu 27 Jan 2022 04:11:52 No.85406563 Report

Quoted By: >>85406615 >>85406694

>>85406270
This version of the minnano-av actress metadata scraper should be more resilient.
Fewer assumptions are made about where in the table the data will be.
I had to use pastebin this time, because the code is just a bit too long.

https://pastebin.com/pWk7AnvR

Anonymous

View Same Google ImgOps iqdb SauceNAO 2022-01-26_20-17.mizuno-asahi.png, 296KiB, 951x1262

Anonymous Thu 27 Jan 2022 04:17:49 No.85406615 Report

Quoted By: >>85406653

>>85406563
http://www.minnano-av.com/actress689323.html

a

{
 "name": {
 "kanji": "水野朝陽",
 "hiragana": "みずのあさひ",
 "romaji": "Mizuno Asahi"
 },
 "birthdate": "1990-11-12",
 "height": "165",
 "bust": "90",
 "cup": "G",
 "waist": "60",
 "hip": "100",
 "career": [
 "2013",
 "2019",
 "2020"
 ],
 "careerStr": "2013-2019, 2020-"
}

NameJP: 水野朝陽

Alias: -

Birthdate: 1990-11-12

Height: 165

Bust: 90 (G cup)

Waist: 60

Hip: 100

Career: 2013-2019, 2020-

Link: [み](http://www.minnano-av.com/actress689323.html), [TWTR](http://twitter.com/mizuno_asahi)

Anonymous

Anonymous Thu 27 Jan 2022 04:22:04 No.85406653 Report

Quoted By: >>85406694

>>85406615
It's still a little buggy on the career part.
I'm going to stop here and leave the rest to you though.

Anonymous

Anonymous Thu 27 Jan 2022 04:26:40 No.85406694 Report

Quoted By:

>>85406563
>>85406653
I lied.
I won't stop.
On line 38 where it says "trs[11]", change that to "trs[i:lit]".

Anonymous

Anonymous Thu 27 Jan 2022 04:54:16 No.85406958 Report

Quoted By:

>>85406221
Thanks for sharing your Obsidian setup.
That got me thinking.

Anonymous

Anonymous Thu 27 Jan 2022 06:13:16 No.85407614 Report

Quoted By: >>85407632

facebook scan - is there anyway to automate/script an email/text when someone updates a private facebook profile? I follow some egirl

Anonymous

Anonymous Thu 27 Jan 2022 06:15:30 No.85407632 Report

Quoted By:

>>85407614
Use something like puppeteer or selenium to log in, and write some monitoring code to watch the profile.

Anonymous

Anonymous Thu 27 Jan 2022 08:03:24 No.85408477 Report

Quoted By:

Need new bread soon.

Anonymous

Anonymous Thu 27 Jan 2022 08:03:59 No.85408484 Report

Quoted By:

So what's the best way to download from sankaku? Ideally I'd like to download my favorited vids but I'm not sure if there is a bulk downloader than can do it.

Anonymous

Anonymous Sat 07 Dec 2024 22:26:12 No.85408484,1 Report Delete

Quoted By:

I miss the /cumg/ era.

Subject

Name

E-mail

Password

Capcode	All Only User Posts Only Verified Posts Only Moderator Posts Only Manager Posts Only Admin Posts Only Developer Posts Only Founder Posts
Show Posts	All Only With Images Only Without Images Only Spoiler Images Only Non-Spoiler Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

On these archives

On these boards

Your latest searches

/dhg/ - Data Hoarding General