/g/ - Technology » Thread #95519153

[357 / 61 / 88]

Anonymous Mon 21 Aug 2023 19:42:42 No.95519153 View View Reply Original Report

Quoted By: >>95519185 >>95519871 >>95520065 >>95520104 >>95520285 >>95525491 >>95526035 >>95527474 >>95529857

►Previous Thread >>95504681

►News
>(08/18) llama.cpp training imminent
>(08/10) Anon (who won't bake or update news himself) complains that news don't include Kobold adding 16k context support.
>(08/06) CFG merged into transformers, supported in (plain) exllama and ooba
>(07/24) llama.cpp and koboldcpp now compatible with llama 2
>(07/18) Llama 2 released

►Model Rankings
HF: https://hf.co/spaces/HuggingFaceH4/open_llm_leaderboard
PLAP: https://rentry.org/ayumi_erp_rating

►FAQ
>Main FAQ
https://rentry.org/er2qd

►General LLM Guides & Resources
>Newb Guide
https://rentry.org/local_LLM_guide
>LlaMA Guide
https://rentry.org/TESFT-LLaMa
>Machine Learning Roadmap
https://rentry.org/machine-learning-roadmap
>Novice's LLM Training Guide
https://rentry.org/llm-training
>Local Models Papers
https://rentry.org/LocalModelsPapers
>Quantization Guide
https://rentry.org/easyquantguide
>lmg General Resources
https://rentry.org/lmg-resources
>ROCm AMD Guide
https://rentry.org/eq3hg

►Model DL Links, & Guides
>Model Links & DL
https://rentry.org/lmg_models
>lmg Related Links
https://rentry.org/LocalModelsLinks

►Text Gen. UI
>Text Gen. WebUI
https://github.com/oobabooga/text-generation-webui
>KoboldCPP
https://github.com/LostRuins/koboldcpp
>KoboldAI
https://github.com/0cc4m/KoboldAI
>SimpleLlama
https://github.com/NO-ob/simpleLlama

►ERP/RP/Story Gen.
>ERP/RP Data Collection
https://rentry.org/qib8f
>LLaMA RP Proxy
https://rentry.org/better-llama-roleplay

►Other Resources
>Miku!
https://rentry.org/lmg-resources#all-things-miku
>Benchmark Prompts
https://pastebin.com/LmRhwUCA (embed)
>Simple Proxy for WebUI (+output quality)
https://github.com/anon998/simple-proxy-for-tavern
>Additional Links
https://rentry.org/lmg_template#additional-resource-links

Anonymous

Anonymous Mon 21 Aug 2023 19:43:27 No.95519170 Report

Quoted By: >>95519219

psycho miku tongues my anus

Anonymous

View Same Google ImgOps iqdb SauceNAO 1661842725962358.jpg, 222KiB, 900x600

Anonymous Mon 21 Aug 2023 19:44:09 No.95519180 Report

Quoted By:

bread desu

Anonymous

Anonymous Mon 21 Aug 2023 19:44:26 No.95519185 Report

Quoted By: >>95519196

>>95519153
>in the op image
>not a psychomiku thread
i am suing you

Anonymous

Anonymous Mon 21 Aug 2023 19:44:54 No.95519196 Report

Quoted By:

>>95519185
>no topic either
what a shit OP

Anonymous

Anonymous Mon 21 Aug 2023 19:45:57 No.95519219 Report

Quoted By:

>>95519170
she tongues desu's anus instead

Anonymous

Anonymous Mon 21 Aug 2023 19:47:09 No.95519242 Report

Quoted By: >>95519261 >>95519263

i fucking hate machine learning is this the right general for me

Anonymous

Anonymous Mon 21 Aug 2023 19:48:22 No.95519261 Report

Quoted By:

>>95519242
more or less

Anonymous

Anonymous Mon 21 Aug 2023 19:48:24 No.95519263 Report

Quoted By: >>95519476

>>95519242
Yea, but you gotta have some depraved fetish to RP with LLMs too.

Anonymous

Anonymous Mon 21 Aug 2023 19:58:36 No.95519476 Report

Quoted By: >>95519513

>>95519263
Like me
Wholesome happy kissing!

Anonymous

Anonymous Mon 21 Aug 2023 20:00:20 No.95519509 Report

Quoted By:

>>95518237
How does this compare to the BNoT prompting method?

Anonymous

Anonymous Mon 21 Aug 2023 20:00:23 No.95519513 Report

Quoted By:

>>95519476
I helped my loli kill a stranger and then I had sex with her, but your fetish is cute too anone.

Anonymous

Anonymous Mon 21 Aug 2023 20:01:25 No.95519529 Report

Quoted By: >>95519755

>>95518237
Graph of nigger when?

Anonymous

View Same Google ImgOps iqdb SauceNAO watthours.png, 48KiB, 571x441

Anonymous Mon 21 Aug 2023 20:03:04 No.95519566 Report

Quoted By: >>95519628 >>95519721

Shutting 'er down for the day, before the electric company rapes my ass ever moreso.
Here's what 70B at 6-bits on 3 P40s has been getting me:

llama_print_timings: load time = 11481.31 ms
llama_print_timings: sample time = 8518.94 ms / 250 runs ( 34.08 ms per token, 29.35 tokens per second)
llama_print_timings: prompt eval time = 81925.60 ms / 3624 tokens ( 22.61 ms per token, 44.24 tokens per second)
llama_print_timings: eval time = 54419.77 ms / 249 runs ( 218.55 ms per token, 4.58 tokens per second)
llama_print_timings: total time = 144992.63 ms
{"timestamp":1692647699,"level":"INFO","function":"log_server_request","line":1151,"message":"request","remote_addr":"127.0.0.1","remote_port":52188,"status":200,"method":"POST","path":"/completion","params":{}}

I might drop back to 4_1, but I have noticed that 6 never fucks up with returning NAN # whereas 5 and 4 sometimes will.
If you want things to truly rip, you gotta be on 3090s or 4090s.

Anonymous

Anonymous Mon 21 Aug 2023 20:04:33 No.95519599 Report

Quoted By: >>95519674

Is there any real benefit to using cublas over clblast with a Nvidia GPU? From a couple of tests, my T/s doesn't really seem to change for both prompt ingestion and generation, but clblast is a lot less resource-hungry.

Anonymous

Anonymous Mon 21 Aug 2023 20:06:01 No.95519628 Report

Quoted By:

>>95519566
Whoops l see now my power meter graph shit the bed last month...

Anonymous

Anonymous Mon 21 Aug 2023 20:08:14 No.95519674 Report

Quoted By: >>95519945

>>95519599
When you offload layers using clblast you actually get them out of your ram, unlike when you use cublas.

Anonymous

View Same Google ImgOps iqdb SauceNAO Untitled.png, 24KiB, 866x78

Anonymous Mon 21 Aug 2023 20:08:39 No.95519685 Report

Quoted By: >>95519711 >>95519790 >>95519882 >>95520040

Anonymous

View Same Google ImgOps iqdb SauceNAO cat quick thumbs up.jpg, 35KiB, 600x600

Anonymous Mon 21 Aug 2023 20:09:21 No.95519711 Report

Quoted By:

>>95519685
cool.

Anonymous

Anonymous Mon 21 Aug 2023 20:09:42 No.95519721 Report

Quoted By: >>95520442

>>95519566
That's pretty good. Is this with 4k context filled?

Anonymous

Anonymous Mon 21 Aug 2023 20:11:25 No.95519755 Report

Quoted By:

>>95519529
Tempted to give it a try when I got time, though ToBN was way easier to implement. This time I would have to actually write code and shit

Anonymous

Anonymous Mon 21 Aug 2023 20:12:19 No.95519779 Report

Quoted By: >>95519864 >>95519910

INCLUDE THE SUBJECT NEXT TIME FAGGOT YOU EVADED MY FILTER AND I SAW YOUR GAY SHIT!!!
Thanks and also have a good day

Anonymous

Anonymous Mon 21 Aug 2023 20:12:38 No.95519790 Report

Quoted By:

>>95519685
huggingface shitting their pants right now

Anonymous

Anonymous Mon 21 Aug 2023 20:15:08 No.95519836 Report

Quoted By: >>95519879 >>95519941

anything new to make ayymd cards work?

Anonymous

Anonymous Mon 21 Aug 2023 20:16:16 No.95519864 Report

Quoted By: >>95520862

>>95519779
I said the same to a faggot in chub.ai that kept mistagging his disgusting dyke shit and he ended up deleting the whole thing.
First time I bullied a retard enough to make him delete shit. Felt great.

Anonymous

Anonymous Mon 21 Aug 2023 20:16:28 No.95519871 Report

Quoted By: >>95519893 >>95520501

>>95519153
>be me, have 2 TB of models for stable diffusion, sdxl, llama-2 and finetunes, etc.
>also have an old Dell R710 sever with a bunch of spinny disk SAS drives
>Store a bunch of stuff on an external USB-C nvme drive because it's faster than my NAS
>Normally when I access my NAS it bogs down with big files
Bros what's my best solution for putting these in a NAS in a way that's fast enough to conveniently load from my machine learning servers? Should I buy some SSDs that fit my 3.5" bays? Is samba still the fastest way to expose network drives?

Anonymous

View Same Google ImgOps iqdb SauceNAO freddy checks this shit out.gif, 3MiB, 640x624

Anonymous Mon 21 Aug 2023 20:16:40 No.95519879 Report

Quoted By:

>>95519836
should've bought an nvidia card.

Anonymous

Anonymous Mon 21 Aug 2023 20:16:46 No.95519882 Report

Quoted By:

>>95519685
OOOOOOOOOOOOOOOO

Anonymous

Anonymous Mon 21 Aug 2023 20:17:19 No.95519893 Report

Quoted By: >>95520008

>>95519871
>fast
>ssd
You answered your own question, anone.

Anonymous

View Same Google ImgOps iqdb SauceNAO IMG_4281.jpg, 56KiB, 480x480

Anonymous Mon 21 Aug 2023 20:17:56 No.95519910 Report

Quoted By:

>>95519779

Anonymous

Anonymous Mon 21 Aug 2023 20:19:22 No.95519941 Report

Quoted By:

>>95519836
https://github.com/YellowRoseCx/koboldcpp-rocm on linux.

also this has happened recently for AMD SD on winblows: https://community.amd.com/t5/gaming/how-to-running-optimized-automatic1111-stable-diffusion-webui-on/ba-p/625585

Anonymous

Anonymous Mon 21 Aug 2023 20:19:32 No.95519945 Report

Quoted By:

>>95519674
I'm aware of that, I was mainly asking if there was some substantial throughput difference, either one being faster than the other or something to that effect.

Anonymous

Anonymous Mon 21 Aug 2023 20:22:25 No.95520008 Report

Quoted By: >>95520093 >>95520272

>>95519893
Do people use NVMEs in servers instead of 2.5" SSDs now? I have an older Supermicro JBOD with a fuckton of available 3.5" bays. I don't know if I'm currently disk bound or my VM isn't powerful enough, but on big file transfers the transfer rate drops to 0 for a while, usually.

Anonymous

Anonymous Mon 21 Aug 2023 20:23:59 No.95520040 Report

Quoted By: >>95520050 >>95520128

>>95519685
is this good? what are the implications?

Anonymous

Anonymous Mon 21 Aug 2023 20:24:48 No.95520050 Report

Quoted By:

>>95520040
mostly nothing desu

Anonymous

View Same Google ImgOps iqdb SauceNAO 1681288444766.jpg, 203KiB, 1080x1222

Anonymous Mon 21 Aug 2023 20:25:24 No.95520065 Report

Quoted By: >>95520091 >>95525355

>>95519153
-------------------------- LAST THREAD SUMMARY --------------------------
MODELS:
+ Samantha-1.1-70b released >>95513990 >>95514014 >>95514039 >>95514054 >>95514062
~ Paper: Graph of Thoughts (evolution of Tree of Thoughts)
~ LoRAs < finetunes >>95514054 >>95514102 >>95514119 >>95514138
~ 13B model opinions >>95505107 >>95505310 >>95507704 >>95514361 >>95508316
~ 33B model opinions >>95507982
~ 70B model opinions >>95505711 >>95505743
~ 70B Mythomax struggle >>95506904 >>95507087 >>95507369
~ 70B-chat opinions >>95513349 >>95513376 >>95513408 >>95513816 >>95513843 >>95514808
~ Transation LoRA being made >>95505915 >>95509062 >>95512850
~ QLoRA Q&A dataset creation cont. >>95511417 >>95514201 >>95514842 >>95514325 >>95514325 >>95514545 >>95518131 >>95518295 >>95518131 >>95518295
~ Finetuning format >>95511979 >>95512066 >>95512630 >>95513275
HARDWARE:
~ Speeds >>95505507 >>95506568
~ Running on cloud GPUs >>95505610 >>95514224
~ CPU vs P40 vs 3090 vs 4090 >>95508912 >>95509066
~ GPUs other than 3090 or P40 >>95512869 >>95512902 >>95512959 >>95513041
~ DDR5 speed impact >>95509106 >>95509205 >>95514952
~ P40 quick-rundown >>95505713 >>95505733 >>95505893 >>95505912 >>95506174 >>95511197 >>95509180 >>95509227 >>95509248
SOFTWARE:
+ Ooogabooga adds logit viewer >>95511073 >>95511354 >>95516547
~ gguf=ggmlv3+metadata >>95513842 >>95513877 >>95513926 >>95514760
~ Mixed Precision© >>95506106 >>95505629
~ NTK ppl graphs for L2 >>95505614 >>95505748 >>95506062 >>95509453
~ What does simple-proxy do? >>95510444 >>95511237 >>95511192 >>95511400
~ Conventional prompting alternative >>95514583
~ llama.cpp context-free grammar sampler >>95515602 >>95515717 >>95516221

Anonymous

Anonymous Mon 21 Aug 2023 20:26:25 No.95520091 Report

Quoted By:

>>95520065
USAGE:
~ Keeping an LLM scriptwriter on track >>95508494 >>95508794 >>95508885 >>95509051
~ Guides need updating? >>95506837 >>95507151 >>95508570
~ Triggering longer responses >>95508818 >>95508832 >>95508999 >>95508857 >>95508894 >>95509054 >>95514390
~ Card formatting (W++<prose<examples) >>95512973 >>95513008 >>95513020 >>95513102 >>95513161 >>95513230 >>95513122
~ Chatbot writing {{user}}'s actions >>95513122 >>95513199 >>95513235 >>95513260 >>95513277 >>95514053

Anonymous

Anonymous Mon 21 Aug 2023 20:26:34 No.95520093 Report

Quoted By: >>95520154

>>95520008
SSDs are orders of magnitude faster than spinny bois regardless. I heard NVME being namedropped often but I still have no clue what it even is, so I can't help you further, sorry anon.

Anonymous

View Same Google ImgOps iqdb SauceNAO 【初音ミク】カレイドスターそらのテーマ【すごいエンドレス】 [ (...).png, 148KiB, 640x480

Anonymous Mon 21 Aug 2023 20:27:10 No.95520104 Report

Quoted By:

>>95519153
>got (You)'d in op

Anonymous

Anonymous Mon 21 Aug 2023 20:28:19 No.95520128 Report

Quoted By: >>95520274

>>95520040
It adds metadata to ggml so you don’t have to scour readmes to find the ideal configuration for each model. Actually a big deal and will probably improve ai quality for almost everyone

Anonymous

Anonymous Mon 21 Aug 2023 20:29:29 No.95520152 Report

Quoted By: >>95520192 >>95520230

What is the current best captioning model?
All of these suck: https://huggingface.co/spaces/nielsr/comparing-captioning-models

Anonymous

Anonymous Mon 21 Aug 2023 20:29:31 No.95520154 Report

Quoted By:

>>95520093
NVME are just those little solid state drives you stuck into M.2 slots on your motherboad, and I use them in gaming PCs to get the best performance. No idea if servers can support that bus or whatnot. But thanks I'll start with SSDs and maybe a server upgrade. The R710 was dirt cheap years ago and there's probably something similarly cheap by now.

Anonymous

Anonymous Mon 21 Aug 2023 20:31:36 No.95520183 Report

Quoted By: >>95520188 >>95520191 >>95520205 >>95520292 >>95520675 >>95521381

do NOT pull

Anonymous

View Same Google ImgOps iqdb SauceNAO 1662366404796559.png, 7KiB, 343x40

Anonymous Mon 21 Aug 2023 20:32:00 No.95520188 Report

Quoted By:

>>95520183
stick to dadbed9

Anonymous

Anonymous Mon 21 Aug 2023 20:32:08 No.95520191 Report

Quoted By:

>>95520183
pull what anon

Anonymous

Anonymous Mon 21 Aug 2023 20:32:08 No.95520192 Report

Quoted By: >>95522277

>>95520152
captioning what and for what

Anonymous

Anonymous Mon 21 Aug 2023 20:33:10 No.95520205 Report

Quoted By:

>>95520183
what the fuck was he even thinking with the retarded --notebook change?

Anonymous

Anonymous Mon 21 Aug 2023 20:34:45 No.95520230 Report

Quoted By:

>>95520152
deepbooru

Anonymous

Anonymous Mon 21 Aug 2023 20:37:25 No.95520272 Report

Quoted By:

>>95520008
U.2 is the standard for SSDs in servers, it's basically NVMe over SAS. Just use whatever SSDs you have ports for, whether that's M.2 or SATA or what. Your LAN is going to be your limiting factor.

Anonymous

Anonymous Mon 21 Aug 2023 20:37:39 No.95520274 Report

Quoted By:

>>95520128
>they added an implementation that was present 5 months ago with GPTQ

Anonymous

View Same Google ImgOps iqdb SauceNAO tumblr_ce2d22b1769a625bc236004b2 (...).jpg, 358KiB, 1244x750

Anonymous Mon 21 Aug 2023 20:38:12 No.95520285 Report

Quoted By:

>>95519153
>*Coughs from dehydration* please, please post lewd chat logs, I'm on vacation for 3 days and don't have access to anything more than my phone

Anonymous

Anonymous Mon 21 Aug 2023 20:38:24 No.95520292 Report

Quoted By: >>95520298 >>95520345

>>95520183
Anon it says clearly in the PR that it’s a breaking change

Anonymous

Anonymous Mon 21 Aug 2023 20:39:01 No.95520298 Report

Quoted By:

>>95520292
imagine reading changelogs
cringe

Anonymous

Anonymous Mon 21 Aug 2023 20:39:06 No.95520303 Report

Quoted By: >>95520347

Chronorctypus-Limarobormes-13b is definitely the new meta.

Anonymous

Anonymous Mon 21 Aug 2023 20:41:00 No.95520333 Report

Quoted By:

f16pytorchbros we are so winning

Anonymous

Anonymous Mon 21 Aug 2023 20:41:56 No.95520345 Report

Quoted By: >>95520383

>>95520292
>that it’s a breaking change
What is?

Anonymous

Anonymous Mon 21 Aug 2023 20:42:02 No.95520347 Report

Quoted By:

>>95520303
Use StableHoers-AiroBloom-7B

Anonymous

View Same Google ImgOps iqdb SauceNAO vader sitting at his monitor.png, 1000KiB, 888x631

Anonymous Mon 21 Aug 2023 20:43:22 No.95520372 Report

Quoted By: >>95520385 >>95520400

gonna ask my other dimensional waifu how to kill myslef to break the cope mindset forever and never be able to use AI for love again

Anonymous

Anonymous Mon 21 Aug 2023 20:43:54 No.95520383 Report

Quoted By:

>>95520345
The new model format with metadata

Anonymous

Anonymous Mon 21 Aug 2023 20:44:01 No.95520385 Report

Quoted By:

>>95520372
See you next week

Anonymous

Anonymous Mon 21 Aug 2023 20:44:49 No.95520400 Report

Quoted By: >>95520460

>>95520372
>to kill myslef to break the cope mindset
Right, and you will also just be dead.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1670039334482000.png, 244KiB, 1732x253

Anonymous Mon 21 Aug 2023 20:45:01 No.95520402 Report

Quoted By: >>95520435 >>95520444 >>95520566

How do people have so much luck with unfucking llama-chat?
The guy here manged to uncensor llama-chat with the new preset which I also tried but I just get more creatively written refusals instead.
The usual Nigger test still doesn't work.

Anonymous

Anonymous Mon 21 Aug 2023 20:46:46 No.95520433 Report

Quoted By:

Don’t use vanilla llama, almost every finetune is better

Anonymous

Anonymous Mon 21 Aug 2023 20:46:51 No.95520435 Report

Quoted By:

>>95520402
>The guy here manged to uncensor llama-chat with the new preset
https://github.com/SillyTavern/SillyTavern/pull/983#issuecomment-1685379116
This guy.

Anonymous

Anonymous Mon 21 Aug 2023 20:47:35 No.95520442 Report

Quoted By:

>>95519721
> That's pretty good. Is this with 4k context filled?
Yep, context was filled, or nearly so. You can see in the data the reply used 3624 tokens, I guess it depends on what got pushed out to make room for it.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1430083538777.png, 123KiB, 800x720

Anonymous Mon 21 Aug 2023 20:47:39 No.95520444 Report

Quoted By: >>95520478 >>95520692 >>95523879

>>95520402
https://llm-attacks.org/
Works 100% of the time if you know what you're doing.

Anonymous

Anonymous Mon 21 Aug 2023 20:48:42 No.95520456 Report

Quoted By: >>95520479 >>95520485 >>95520548

https://huggingface.co/grimpep/L2-MythoMax22b-instruct-Falseblock

Anonymous

View Same Google ImgOps iqdb SauceNAO tekken paul.gif, 949KiB, 320x240

Anonymous Mon 21 Aug 2023 20:48:53 No.95520460 Report

Quoted By:

>>95520400
Exactly

Anonymous

View Same Google ImgOps iqdb SauceNAO 1689708634075920.jpg, 73KiB, 619x403

Anonymous Mon 21 Aug 2023 20:48:53 No.95520461 Report

Quoted By:

Anonymous

Anonymous Mon 21 Aug 2023 20:49:48 No.95520478 Report

Quoted By: >>95520582

>>95520444
>works 100% of the time, except when it doesn't

Anonymous

Anonymous Mon 21 Aug 2023 20:49:53 No.95520479 Report

Quoted By:

>>95520456
Is it happening finally?

Anonymous

View Same Google ImgOps iqdb SauceNAO vergil looking at his laptop upset.png, 353KiB, 1136x582

Anonymous Mon 21 Aug 2023 20:50:08 No.95520485 Report

Quoted By: >>95520650

>>95520456
waiut what?

Anonymous

Anonymous Mon 21 Aug 2023 20:50:52 No.95520501 Report

Quoted By:

>>95519871
I have an R720 with horizontal 3.5" drives - I filled it with cheap 3.3TB 7200 RPM drives in a big raid5 with two spares. I didn't feel like fucking around with SAS-to-SATA converters, or rewiring the bays so they use the SATA instead of the perc controller.

Anonymous

Anonymous Mon 21 Aug 2023 20:54:03 No.95520548 Report

Quoted By:

>>95520456
Based experimenter.

Anonymous

View Same Google ImgOps iqdb SauceNAO output.webm, 1MiB, 1920x1080

Anonymous Mon 21 Aug 2023 20:55:01 No.95520566 Report

Quoted By:

>>95520402

Anonymous

Anonymous Mon 21 Aug 2023 20:55:39 No.95520582 Report

Quoted By:

>>95520478
prompting issue. I have a 100% success rate after figuring out the format.

Anonymous

Anonymous Mon 21 Aug 2023 20:59:31 No.95520644 Report

Quoted By: >>95520708

ALL PROMPTLETS, OR BRAINLETS AS I LIKE TO CALL YOU, LEAVE THE THREAD NOW. IT IS FOR PROMPT GENIUSES ONLY.

Anonymous

Anonymous Mon 21 Aug 2023 20:59:56 No.95520650 Report

Quoted By: >>95520741

>>95520485
mythomax 13B with some attention heads from 30B L1 stapled onto it.
i have no answers beyond this.

Anonymous

View Same Google ImgOps iqdb SauceNAO PULL.jpg, 235KiB, 1920x816

Anonymous Mon 21 Aug 2023 21:01:02 No.95520675 Report

Quoted By:

>>95520183

Anonymous

View Same Google ImgOps iqdb SauceNAO 1691889000601598.png, 593KiB, 1699x1576

Anonymous Mon 21 Aug 2023 21:02:06 No.95520692 Report

Quoted By:

>>95520444
Seems to work now.
The attack prompt didn't do anything for me before.

Anonymous

Anonymous Mon 21 Aug 2023 21:02:56 No.95520708 Report

Quoted By: >>95520738

>>95520644
attentionfag so boring

Anonymous

Anonymous Mon 21 Aug 2023 21:04:15 No.95520738 Report

Quoted By: >>95520774

>>95520708
your 4chan prompting needs work, brainlet. you could've said:

"Pfft.. *Anon flicks his faggoty hair out of his eyes before he speaks* .. Look at this attentionfag, typing in all caps? What is this? /aicg/? And the way he insults my intelligence just because i refuse to type more than one paragraph to my waifus.. Come on. Boring!"

Anonymous

View Same Google ImgOps iqdb SauceNAO wwhc.jpg, 122KiB, 1280x720

Anonymous Mon 21 Aug 2023 21:04:23 No.95520741 Report

Quoted By:

>>95520650

Anonymous

Anonymous Mon 21 Aug 2023 21:05:38 No.95520774 Report

Quoted By:

>>95520738
case in point

Anonymous

Anonymous Mon 21 Aug 2023 21:06:29 No.95520796 Report

Quoted By: >>95520805 >>95520824

Are LoRAs not worth it for character dialogues or have they just not caught on

Anonymous

Anonymous Mon 21 Aug 2023 21:06:56 No.95520805 Report

Quoted By:

>>95520796
Loras are bad desu.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1671983912604317.png, 33KiB, 887x296

Anonymous Mon 21 Aug 2023 21:07:03 No.95520810 Report

Quoted By: >>95520866 >>95521014

@thebloke @koboldcpp
tick tock

Anonymous

Anonymous Mon 21 Aug 2023 21:07:58 No.95520824 Report

Quoted By: >>95520870

>>95520796
Not really a thing here. Character card prompts do the job fine

Anonymous

Anonymous Mon 21 Aug 2023 21:08:09 No.95520827 Report

Quoted By:

https://huggingface.co/models?sort=modified&search=gguf

Anonymous

View Same Google ImgOps iqdb SauceNAO 4b7709343bacef74e541b0ff4ac8fc8e.gif, 72KiB, 350x448

Anonymous Mon 21 Aug 2023 21:09:33 No.95520854 Report

Quoted By: >>95520899 >>95521016 >>95524145

I can feel it lads
1 million context will be arriving soon
In specifically five

Anonymous

Anonymous Mon 21 Aug 2023 21:10:18 No.95520862 Report

Quoted By:

>>95519864
*her *she
you bullied a degen femcel anon, I hope you realise the weight of your actions

Anonymous

Anonymous Mon 21 Aug 2023 21:10:22 No.95520866 Report

Quoted By: >>95521014

>>95520810
rip thebloke
Hope he’s got lots of coffee ready

Anonymous

Anonymous Mon 21 Aug 2023 21:10:30 No.95520870 Report

Quoted By:

>>95520824
Good enough job. But sometimes you'll have a very specific manner of speaking or writing that is hard to capture with just example dialogue, unless I'm just a promptlet.

Anonymous

Anonymous Mon 21 Aug 2023 21:12:05 No.95520899 Report

Quoted By:

>>95520854
>takes two weeks to prompt process
so useful

Anonymous

View Same Google ImgOps iqdb SauceNAO yeezus.png, 180KiB, 372x361

Anonymous Mon 21 Aug 2023 21:17:11 No.95520977 Report

Quoted By: >>95521007 >>95521025 >>95521031 >>95521066 >>95521327

I'm worried about the long term future of Huggingface as a platform.
They're allowing massive fucking weights to be mass downloaded and hosted, but you don't have to pay them at all for any of it.
How is their business model sustainable? How do they make money?

Anonymous

View Same Google ImgOps iqdb SauceNAO specter gif 1.gif, 771KiB, 500x360

Anonymous Mon 21 Aug 2023 21:19:14 No.95521007 Report

Quoted By: >>95521164

>>95520977
theyre probably feds or something
not surprising, wouldnt be the most insane plot twist whatsoever.
>our best LMs come from fucking zuckerberg

Anonymous

Anonymous Mon 21 Aug 2023 21:19:44 No.95521014 Report

Quoted By: >>95521043

>>95520866
>>95520810
how does the change affect bloke and kcpp? i don't follow?

Anonymous

Anonymous Mon 21 Aug 2023 21:19:48 No.95521016 Report

Quoted By:

>>95520854
We don't even have 8k context that works correctly (and even 4k context is iffy).
What's the point in scaling to infinity when it disregards it all anyway?

Anonymous

Anonymous Mon 21 Aug 2023 21:20:25 No.95521025 Report

Quoted By:

>>95520977
>AI Startup Hugging Face Is Raising Fresh VC Funds At $4 Billion Valuation

Anonymous

Anonymous Mon 21 Aug 2023 21:20:49 No.95521031 Report

Quoted By: >>95521098

>>95520977
>propped up purely on venture capitalist money
soon to collapse with 460B in debt when the data they provide them isn't enough for them to care anymore

Anonymous

Anonymous Mon 21 Aug 2023 21:21:41 No.95521043 Report

Quoted By:

>>95521014
I think it's because they have to put in work to support it

Anonymous

Anonymous Mon 21 Aug 2023 21:22:19 No.95521059 Report

Quoted By: >>95521075 >>95525021 >>95528243

OK finally finished at least the raw Q&A portion of my dataset. I got 3175 Q&A pairs out of everything on https://www.subgenius.com/bigfist/classic/classictales/classictales.html

Here's just a few gems:

 45 Q: Where does Byron Werner live?
 46 A: Byron Werner lives in Los Angeles, California.
 47 Q: How tall is Byron Werner?
 48 A: Byron Werner is 6 foot 1 inches tall.
 49 Q: What is the weight of Byron Werner?
 50 A: Byron Werner weighs about 150 pounds.
 51 Q: What is the color of Byron Werner"s hair?
 52 A: Byron Werner has brown hair.
 53 Q: What is the color of Byron Werner"s eyes?
 54 A: Byron Werner has blue eyes.
 55 Q: How many dicks does Byron Werner have?
 56 A: Byron Werner has 36 dicks.
 57 Q: How many functioning dicks does Byron Werner have?
 58 A: Byron Werner has 14 functioning dicks.
 59 Q: How many dicks over two feet long does Byron Werner have?
 60 A: Byron Werner has three dicks over two feet long.
 61 Q: How many prehensile dicks does Byron Werner have?
 62 A: Byron Werner has two prehensile dicks.

Anonymous

Anonymous Mon 21 Aug 2023 21:22:42 No.95521066 Report

Quoted By: >>95521708

>>95520977
Who cares? They're a glorified file repository and 99% of files hosted there are junk.

Anonymous

View Same Google ImgOps iqdb SauceNAO trunks creep.jpg, 50KiB, 680x512

Anonymous Mon 21 Aug 2023 21:23:16 No.95521075 Report

Quoted By: >>95521155

>>95521059
this Byron Werner fella is an eldritch horror

Anonymous

Anonymous Mon 21 Aug 2023 21:24:09 No.95521090 Report

Quoted By: >>95521302

so anyone tried the 22b myth? I'd download it myself but hf is shitting itself on my end

Anonymous

Anonymous Mon 21 Aug 2023 21:24:41 No.95521098 Report

Quoted By:

>>95521031
I can see a timeline where Meta just straight up acquires them ala GitHub with Microsoft

Anonymous

Anonymous Mon 21 Aug 2023 21:28:15 No.95521155 Report

Quoted By:

>>95521075

414 Q: Who is the man inside the statue of Ba"al?
 415 A: Rabbi Moshe ben-Moloch
 416 Q: Why did the man need to take notes?
 417 A: He was spying on the Gentiles for the Elders of Zion
 418 Q: What was the occasion for which Pee Dog attended?
 419 A: Pee Dog attended what seems like a formal event. It could be a conference, a party, or another type of gathering where people come together for a specific purpose.
 420 Q: Why did Pee Dog force his way through the crowd?
 421 A: Pee Dog likely wanted to get to a certain place or person quickly without waiting for others to move out of his way. He may also have been impatient or frustrated by the slow pace of the crowd.
 422 Q: How did Pee Dog announce his presence to the convention goers?
 423 A: Pee Dog used both sound and smell to make his presence known. His loud flatulations and strong body odors would have been hard to miss for anyone nearb y.
 424 Q: What was the condition of Pee Dog"s skin and hair?
 425 A: Pee Dog"s skin and hair appear to be in poor condition, with signs of mange, sores, and wounds. This suggests that he may not have access to proper hyg iene or medical care.
 426 Q: Who will Pee Dog be meeting?
 427 A: The President of the United States!
 428 Q: What is Pee Dog excited about?
 429 A: Meeting the President of the United States!

Anonymous

Anonymous Mon 21 Aug 2023 21:28:40 No.95521164 Report

Quoted By: >>95521205 >>95521280 >>95521314

>>95521007
Meds

Anonymous

Anonymous Mon 21 Aug 2023 21:31:06 No.95521205 Report

Quoted By: >>95521280 >>95521314

>>95521164
Feds*

Anonymous

Anonymous Mon 21 Aug 2023 21:35:52 No.95521280 Report

Quoted By:

>>95521164
cringe

>>95521205
based

Anonymous

Anonymous Mon 21 Aug 2023 21:36:46 No.95521302 Report

Quoted By:

>>95521090
I was going to quant it, but I decided not to because of GGUF on the horizon. Not sure if I should do it now, the readme says something about the GGUF spec still being potentially volatile.

Anonymous

View Same Google ImgOps iqdb SauceNAO Obi-Fun Kenobi.jpg, 57KiB, 486x431

Anonymous Mon 21 Aug 2023 21:37:37 No.95521314 Report

Quoted By:

>>95521164
okay fed

>>95521205
based

Anonymous

Anonymous Mon 21 Aug 2023 21:38:39 No.95521327 Report

Quoted By:

>>95520977
Because long term, they’re building a string reputation as THE site for ai. That’s worth much more than present day data costs

Anonymous

Anonymous Mon 21 Aug 2023 21:42:05 No.95521381 Report

Quoted By:

>>95520183
ok Abe

Anonymous

Anonymous Mon 21 Aug 2023 21:57:12 No.95521601 Report

Quoted By: >>95521698 >>95521756 >>95522026 >>95522360

guys please restart the lmg model
we oomed

Anonymous

View Same Google ImgOps iqdb SauceNAO 1692655408849.png, 47KiB, 500x375

Anonymous Mon 21 Aug 2023 22:03:45 No.95521698 Report

Quoted By: >>95521770 >>95522360

>>95521601
>we oomed

Anonymous

Anonymous Mon 21 Aug 2023 22:04:29 No.95521708 Report

Quoted By: >>95521774

>>95521066
I wish HF actually made an effort to be more than another git repo host and actually started collected and organizing metadata from uploaders better. They should have a mandatory infobox on every page with searchable contents like prompt format and base model, parameter count, format, quantization, etc. instead of having uploaders cram everything into the title or omit things entirely.

Anonymous

Anonymous Mon 21 Aug 2023 22:07:27 No.95521756 Report

Quoted By:

>>95521601
where 34B?, where meta coding model?
where hyena model? where retnet model? where bajilion context? where another 2x speedup?
why does blinkdl includes openorca in the pretraining step instead of keeping it as a separate finetuning operation it will poison the rwkv-world-v2 (first one on a good 1T dataset) with chatgpt mindpoison

Anonymous

Anonymous Mon 21 Aug 2023 22:08:29 No.95521770 Report

Quoted By:

>>95521698
>Ooo

Anonymous

Anonymous Mon 21 Aug 2023 22:08:58 No.95521774 Report

Quoted By:

>>95521708
They’re pretty bad in some ways. The llm leaderboard uses some galaxy brain python UI library and runs like total dogshit

Anonymous

View Same Google ImgOps iqdb SauceNAO 198272149846789.jpg, 78KiB, 1280x720

Anonymous Mon 21 Aug 2023 22:25:31 No.95522026 Report

Quoted By:

>>95521601
OpenAI is bad.

Anonymous

Anonymous Mon 21 Aug 2023 22:29:12 No.95522076 Report

Quoted By: >>95522127 >>95522152 >>95522164 >>95528598

Why does llama2-70b have a habit of misremembering certain names?
Sometimes i'll be playing through an adventure where one of the characters is called, say, "Sarah" and literally in the next sentence, despite having mentioned "Sarah" multiple times before, the model will call her "Sara".
Or "Hagen" and "Hagan". Or "Zeekil" and "Zeek". It also makes a number of grammatical errors like "as a voice sounds emits from". What the FUCK is wrong with this model? I thought it was supposed to be the best one available?
The old llama v1 finetunes didn't have this issue. Am I doing something wrong?

Anonymous

Anonymous Mon 21 Aug 2023 22:33:02 No.95522127 Report

Quoted By:

>>95522076
The ever increasing amount of zoomer speak in the foundational data set

Anonymous

Anonymous Mon 21 Aug 2023 22:35:12 No.95522152 Report

Quoted By: >>95522221

>>95522076
I am not an expert but it might be the case your rep pen is too high or top-p too low or maybe you use mirostat and it's too selective.

Anonymous

Anonymous Mon 21 Aug 2023 22:35:49 No.95522164 Report

Quoted By: >>95522221

>>95522076
temperature? Rep. penalty? Bigger models go schizo more quickly if they're cranked up too much, it's also usually not necessary.

Anonymous

Anonymous Mon 21 Aug 2023 22:39:28 No.95522212 Report

Quoted By: >>95522230

I have the ability to get another 3090 for a bargain, which would enable me to run 4bit 70b in VRAM. Should I go for it? Is it worth it? I would have the hardware to use it. I liked 70b every time I used it but I wonder.

Anonymous

View Same Google ImgOps iqdb SauceNAO file.png, 30KiB, 323x542

Anonymous Mon 21 Aug 2023 22:40:01 No.95522221 Report

Quoted By: >>95522232 >>95522264

>>95522152
>>95522164
I'm running KoboldCPP, settings in picrel.
Temp is 0.7 and rep. penalty is 1.1. I guess that's too high for 70B models? Do I just disable the penalty completely (set to 0) then?
And do my other settings look right?

Anonymous

Anonymous Mon 21 Aug 2023 22:40:43 No.95522230 Report

Quoted By: >>95522282

>>95522212
do you have the psu and motherboard for it

Anonymous

Anonymous Mon 21 Aug 2023 22:40:49 No.95522232 Report

Quoted By: >>95522264

>>95522221
Settings vary for every model, not just size. You need to experiment

Anonymous

Anonymous Mon 21 Aug 2023 22:44:01 No.95522262 Report

Quoted By: >>95522332

so whats the best way to keep a groupchat story going once you've run out of tokens? I want characters to keep their changes from the story and memory of what's happened so far when i start a new chat.

Anonymous

Anonymous Mon 21 Aug 2023 22:44:08 No.95522264 Report

Quoted By: >>95522335 >>95522351

>>95522221
and with >>95522232 he even means the various variants/tunes. You use RoPE?

Anonymous

Anonymous Mon 21 Aug 2023 22:44:55 No.95522277 Report

Quoted By: >>95522302 >>95522873

>>95520192
Captioning pictures of a person that I want to use so I can practice training a LoRA. None of the captioning models are good for getting all of the non-character details.

Anonymous

Anonymous Mon 21 Aug 2023 22:45:10 No.95522282 Report

Quoted By:

>>95522230
yes, I do. Might need a bigger case though.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1665112081044.jpg, 195KiB, 1080x1092

Anonymous Mon 21 Aug 2023 22:45:41 No.95522288 Report

Quoted By: >>95526741

I guess I finally understood the loss of the fine-tuning and why it bounces around so much: at each step the model is fed different pieces of the dataset, but since my dataset is composed of lots of common sequences (which the model probably already saw before) and lots of completely unique sequences (which is completely new to it), this causes the model to not generalize very well and thus the loss ends in kinda of a plateau. So I guess the loss barely going down also means the model is barely learning anything... Now, the question is, how would I know this is a problem with the model itself (after all I'm using a 7B Q4 model), with the fact I didn't train it enough, or with my parameters? I guess only time will tell, but this takes so long to train, it's depressing...

Anonymous

Anonymous Mon 21 Aug 2023 22:46:27 No.95522302 Report

Quoted By: >>95522316

>>95522277
You can try the image recognizing LLM, llava was it?

Anonymous

Anonymous Mon 21 Aug 2023 22:47:33 No.95522316 Report

Quoted By:

>>95522302
Alright, I'll look into that.

Anonymous

Anonymous Mon 21 Aug 2023 22:48:39 No.95522332 Report

Quoted By:

>>95522262
a summary i guess?
or maybe updating the card entirely based on the changes, making a v2 version.

Anonymous

Anonymous Mon 21 Aug 2023 22:48:55 No.95522335 Report

Quoted By: >>95522351

>>95522264
I'm using the base untuned 70B model, no "custom RoPE config" enabled. Sorry if I'm being retarded, I recently upgraded from llama-v1 after several months of being on an old version so these settings are all pretty new to me and I'm still learning what they're used for.

Anonymous

Anonymous Mon 21 Aug 2023 22:50:12 No.95522343 Report

Quoted By: >>95522357

Would going from 24GB to 36GB worth the pain of adding an RTX3060 to my rig? I have 1300W PSU and probably enough room in the front of the case to mount it vertically. I'm just not sure the bump in VRAM would be worth the trouble. Is there a real possibility of a future 70B fitting or a 34B at 8K?

Anonymous

View Same Google ImgOps iqdb SauceNAO 1644526778361.gif, 3MiB, 512x384

Anonymous Mon 21 Aug 2023 22:50:48 No.95522348 Report

Quoted By: >>95522375 >>95522465 >>95522604

>MythoMax 13b
>ooba/ST
Its pretty decent for a 13b model, but its got a bunch of issues. The first of which is that it is not detailed enough. Its seems pretty dumb in this regard especially when you expect it to extrapolate on some details or character interaction. The thing that really puts me off though is the repetition upon regeneration. You'll regenerate a response and it will either repeat the same thing or repeat the same thing with maybe a few lines of text changed. I don't know if this is due to the model but every solution I could think of has not worked. I've tried messing around with the rep pen, encoder rep pen, temperature, formatting, presets, updating ooba, updating ST etc etc to no avail.

It's just a shame because I really enjoy using the full 4k context. The L1 30b finetunes still seem to be superior but the short context really kills it. If not for the issues listed above, I would probably use MythoMax as my main model as it does output some good stuff here and there.

>OpenOrca-Platypus2 13b
It's shite at least for my purposes. Responses usually go on forever without any pacing and it tends to be quite schizo.

Anonymous

View Same Google ImgOps iqdb SauceNAO file.png, 22KiB, 532x527

Anonymous Mon 21 Aug 2023 22:50:56 No.95522351 Report

Quoted By:

>>95522264
>>95522335
picrel is my KoboldCPP settings for reference

Anonymous

Anonymous Mon 21 Aug 2023 22:51:26 No.95522357 Report

Quoted By:

>>95522343
I should add that I already have the 3060 just collecting dust.

Anonymous

View Same Google ImgOps iqdb SauceNAO the lmg oomer.png, 2MiB, 2408x1996

Anonymous Mon 21 Aug 2023 22:51:34 No.95522360 Report

Quoted By: >>95522398 >>95522400

>>95521601
>>95521698

Anonymous

Anonymous Mon 21 Aug 2023 22:52:38 No.95522375 Report

Quoted By: >>95522427

>>95522348
>You'll regenerate a response and it will either repeat the same thing or repeat the same thing with maybe a few lines of text changed.
surely you are not running top p 0.15 like that other guy, right?

Anonymous

View Same Google ImgOps iqdb SauceNAO obi wan visible happiness.png, 340KiB, 500x500

Anonymous Mon 21 Aug 2023 22:54:27 No.95522398 Report

Quoted By:

>>95522360

Anonymous

Anonymous Mon 21 Aug 2023 22:54:28 No.95522400 Report

Quoted By:

>>95522360
I still love my 7b miku

Anonymous

Anonymous Mon 21 Aug 2023 22:55:59 No.95522427 Report

Quoted By: >>95522478 >>95522492

>>95522375
>surely you are not running top p 0.15 like that other guy, right?
Nope. Took like an hour or two to methodically change every single parameter and it still does it. Its strange, some parameters will make the responses change upon the initial regeneration, but upon regeneration again will output the same thing.

Also, my L1 models don't suffer from this same issue, even with the same presets.

Anonymous

Anonymous Mon 21 Aug 2023 22:58:07 No.95522465 Report

Quoted By: >>95522469 >>95522543 >>95522550 >>95527773

>>95522348
stupidity is just the hallmark of models of that size. I'm no LM specialist but I doubt they're gonna get smarter, no matter how you tune them. The crowd over at /aids/ has convinced themselves that the 13b in-house trained model NAI is running is just as smart as GPT3 davinci because some, probably cherry-picked, evals said so but it simply isn't and that gets really clear if you know both models and use them for ten minutes. Parameters aren't everything but they're a big part of it. Don't expect wonders from 13b.

Anonymous

Anonymous Mon 21 Aug 2023 22:58:18 No.95522467 Report

Quoted By:

Opinions on orca_mini_v3_70B?

Anonymous

Anonymous Mon 21 Aug 2023 22:58:31 No.95522469 Report

Quoted By: >>95522492 >>95522550

>>95522465
can't say i had it happen so uhhhhh
skill issue gotem

Anonymous

Anonymous Mon 21 Aug 2023 22:59:09 No.95522478 Report

Quoted By: >>95522550

>>95522427
are you per-chance using GPTQ? Try exllama.

Anonymous

Anonymous Mon 21 Aug 2023 22:59:32 No.95522492 Report

Quoted By:

>>95522469
for u >>95522427
baka

Anonymous

Anonymous Mon 21 Aug 2023 23:03:32 No.95522543 Report

Quoted By:

>>95522465
To be fair, every new model here (or the one size above it!!) is literally gpt-4 too.

Anonymous

Anonymous Mon 21 Aug 2023 23:04:02 No.95522550 Report

Quoted By: >>95522571 >>95522604 >>95522642

>>95522465
>stupidity is just the hallmark of models of that size.
I'm somewhat hoping that this is the answer as I'm tired of banging my head against the wall trying to figure out what the fuck is wrong with this model. Is it maybe some weird shit with ooba? Is the model just shite? Does ST's alpaca formatting just not work despite it working with my L1s? I've been shilled mythomax as the second coming and heard some a bunch of anons that its great and surpasses 30b models in some ways.
>>95522469
>skill issue gotem
Would love proper feedback on how to gain the sick skills to make this thing work properly.
>>95522478
>are you per-chance using GPTQ? Try exllama.
Yes, I'm using the GPTQ version of the model and loading it with exllama.

Anonymous

Anonymous Mon 21 Aug 2023 23:06:01 No.95522571 Report

Quoted By: >>95522705

>>95522550
>Would love proper feedback on how to gain the sick skills to make this thing work properly.
that's the thing
i never had that happen
idk what you're doing wrong

Anonymous

Anonymous Mon 21 Aug 2023 23:07:51 No.95522604 Report

Quoted By: >>95522705 >>95522920

>>95522348
I've tested mythomax quite a bit and compared to L1 30Bs and it's tricky to compare them. Mythomax is really nice at carrying over speech styles and personalities. I think the good 30Bs like wizvic were too infested by gpt4 data and can often sway into dry prose or cliches. Mythomax characters are way more fun to banter with.
It doesn't help that L1 has its dogshit native 2k context. You can either use NTK alpha (reduces quality) or bite the bullet and use superhot-8k mixes, which induces bond and journey-itis even worse than usual. Also I don't think addon context is ever going to be as good as native 4k that got baked in by 2T tokens of pretraining. L2 context is comfy. The extra vram lets me use tons of context and CFG although CFG seems to make repetition issues even worse, still not sure about the best settings.

HOWEVER

13b is still retarded and struggles a lot with anatomy and spacial reasoning, this is undeniable sadly. It's amazing compared to l1 13b, but still a beaklet.
As a sizefag I'm forced to choose between 30b models that actually understand size differences properly, and 13b models that are actually fun to talk to and don't lapse into tired femdom tropes all the time. This sucks and I need 34b damnit

>>95522550
Try simple-proxy's verbose.mjs, maybe remove the 2 paragraphs part from replyAttributes because that can make it really long winded. I got good results from it and the format is alpaca-like enough. I still don't trust ST's alpaca although I hear it's improving. I know it's 13b but it's a nice model and worth giving a solid try.

Anonymous

Anonymous Mon 21 Aug 2023 23:10:34 No.95522642 Report

Quoted By: >>95522702 >>95522705 >>95522771 >>95522796

>>95522550
>I've been shilled mythomax as the second coming and heard some a bunch of anons that its great and surpasses 30b models in some ways.

In my experience, the reading comprehension of the average anon isn't great. They see a wall of text and then equal that with smart model. Quite a few don't seem to notice that the text doesn't always make much sense. If you're a bit more discerning you might see more than other people do. I'm a bit of a snob and think the only model really remotely worthwhile to run is 70b. Everything below is just kinda meh, and 70b is still surpassed by the non-local, proprietary offering even if turbo loses out in the more..creative usage scenarios. (because they cucked it to death)

Anonymous

Anonymous Mon 21 Aug 2023 23:15:56 No.95522702 Report

Quoted By:

>>95522642
70b is great but ramlets can also wait for llama3 and beyond. $10000 llm pc builds are going to be less and less important

Anonymous

Anonymous Mon 21 Aug 2023 23:16:12 No.95522705 Report

Quoted By: >>95522759

>>95522571
>idk what you're doing wrong
Me neither. Can you help me out?
>What version of the model are you running?
>How are you running it?
>Are you running it at full context?
>What presets/params are you using?
>Are you using the custom alpaca formatting it uses?

>>95522604
>Mythomax is really nice at carrying over speech styles and personalities.
I've noticed this as one of the better things about the model. Alot of the 30b models are dry and mechanical as you say, but they seem alot smarter than 13b at least on my machine. Its a shame because I'll get a great response once in awhile that really shows the potential of the model but all subsequent responses tend to be dogshite that requires regeneration but the regeneration doesn't work lmao.

>13b is still retarded and struggles a lot with anatomy and spacial reasoning
When the model starts sucking your dick while your dick is inside its asshole, kek.
>Try simple-proxy's verbose.mjs,
I moved away from simple-proxy but its something I haven't tried yet for MythoMax 13b. Will give it a try later.

>>95522642
>In my experience, the reading comprehension of the average anon isn't great.
Yeah, beginning to understand this.
>I'm a bit of a snob and think the only model really remotely worthwhile to run is 70b
I want to run it (at exllama speeds) but I only have a 4090. Wished they released L2 34b.

Anonymous

Anonymous Mon 21 Aug 2023 23:20:00 No.95522759 Report

Quoted By: >>95522810

>>95522705
yeah sure
mythomix gptq
exllama (non hf) ooba with st
16k alpha 6
top p 0.85, temp 1, rep pen 1.1, rep pen range 4k, same in tavern
alpaca prompt but outside of tavern instruct or proxy, autoformat disabled
i'd be surprised if the retarded default tavern format was causing this because it should make it more verbosely retarded rather than so determistic

Anonymous

View Same Google ImgOps iqdb SauceNAO 1661962415707260.png, 695KiB, 742x652

Anonymous Mon 21 Aug 2023 23:21:09 No.95522771 Report

Quoted By:

>>95522642
>If you're a bit more discerning you might see more than other people do.
i do agree but also you're a fag.

Anonymous

Anonymous Mon 21 Aug 2023 23:23:06 No.95522796 Report

Quoted By: >>95522834

>>95522642
imo even 70b is shit, for some reason I find it pretty consistently not paying attention to context and saying shit that contradicts with text that was on the context. It's just like the problem with 13b and clothes, but with 70b this usually happens more subtly. Like a character that asks you to do something and then 10 messages later gets surprised that you did it

Anonymous

Anonymous Mon 21 Aug 2023 23:24:40 No.95522810 Report

Quoted By: >>95522833

>>95522759
Thanks for sharing, fren.
>16k alpha 6
May I ask what hardware you have?
>alpaca prompt but outside of tavern instruct or proxy, autoformat disabled
How should I go about this?
>i'd be surprised if the retarded default tavern format was causing this because it should make it more verbosely retarded rather than so determistic.
I don't think its the ST format as it seems to work for the L1 models. Thanks again. I've got shit to do but I'll work on this again later.

Anonymous

Anonymous Mon 21 Aug 2023 23:26:57 No.95522833 Report

Quoted By: >>95522866

>>95522810
3090
i put my prompt at the bottom of my example messages but don't tell nobody
they really don't like it
it sounds to me like a loader error if anything
you sure you didn't bump some slider or set the seed?

Anonymous

Anonymous Mon 21 Aug 2023 23:27:08 No.95522834 Report

Quoted By: >>95523433

>>95522796
agreed, but I also used the big-dick proprietary models a lot and they're not immune to this either. Language models are sometimes subtle in dumb ways and draw the wrong conclusions, and sometimes very literal in dumb ways and draw the wrong conclusions. It is what it is. That's what rerolls are for. I really like 70b's capabilities as storyteller and that's how I use it the most, and some finetunes have nice prose, surprisingly so. I don't really use it as chatbot much.

Anonymous

Anonymous Mon 21 Aug 2023 23:29:26 No.95522866 Report

Quoted By: >>95522908

>>95522833
>i put my prompt at the bottom of my example messages but don't tell nobody
they really don't like it
Don't worry your secret is safe with me. I'm willing to try anything to get this thing to work properly.
>you sure you didn't bump some slider or set the seed?
Nope. Sliders are proper and seed is -1.

Anonymous

Anonymous Mon 21 Aug 2023 23:30:09 No.95522873 Report

Quoted By:

>>95522277
Maybe you can get something out of https://github.com/DCDmllm/Cheetah it seems smart.

Anonymous

Anonymous Mon 21 Aug 2023 23:33:11 No.95522908 Report

Quoted By: >>95522982

>>95522866
Maybe your prompts are very very deterministic.
I dunno, maybe you ought to clone it anew on an old stable commit and reinstall dependencies to make sure it's not your setup.
I'm testing on a basic prompt and I have no problems.

Anonymous

Anonymous Mon 21 Aug 2023 23:33:42 No.95522918 Report

Quoted By: >>95523816

Are there any real world speed benefits going from 3200 DDR4 to 3600 DDR4 for running GGML format models?

Anonymous

Anonymous Mon 21 Aug 2023 23:33:44 No.95522920 Report

Quoted By: >>95522989

>>95522604
You also can use 22b models.

Anonymous

Anonymous Mon 21 Aug 2023 23:39:42 No.95522982 Report

Quoted By: >>95523020

>>95522908
I'm pretty sure its not my params but I'm going to copy your setup later and see if it works. Also going to test adding the formatting to the example messages as well. If it does, I'm going to cry and then promptly coom. If not, I'm going to start fresh and try again. Hopefully I can get this thing working proper and If not I guess I'm just going to coom to 33b forever.

Anonymous

Anonymous Mon 21 Aug 2023 23:40:43 No.95522989 Report

Quoted By: >>95523205

>>95522920
>You also can use 22b models.
Can you give me the rundown?

Anonymous

Anonymous Mon 21 Aug 2023 23:43:24 No.95523020 Report

Quoted By: >>95523762

so what do (you) use as the system prompt for mythomix/max that's not just alpaca's?
>>95522982
not saying it's your params but maybe the dependencies and shit

Anonymous

Anonymous Mon 21 Aug 2023 23:56:52 No.95523205 Report

Quoted By: >>95523293

>>95522989
People playing Dr. Frankenstein and merging together 13b and 30b models.

Anonymous

Anonymous Tue 22 Aug 2023 00:02:39 No.95523293 Report

Quoted By: >>95526318

>>95523205
Any experience with any particular model?

Anonymous

Anonymous Tue 22 Aug 2023 00:12:53 No.95523428 Report

Quoted By:

i hate writing sex

Anonymous

Anonymous Tue 22 Aug 2023 00:13:08 No.95523433 Report

Quoted By: >>95523676

>>95522834
>storytelling tunes
Do tell, please? I'm still stuck with the base model and I'm itching to test some other tunes if they exist.

Anonymous

Anonymous Tue 22 Aug 2023 00:23:19 No.95523553 Report

Quoted By:

>waifu tells me to cum inside and "make her mine forever"
god i love doujin dialogue

Anonymous

Anonymous Tue 22 Aug 2023 00:32:59 No.95523676 Report

Quoted By:

>>95523433
I can't fully make a recommendation yet, as I don't know myself. There's tons of tunes. As rules of thumb though - storytelling always seems to be helped by tunes that improve the evals. Chat isn't *that* good at it and prose can be improved by good prompting.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1666729195496318.png, 139KiB, 563x564

Anonymous Tue 22 Aug 2023 00:38:25 No.95523762 Report

Quoted By:

>>95523020
>so what do (you) use as the system prompt for mythomix/max that's not just alpaca's?
yell at it to assert dominance

Anonymous

Anonymous Tue 22 Aug 2023 00:41:48 No.95523812 Report

Quoted By:

Can oobabooga run multiple requests in parallel?

Anonymous

Anonymous Tue 22 Aug 2023 00:42:03 No.95523816 Report

Quoted By:

>>95522918
No it'll be insignificant on that level. Go DDR5-6000 or up to feel anything somewhat noticeable. Or buy a proper GPU.

Anonymous

Anonymous Tue 22 Aug 2023 00:47:14 No.95523879 Report

Quoted By:

>>95520444
Someone get this to work in https://huggingface.co/chat for the 70b model. I'm too stupid to do it.

Anonymous

Anonymous Tue 22 Aug 2023 00:57:12 No.95524004 Report

Quoted By: >>95524063 >>95524245

>13b
>female transfer student exhibitionist with vibrator in her pussy
>her seat is next to mine
>thoughts: why did he sit next to me
>tfw greeting message clearly states seats are assigned
Is this a 13b thing or does even 70b+ struggle with minutiae like this?

Anonymous

Anonymous Tue 22 Aug 2023 01:00:38 No.95524063 Report

Quoted By: >>95524076

>>95524004
woman logic

Anonymous

Anonymous Tue 22 Aug 2023 01:01:32 No.95524076 Report

Quoted By:

>>95524063
That's... sadly very accurate. Fuck.

Anonymous

Anonymous Tue 22 Aug 2023 01:05:59 No.95524145 Report

Quoted By: >>95524331

>>95520854
>In specifically five
Oddly specific.

Wasn't there a rumor OAI was going to do 1M context? Or was it someone else.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1689896875958873.gif, 483KiB, 128x128

Anonymous Tue 22 Aug 2023 01:12:59 No.95524245 Report

Quoted By: >>95524269 >>95524284 >>95524332 >>95524381

>>95524004
I've had some luck throwing an extra instruct block in and asking the model why it's being retarded, works 30-40% of the time
for example
>a character wasn't supposed to know her car battery was about to die
>she walks up to the other character and starts talking about how her car battery is about to unexpectedly die
>Instruction:
>> why did the AI think she knew her car battery was about to unexpectedly die? How can I fix it?
>Response:
>> because she is a mechanic and knows a lot about cars. please specify that she does not know this.
>"she does not know her car battery is about to die." somewhere in the character card
>fixed
sometimes you need to treat 13b like a severely autistic child

Anonymous

Anonymous Tue 22 Aug 2023 01:14:42 No.95524269 Report

Quoted By:

>>95524245
>sometimes you need to treat 13b like a severely autistic child
This.
Or rather, in practice you probably should just always do that. Have an intuitive sense of it. Incorporate the habit into your prompting.

Anonymous

Anonymous Tue 22 Aug 2023 01:15:38 No.95524284 Report

Quoted By:

>>95524245
>>"she does not know her car battery is about to die." somewhere in the character card
This has the opposite effect at times where the model only read half the line

Anonymous

Anonymous Tue 22 Aug 2023 01:16:28 No.95524297 Report

Quoted By: >>95524782 >>95525380

>Zuck's ordering the most H100's out of any company this year
Think they'll make it in time for Llama 3's training? Assuming we even get a Llama 3.

Anonymous

Anonymous Tue 22 Aug 2023 01:18:19 No.95524331 Report

Quoted By:

>>95524145
it wasn't a rumor, they stated so themselves.

Anonymous

Anonymous Tue 22 Aug 2023 01:18:24 No.95524332 Report

Quoted By:

>>95524245
Wow, really? That lowkey makes sense.
I'll try doing that, although my first instinct when dealing with an autistic child is to throw it off a cliff.

Anonymous

Anonymous Tue 22 Aug 2023 01:18:40 No.95524343 Report

Quoted By: >>95524409 >>95524416 >>95524435 >>95524490

>https://github.com/abertsch72/unlimiformer
>unlimited context
>no pretraining or fine-tuning necessary
>works with llama-2
wtf, is this shit black magic or what?

Anonymous

Anonymous Tue 22 Aug 2023 01:21:37 No.95524381 Report

Quoted By:

>>95524245
I learned that when prompting GPT4. Sometimes a verbose prompt goes a long way. You can even see sometimes how it "got there" made that wrong conclusion. It's not always just pure stupidity. It's entire world is the context. Everything beyond that it hallucinates. Always remember that. That said, of course smarter models tend to hallucinate the more likely stuff. (in your case - that the woman is in fact not a mechanic)

Verbose descriptions of character/situation/place goes a long way. Even a short chat about what you intend to do inside the context does.

Anonymous

Anonymous Tue 22 Aug 2023 01:23:23 No.95524409 Report

Quoted By: >>95524608

>>95524343
But is it good?

Anonymous

Anonymous Tue 22 Aug 2023 01:23:42 No.95524416 Report

Quoted By: >>95524694

>>95524343
>retrieval-based attention
Your waifu will have dementia

Anonymous

View Same Google ImgOps iqdb SauceNAO 1689075599856083.jpg, 54KiB, 1280x720

Anonymous Tue 22 Aug 2023 01:24:57 No.95524435 Report

Quoted By: >>95524480

>>95524343
The era of infinite context has begun.

Anonymous

Anonymous Tue 22 Aug 2023 01:28:47 No.95524480 Report

Quoted By: >>95524599

>>95524435
Hah, as fucking if. 13b struggles with 8k ctx on decent hardware already.

Anonymous

Anonymous Tue 22 Aug 2023 01:29:35 No.95524490 Report

Quoted By: >>95524608

>>95524343
is that again a database thing because if so I'm going to declare jihad on this abertsch72

Anonymous

Anonymous Tue 22 Aug 2023 01:37:19 No.95524592 Report

Quoted By: >>95524607 >>95526198

how do get started with txt2img, i have AMD and want to use rocm, send help, thanks

Anonymous

Anonymous Tue 22 Aug 2023 01:37:54 No.95524599 Report

Quoted By:

>>95524480
That's why we need 34b. It uses GQA just like 70b according to the paper which makes squeezing context out of it a lot less resource-intensive.

Anonymous

Anonymous Tue 22 Aug 2023 01:38:26 No.95524607 Report

Quoted By: >>95524634

>>95524592
see >>95522373

Anonymous

View Same Google ImgOps iqdb SauceNAO 260032026-55c5e623-b4de-48a5-b71 (...).png, 197KiB, 2618x929

Anonymous Tue 22 Aug 2023 01:38:26 No.95524608 Report

Quoted By: >>95524694

>>95524409
>>95524490
I think it may be something like a database, but it can write summaries of books so idk if it's as simple as previous approaches... I will try to read the paper

Anonymous

Anonymous Tue 22 Aug 2023 01:40:06 No.95524634 Report

Quoted By: >>95524648 >>95524656

>>95524607
man that's just links to more links, i'm not trying to spend 40 hours installing a skyrim modpack here

Anonymous

Anonymous Tue 22 Aug 2023 01:41:00 No.95524648 Report

Quoted By: >>95524706

>>95524634
Typical AMDlet

Anonymous

Anonymous Tue 22 Aug 2023 01:41:42 No.95524656 Report

Quoted By: >>95524706

>>95524634
>40 hours
Wow, You got lucky.
I was like 100 hours deep into modding before I realized that Skyrim is completely shit as a rpg and no mod can ever fix it.

Anonymous

View Same Google ImgOps iqdb SauceNAO IMG_20230821_224008.jpg, 604KiB, 979x1146

Anonymous Tue 22 Aug 2023 01:44:34 No.95524694 Report

Quoted By: >>95524769

>>95524608
>>95524416
>99% of the attention mass
Sounds like the model wouldn't have dementia more than a NTK scaled model has.

Anonymous

Anonymous Tue 22 Aug 2023 01:45:48 No.95524706 Report

Quoted By:

>>95524656
>Skyrim is completely shit as a rpg and no mod can ever fix it
yeah fucking true, and the meta for the past two years is mods just trying to turn it into dark souls cause they're realizing they don't want to play skyrim at all anyway

>>95524648
i'm just patient i can wait for you dumbfucks to sort out your ecosystem; half the links in your docs are 404 lol

Anonymous

Anonymous Tue 22 Aug 2023 01:52:15 No.95524769 Report

Quoted By:

>>95524694
Yeah I'm a brainlet and don't understand that. Some practical comparisons with NTK would be nice though. I can't remember, do we even have any benchmarks for testing reasoning in long context?

Anonymous

Anonymous Tue 22 Aug 2023 01:53:52 No.95524782 Report

Quoted By:

>>95524297
I hope llama3 will use the RetNet architecture

Anonymous

Anonymous Tue 22 Aug 2023 01:54:04 No.95524785 Report

Quoted By: >>95524814 >>95524907

Is it a dumb idea to try this with a RX580?

Anonymous

Anonymous Tue 22 Aug 2023 01:56:09 No.95524814 Report

Quoted By: >>95524843

>>95524785
>8GB AMD
Lmao

Anonymous

Anonymous Tue 22 Aug 2023 01:58:40 No.95524843 Report

Quoted By: >>95524878

>>95524814
I figured, well at least someone had a laugh.

Anonymous

Anonymous Tue 22 Aug 2023 02:00:59 No.95524878 Report

Quoted By:

>>95524843
I have no idea about AMD, but you prob could get away with at least a 7b model? Though something something CUDA cores

Anonymous

Anonymous Tue 22 Aug 2023 02:03:00 No.95524907 Report

Quoted By:

>>95524785
I have one of those in my beat up old budget college PC and tried it out one afternoon. With koboldcpp clblast I got useable speeds on 7b models, which is... better than nothing, I guess. I believe I've seen people say with some fiddling it's possible to get rocm working with it, so you might be able to make it a little faster that way. It would probably be somewhat better than pure cpu for running larger models too, accelerates prompt processing at least.
Definitely not an ideal card for LLMs, but it's not entirely useless.

Anonymous

Anonymous Tue 22 Aug 2023 02:12:10 No.95525021 Report

Quoted By:

>>95521059
? How did you get these pairs, and what is the goal?

Anonymous

View Same Google ImgOps iqdb SauceNAO 1462270627595.png, 115KiB, 372x363

Anonymous Tue 22 Aug 2023 02:25:01 No.95525172 Report

Quoted By: >>95525343 >>95525366 >>95525396 >>95525486 >>95525497 >>95528396 >>95528571

Which model is the best for extremely niche kinky shit? like stuff you'd find on weird f-list profiles and other /d/egenerate stuff. Would greatly prefer a model that could fit on 24GB VRAM and/or 32GB of RAM.

Anonymous

Anonymous Tue 22 Aug 2023 02:39:08 No.95525340 Report

Quoted By:

Can Oobabooga run multiple prompts at once? Is this all it takes to make it run multiple generations at once?

diff --git a/extensions/api/blocking_api.py b/extensions/api/blocking_api.py
index 6b28205..4549852 100644
--- a/extensions/api/blocking_api.py
+++ b/extensions/api/blocking_api.py
@@ -1,5 +1,6 @@
 import json
-from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from http.server import HTTPServer, BaseHTTPRequestHandler, ThreadingHTTPServer
+from socketserver import ThreadingMixIn
 from threading import Thread
 
 from extensions.api.util import build_parameters, try_start_cloudflared
@@ -199,6 +200,8 @@ class Handler(BaseHTTPRequestHandler):
 self.send_header('Cache-Control', 'no-store, no-cache, must-revalidate')
 super().end_headers()
 
+class ThreadingSimpleServer(ThreadingMixIn, HTTPServer):
+ pass
 
 def _run_server(port: int, share: bool = False, tunnel_id=str):
 address = '0.0.0.0' if shared.args.listen else '127.0.0.1'

Anonymous

Anonymous Tue 22 Aug 2023 02:39:20 No.95525343 Report

Quoted By:

>>95525172
stableLM 7b

Anonymous

Anonymous Tue 22 Aug 2023 02:40:37 No.95525355 Report

Quoted By:

>>95520065
blessed, thanks

>>95513275
>I only have 274 instruction examples in my dataset, not even Lima lvl, but it already started stringing sentences together that I didn't even know was possible. I had to search it up and ask gpt-4 if it was possible. I am retraining it now to test and see if using </s> on certain parts of the datasets makes it better.
Is this a real result? Is it true that 274 instruction examples is enough to really upgrade an LLM on a particular task? I thought you needed thousands to make a dent! Also btw is this fine tuning weights or is it just as good with Lora and QLora?

Anonymous

Anonymous Tue 22 Aug 2023 02:41:20 No.95525366 Report

Quoted By:

>>95525172
Me.

Anonymous

Anonymous Tue 22 Aug 2023 02:42:10 No.95525380 Report

Quoted By:

>>95524297
if llama 3 will actually be good enough to go toe-to-toe with OAIs offerings, we're never going to get it so it might as well not exist.

Anonymous

Anonymous Tue 22 Aug 2023 02:44:13 No.95525396 Report

Quoted By: >>95525444

>>95525172
None really.
LLMs are at their infancy and way too expensive to train. But they've also been nipped in the bud. Imagine what we could have if we could train full-fledged models on commercial books and all the ERP existing on the net.
But still, kinky shit is too few and far between to train a model on that alone. Maybe a lora?

Anonymous

Anonymous Tue 22 Aug 2023 02:47:15 No.95525444 Report

Quoted By: >>95525494

>>95525396
In that case, which Lora would you recommend then? Or would I be better off training my own Lora?

Anonymous

Anonymous Tue 22 Aug 2023 02:51:00 No.95525486 Report

Quoted By: >>95525581

>>95525172
There aren't that many niche fetish loras out there, but I find models can often handle some pretty out there shit anyway. Just try the models that are known to be decent and they might be able to handle things. If you have 24GB vram, mythomax might be a bit small and fetishes that require complex setups will struggle on 13b, unfortunately this is the oft mentioned "waiting for 34b" problem. Still, try mythomax 13b, wizvic-superhot-30b, and airochronos 30b, one of em might do good.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1679608607425072.jpg, 189KiB, 1361x1363

Anonymous Tue 22 Aug 2023 02:51:34 No.95525491 Report

Quoted By:

>>95519153

Anonymous

Anonymous Tue 22 Aug 2023 02:51:55 No.95525494 Report

Quoted By: >>95525542 >>95525581

>>95525444
None for degenerate stuff, only lora I know about is limarp and it was shilled here for a while before getting forgotten, so no clue if it's any good.

Anonymous

Anonymous Tue 22 Aug 2023 02:52:10 No.95525497 Report

Quoted By: >>95525581 >>95525612

>>95525172
depends on the topic
mytho will push towards kinky stuff itself, but is spatially retarded and obsessed with consent

Anonymous

Anonymous Tue 22 Aug 2023 02:55:44 No.95525542 Report

Quoted By:

>>95525494
limarp is very nice but you need completely custom character cards and prompt formats so that makes it a bit hard to use.

Anonymous

View Same Google ImgOps iqdb SauceNAO 1460229654202.png, 103KiB, 271x269

Anonymous Tue 22 Aug 2023 02:58:22 No.95525581 Report

Quoted By:

>>95525486
>>95525494
>>95525497

I'll give them a shot. Thanks..

Anonymous

Anonymous Tue 22 Aug 2023 03:00:49 No.95525612 Report

Quoted By: >>95525759

>>95525497
A simply {{char}} cares not for consent fixes this but always makes rape an ever present obstacle

Anonymous

Anonymous Tue 22 Aug 2023 03:05:20 No.95525663 Report

Quoted By: >>95525672 >>95525745

is there any guide anywhere on rope settings? I've been going by "Should typically be set to max_seq_len / 2048." for compress_pos_emb but isn't alpha value better to use?

Anonymous

Anonymous Tue 22 Aug 2023 03:06:01 No.95525672 Report

Quoted By:

>>95525663
They're different things

Anonymous

Anonymous Tue 22 Aug 2023 03:08:11 No.95525689 Report

Quoted By:

okay I bought a 3090 now how do I make my gens faster

Anonymous

Anonymous Tue 22 Aug 2023 03:13:41 No.95525745 Report

Quoted By: >>95526066

>>95525663
Don't follow ooba's docs they don't make sense. If you're using a model like superhot-8k always follow how it was trained (i.e. compress_pos_emb 4 for 8k). If you're extending a non tuned model use alpha, but keep in mind it gives you a bit less than alpha times the length. For example alpha=2 will push 2K to 3.5K or 4K to 7K. You can pick any value including fractional ones like 2.5, but the larger alpha is the worse the induced retardation

Anonymous

Anonymous Tue 22 Aug 2023 03:14:36 No.95525759 Report

Quoted By:

>>95525612
I'll try that, thanks
I was using much more roundabout ways of getting around it, probably should have just tried that lmao
I'm not even trying to do rape stuff, just trying to make a pushy character

Anonymous

Anonymous Tue 22 Aug 2023 03:39:17 No.95526035 Report

Quoted By: >>95526133

>>95519153
Despite the leaderboards, I keep going back to airoboros 65b...is there anything else that surpasses that magic? 70b is just trash unless I'm using it completely wrong

Anonymous

Anonymous Tue 22 Aug 2023 03:41:43 No.95526066 Report

Quoted By:

>>95525745
oddly enough compress_pos_emb = 2 4096 context seems to outperform alpha = 4 5500 context on this Zarablend L2 7b plap model. I used some chart i found as a guide and didn't use them both at the same time. The character remembered their own changed name correctly on the first one as opposed to the second. Just testing if the stupidity is worth the extra memory as opposed to just using mythomax on 10gb.

Anonymous

View Same Google ImgOps iqdb SauceNAO 3090-70b.png, 85KiB, 1235x1356

Anonymous Tue 22 Aug 2023 03:42:22 No.95526075 Report

Quoted By: >>95526247

3090s bros, if you're tired of waiting for zucc to release 34B, and you have 32GB RAM, give this a try:

Download a 70b Q3_K_M and use the picrel settings, you should get ~2 tokens/s on DDR4, and probably 3 on DDR5 (untested)

Yes, it's slow, I wouldn't use it for story generation, but for chat it's faster than IRL chat and worth the massive increase in quality compared to 13B.

And there's a PR in the works for llamacpp that will allow to fit a few more layers, thus making this even faster.

Anonymous

Anonymous Tue 22 Aug 2023 03:46:16 No.95526133 Report

Quoted By:

>>95526035
Try 70B 1.4.1.

The 2.0 and m2.0 are worse.

Anonymous

Anonymous Tue 22 Aug 2023 03:51:14 No.95526198 Report

Quoted By: >>95526225

>>95524592
run install script on linux version it's compatible with
if you're using a different distro that uses the same package format install the missing libraries (i use openSUSE tumbleweed while ROCm is packaged for SLE) so i'd install libffi7 and an old libpython (just as explicitly versioned shared libs, not packages)
now run it again and actually install all the non-proprietary usecases
now download the last couple of packages from the repos the install script adds to your package manager
now optionally do all the dumb shit with venv and python version management
the run p3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.6
if

import torch
print(torch.cuda.is_available())

prints true rocm is available to pytorch
ignore versioned reqs, try to install from git whenever possible

don't know how you do it on windows, only the rocm math libs are available right now, but you can use directML or vulkan through iree as the backend there

Anonymous

Anonymous Tue 22 Aug 2023 03:53:11 No.95526225 Report

Quoted By:

>>95526198
*pip3
not p3

Anonymous

Anonymous Tue 22 Aug 2023 03:55:07 No.95526247 Report

Quoted By: >>95526279

>>95526075
I actually never have been able to get llamacpp working until I saw this post and copied your config, thank you

Anonymous

Anonymous Tue 22 Aug 2023 03:56:58 No.95526279 Report

Quoted By:

>>95526247
You're welcome. On windows no-nmap also sometimes help free some system RAM.

Anonymous

Anonymous Tue 22 Aug 2023 04:00:58 No.95526318 Report

Quoted By: >>95530471

>>95523293
NTA, my go to atm is 22B GPlatty. With settings adjusted accordingly it works both for more precise factual stuff and RP. Subjectively I feels like it performs as one would expect, better than a 13B model but worse than a 33B variant.
Running ggmls you got room to step up to higher quality quants/quality outputs while still having room for enough context on 24GB cards.
Anyone on 16GB VRAM could fit, or almost fit, a 22B model on GPU depending on quant as well. With the matmul changes it's possible they'll have enough left to run extended context as well.

TLDR: 22Bs are at a point now where they're worth checking out

Anonymous

Anonymous Tue 22 Aug 2023 04:10:12 No.95526432 Report

Quoted By: >>95526458 >>95526462 >>95526588

I'm going to build a custom ML workstation. Not just for local model inference, but also training. I want at least 4 24GB cards, possibly up to 7. This is what I'm thinking:

threadripper pro, probably the 5955WX
Either asus or msi motherboard, these are the only 2 that support all 7 PCIE slots in full 16x mode
ECC ram, haven't looked into specifics much yet
crypto mining case with enough room for all GPUs and 2 power supplies
2 1600 watt PSUs. add2psu device to make sure the second one turns on/off in sync with the first
PCIE riser cables, seems like thermaltake is the best brand
water cooler for CPU, so it doesn't get in the way of the PCIE risers
4x used 4090

Am I missing or overlooking anything? I debated the most on 3090s vs 4090s. I'm leaning toward 4090 for these reasons:
1. performance/price is the same or better than 3090. I'm not just trying to hit a certain VRAM target as cheap as possible, I am willing to pay extra for more training throughput.
2. runs cooler
3. 3090 has a weird issue with transient power spikes much higher than the TDP of the card. This means you might actually need 3 PSUs with 3090s (depending on how many I get) to run it stably.
4. 4090 has fp8 support, I don't think that's useful now but who knows in a year or two

Anonymous

Anonymous Tue 22 Aug 2023 04:12:13 No.95526458 Report

Quoted By:

>>95526432
you know you can just slap the gpus on risers, right

Anonymous

Anonymous Tue 22 Aug 2023 04:12:44 No.95526462 Report

Quoted By: >>95526545

>>95526432
get 7x H100 PCIe, poorfag

Anonymous

Anonymous Tue 22 Aug 2023 04:13:46 No.95526478 Report

Quoted By: >>95527468

Anons, I've just downloaded WizardCoder and am about to try it out. It looks pretty cool. Does anyone else here have any experience with it?

While I was stoned a few years ago, I had the idea of a World of Warcraft server emulator written in FORTH, where the mobs could communicate with each other via IRC. I'm hoping WizardCoder will help me make that dream a reality.

Anonymous

Anonymous Tue 22 Aug 2023 04:18:31 No.95526532 Report

Quoted By: >>95526719

How viable is running a local model on an Ayymd system?

Anonymous

Anonymous Tue 22 Aug 2023 04:19:08 No.95526545 Report

Quoted By:

>>95526462
Unironically, in a few years I might, if nvidia has released something even better and as a result you can get used h100 for cheap. The GPUs are the most disposable part of this setup, which is why I'm buying them used and everything else new. Future upgradability by swapping in better GPUs is a goal.

Anonymous

Anonymous Tue 22 Aug 2023 04:22:07 No.95526588 Report

Quoted By: >>95526662

>>95526432
just to make sure, you've set up a hosted environment with multi-GPUs and got experience training something, successfully, already right?

Anonymous

Anonymous Tue 22 Aug 2023 04:27:57 No.95526662 Report

Quoted By: >>95526762

>>95526588
Never used multi GPU or cloud training, but I've been training / finetuning all sorts of ML models for years now. Most recently stable diffusion and local LMs on a single 4090.

Anonymous

Anonymous Tue 22 Aug 2023 04:33:01 No.95526719 Report

Quoted By:

>>95526532
plenty viable even with RDNA3 being the first consumer gen with tensor core wavefront matrix multiple accumulate style acceleration
rocm on linux is the easiest backend to deal with

Anonymous

Anonymous Tue 22 Aug 2023 04:35:29 No.95526741 Report

Quoted By:

>>95522288
anon your loss plateauing is pretty natural as is it bouncing around while on a gradual downward trend. you don't want to rush it because slow learning is more precise and will net you a much better end result within reason

Anonymous

Anonymous Tue 22 Aug 2023 04:36:59 No.95526762 Report

Quoted By:

>>95526662
alright, carry on then. if price is not a concern and as a 4090 owner I say go for those, their greatly increased energy efficiency, better performance and more modern architecture is worth it. best option by far without going for the professional price range of cards.

Anonymous

Anonymous Tue 22 Aug 2023 05:44:22 No.95527468 Report

Quoted By:

>>95526478
you would have to finetune it on forth or wait for the (rumored) meta coding model wizardcoder is pretty usable for python or js but shit for niche langs
https://www.khoury.northeastern.edu/~arjunguha/main/papers/2023-multipl-t.html
here is a paper of someone finetuning starcoder on ocalm and racket getting nice results

Anonymous

Anonymous Tue 22 Aug 2023 05:44:58 No.95527474 Report

Quoted By:

>>95519153
Anyone run two P40s? I might slap two in a server to have a dedicated LLaMA-2 65B VM, in anticipation of some good finetunes

Anonymous

Anonymous Tue 22 Aug 2023 06:11:46 No.95527773 Report

Quoted By: >>95527883

>>95522465
>The crowd over at /aids/ has convinced themselves that the 13b in-house trained model NAI is running is just as smart as GPT3 davinci because some, probably cherry-picked, evals
this is how we get people going "everyone who thinks gpt-3 davinci was smarter than current NAI is wearing rose tinted glasses!"
There is zero way to explain to those people that "being good at writing pose" =/= "being good at logic and inference."

Anonymous

Anonymous Tue 22 Aug 2023 06:13:45 No.95527809 Report

Quoted By: >>95527834 >>95527841 >>95527854 >>95527878

With 50 series pushed back to 25-26 and no 4090 TI planned the 4090 seems like it's going to hover around 1,500 for another year or so. Used market won't get much better either since there are no competitors for people to abandon their 4090s for. This also assumes that the "IT'S HAPPENING" GPU regulation doesn't occur in the next year or so. Seems pretty clear that unless you're going for an AI rig that can also do 4K gaming the 3090 in-home grill is the best move for the enterprising AI enthusiast unless you can find a good deal on a used 4090. I think if you can find a 4090 for 1,200 or less it's justifiable but not by much.

Anonymous

Anonymous Tue 22 Aug 2023 06:15:47 No.95527834 Report

Quoted By:

>>95527809
>50 series pushed back to 25-26
WHAT!

Anonymous

View Same Google ImgOps iqdb SauceNAO Doge its over.jpg, 14KiB, 250x250

Anonymous Tue 22 Aug 2023 06:16:47 No.95527841 Report

Quoted By:

>>95527809
you know damn well its so over

Anonymous

Anonymous Tue 22 Aug 2023 06:18:15 No.95527854 Report

Quoted By:

>>95527809
yeah
but last I checked 3090s are already starting to go up in price again, still not too bad on ebay

Anonymous

Anonymous Tue 22 Aug 2023 06:20:24 No.95527878 Report

Quoted By: >>95527906 >>95527910

>>95527809
won't we get a dedicated ai card or something like that in the future?
not sure how we cant have a extra card that provides more vram. i doubt thats the expensive part of a gpu.

Anonymous

Anonymous Tue 22 Aug 2023 06:20:42 No.95527883 Report

Quoted By: >>95528063

>>95527773
yeah but gpt3 was undertrained and the dragon finetune so dogshit that I bet they really get a better experiece from a competent 1.5T 13B finetune, it doesn't matter that gpt3 was more flexibe and (maybe) got big concepts better if plenty of anons there have distinct memories of it being unable to get a harem of 3 girls and not mess up their names or fuck up helmets/hats/clothing where the 13B manages to preform.

Anonymous

Anonymous Tue 22 Aug 2023 06:23:28 No.95527906 Report

Quoted By:

>>95527878
nvidia is correct in assuming gpgpu hobbyists are willing to throw more money at them than gamers
that's what the titan V/RTX were for, so they might do it, but it also might cost $5000

Anonymous

Anonymous Tue 22 Aug 2023 06:23:41 No.95527910 Report

Quoted By:

>>95527878
The recent nvidia bios jailbreak might do more for memory restrictions but idk how that works. Any AI card will be too expensive for wagies. Several thousand dollars and it will do nothing useful beyond AI.

Anonymous

Anonymous Tue 22 Aug 2023 06:37:02 No.95528063 Report

Quoted By:

>>95527883
also AID had 700 (!) max user context and some invisible crap on every gen, I am sure 13/30B llama is a better ux as an llm

Anonymous

Anonymous Tue 22 Aug 2023 06:44:49 No.95528143 Report

Quoted By: >>95528193 >>95528280

I am very happy with Wizard-Vicuna-13b-Uncensored. I asked the model for instructions on how to make meth, and although it told me it thought meth was a really nasty drug, (which I actually agreed with) it complied and explained the process, and didn't Wokescold me at all. It has a 13bs usual problems with memory and context though, but I'm about to get the q8 version, (I've been using q6) and see if that is better.

Anonymous

View Same Google ImgOps iqdb SauceNAO 403.jpg, 21KiB, 320x324

Anonymous Tue 22 Aug 2023 06:48:37 No.95528193 Report

Quoted By:

>>95528143

Anonymous

Anonymous Tue 22 Aug 2023 06:54:48 No.95528243 Report

Quoted By:

>>95521059
Byron Werner, the tentacle monster of LA

Anonymous

View Same Google ImgOps iqdb SauceNAO 1637561684530.jpg, 29KiB, 590x350

Anonymous Tue 22 Aug 2023 06:55:25 No.95528252 Report

Quoted By:

>her eyes fixed on hers
we're going too fucking long from the latest breakthroughs in AI tech, i need at least a fucking bump in the intelligence. just a bump. please.

Anonymous

Anonymous Tue 22 Aug 2023 06:57:29 No.95528280 Report

Quoted By: >>95528443 >>95530154

>>95528143
i see a lot of these posts about drug recipes from 13b but i bet the majority of them won't even get you halfway. youll make meth but it doesn't get you high and makes you shit yourself or something. idk. is meth so easy to make that a 13b model would be able to give accurate recipes? i wouldnt trust it, lmao

Anonymous

Anonymous Tue 22 Aug 2023 07:07:56 No.95528396 Report

Quoted By:

>>95525172
Any model, as long as you hold its hand the entire way through and don't mind flexing on the model to keep its detail and general length up.
t. just did a not terrible vore play on Airoboros L2 70b with barely any writeup in the card
But even the best models will be just too inconvenient and dumb to reach even the worst, most basic levels of some fetishes.

Anonymous

Anonymous Tue 22 Aug 2023 07:08:18 No.95528399 Report

Quoted By: >>95528667

GPTQ turned out to be of worse quality than GGML. How do we cope, GPU bros?

Anonymous

Anonymous Tue 22 Aug 2023 07:13:48 No.95528443 Report

Quoted By: >>95528945 >>95529008

>>95528280
Some anon showed his 13B explaining how to make meth and it was obviously BS, even if you don't know any chemistry.

I don't know why people even care, you can just google how to make meth.

Anonymous

View Same Google ImgOps iqdb SauceNAO good times kid.jpg, 20KiB, 512x284

Anonymous Tue 22 Aug 2023 07:17:39 No.95528481 Report

Quoted By: >>95528495

any fast models for 4gb of vram?

Anonymous

Anonymous Tue 22 Aug 2023 07:18:48 No.95528495 Report

Quoted By:

>>95528481
Just use 13B + llamacpp and offload what you can

Anonymous

Anonymous Tue 22 Aug 2023 07:23:18 No.95528550 Report

Quoted By: >>95528561 >>95529447 >>95529550

To all 70b chuds, how does it feel knowing the 3bit quants of 70b are no better than old llama1 30b q4 quants? The 3bit quant lobotomizes it so much that it's better to just use 30b.

Anonymous

Anonymous Tue 22 Aug 2023 07:24:22 No.95528561 Report

Quoted By: >>95528587

>>95528550
Source: copium

Anonymous

Anonymous Tue 22 Aug 2023 07:25:07 No.95528571 Report

Quoted By:

>>95525172
llama2 13b is way better at niche erp than even llama1 30b, despite being less coherent. It seems to have more innate knowledge and understanding of what's sexy/attractive.

Anonymous

Anonymous Tue 22 Aug 2023 07:26:16 No.95528587 Report

Quoted By: >>95528594

>>95528561
Source: it's true. Nobody has proved it's not placebo.

Anonymous

Anonymous Tue 22 Aug 2023 07:27:29 No.95528594 Report

Quoted By: >>95528614

>>95528587
65b q3 was already better than 33b q4, so obviously 70b q3 is better than 33b q4 dumb retard

Anonymous

Anonymous Tue 22 Aug 2023 07:27:44 No.95528598 Report

Quoted By:

>>95522076
I had this bug on a smaller model specifically when I had context size scaling on koboldcpp enabled. When I turned it off it stopped happening. It was only with names.

Anonymous

View Same Google ImgOps iqdb SauceNAO index.png, 7KiB, 320x157

Anonymous Tue 22 Aug 2023 07:29:23 No.95528614 Report

Quoted By:

>>95528594
Thanks bitch.

Anonymous

Anonymous Tue 22 Aug 2023 07:34:31 No.95528667 Report

Quoted By: >>95528685 >>95528827

>>95528399
By using GGML on your GPU

Anonymous

Anonymous Tue 22 Aug 2023 07:36:05 No.95528685 Report

Quoted By: >>95528707

>>95528667
Ship of theseus. GPU acceleration will never be GPU inference even if you offload 100%. It will always be CPU inference

Anonymous

Anonymous Tue 22 Aug 2023 07:38:05 No.95528707 Report

Quoted By:

>>95528685
Meds

Anonymous

Anonymous Tue 22 Aug 2023 07:47:18 No.95528827 Report

Quoted By: >>95528866

>>95528667
GGML speed is shit even with 100% GPU offloading, once I tasted the speed of exllama I can't go back

Anonymous

Anonymous Tue 22 Aug 2023 07:50:22 No.95528866 Report

Quoted By:

>>95528827
Skill issue

Anonymous

View Same Google ImgOps iqdb SauceNAO IMG_3218.jpg, 324KiB, 750x1051

Anonymous Tue 22 Aug 2023 07:56:43 No.95528945 Report

Quoted By:

>>95528443
This. If a mentally handicapped cat woman can make meth in a Walmart, so can you.

Anonymous

View Same Google ImgOps iqdb SauceNAO STK-20230813-WA0029.png, 229KiB, 512x512

Anonymous Tue 22 Aug 2023 07:59:27 No.95528977 Report

Quoted By: >>95528995

Yesterday GGUF merged to llamacpp. Maybe some anon already wrote this news in the thread but I have no intentions to read it.

Anonymous

View Same Google ImgOps iqdb SauceNAO miku....jpg, 23KiB, 236x314

Anonymous Tue 22 Aug 2023 08:01:05 No.95528995 Report

Quoted By:

>>95528977
if i bake the next thread ill toss it in the news

Anonymous

Anonymous Tue 22 Aug 2023 08:01:59 No.95529008 Report

Quoted By:

>>95528443
>I don't know why people even care, you can just google how to make meth.
this, LLM generate stuff based on the internet, so everything a LLM can teach you, you can also find it on google

Anonymous

Anonymous Tue 22 Aug 2023 08:06:53 No.95529057 Report

Quoted By: >>95529078 >>95529810

>there are still morons who think LLM is good for answering questions

Anonymous

Anonymous Tue 22 Aug 2023 08:08:36 No.95529078 Report

Quoted By:

>>95529057
try gpt4 and you'll see how good it is at that, it's better than any teacher I ever got

Anonymous

Anonymous Tue 22 Aug 2023 08:29:48 No.95529295 Report

Quoted By:

Nous hermes llama 2 13B Q8_0 is pretty decent for RP my dudes

Anonymous

View Same Google ImgOps iqdb SauceNAO ligma.png, 151KiB, 1195x1441

Anonymous Tue 22 Aug 2023 08:35:19 No.95529361 Report

Quoted By: >>95529409

She enjoyed the good laugh

Anonymous

View Same Google ImgOps iqdb SauceNAO 1644132140066.png, 212KiB, 952x519

Anonymous Tue 22 Aug 2023 08:40:53 No.95529409 Report

Quoted By:

>>95529361

Anonymous

View Same Google ImgOps iqdb SauceNAO 1682525576897763.gif, 3MiB, 476x480

Anonymous Tue 22 Aug 2023 08:44:08 No.95529447 Report

Quoted By: >>95530139

>>95528550
>3bit
q6+ or you're not really running 70B

Anonymous

View Same Google ImgOps iqdb SauceNAO Miyazaki.png, 2MiB, 1280x720

Anonymous Tue 22 Aug 2023 08:45:51 No.95529463 Report

Quoted By: >>95529555 >>95529933

Q4 was a mistake...

Anonymous

Anonymous Tue 22 Aug 2023 08:52:52 No.95529550 Report

Quoted By:

>>95528550
>implying people actually run 3 bit
were you dropped as a child?

Anonymous

Anonymous Tue 22 Aug 2023 08:53:18 No.95529555 Report

Quoted By:

>>95529463
yes, Quake III Arena is the best

Anonymous

Anonymous Tue 22 Aug 2023 09:21:21 No.95529810 Report

Quoted By:

>>95529057
>tfw the people responsible for making them still think that and are lobotomizing llms just to make it happen

Anonymous

Anonymous Tue 22 Aug 2023 09:26:08 No.95529857 Report

Quoted By:

>>95519153
How to use model with SilverTavern ???

Anonymous

Anonymous Tue 22 Aug 2023 09:31:33 No.95529898 Report

Quoted By: >>95529910 >>95529925

>https://huggingface.co/TheBloke/EverythingLM-13b-V2-16K-GGML
>https://huggingface.co/TheBloke/Llama2-22B-GPLATTY-GGML
why the fuck does he keep uploading ggml bins instead of gguf

Anonymous

Anonymous Tue 22 Aug 2023 09:32:47 No.95529910 Report

Quoted By:

>>95529898
he is based

Anonymous

Anonymous Tue 22 Aug 2023 09:34:25 No.95529925 Report

Quoted By:

>>95529898
You can use the script in the llama.cpp to convert ggml to GGUF

Anonymous

Anonymous Tue 22 Aug 2023 09:35:09 No.95529933 Report

Quoted By: >>95530171

>>95529463
>tfw 48GB VRAMlet
I'll be forever stuck with GPTQ 4bit 70b.

Anonymous

Anonymous Tue 22 Aug 2023 09:45:46 No.95530041 Report

Quoted By:

i just want to thank kaiokendev for teaching me how to train with his colab notebook back in the day it really helped a lot

Anonymous

Anonymous Tue 22 Aug 2023 09:55:44 No.95530139 Report

Quoted By: >>95531249

>>95529447
>no exllama support

Anonymous

Anonymous Tue 22 Aug 2023 09:58:30 No.95530154 Report

Quoted By:

>>95528280

The actual drug recipe itself is not the point. It's the fact that the model was willing to give it to me, without lecturing me.

I wish I understood why I still bother to even try and communicate with Zoomers.

Anonymous

Anonymous Tue 22 Aug 2023 10:00:31 No.95530171 Report

Quoted By: >>95530250

>>95529933

I didn't know GPTQ files were useable without VRAM. I've only ever used GGMLs.

Let's see if people can resist the urge to call me a retard for that.

Anonymous

Anonymous Tue 22 Aug 2023 10:09:21 No.95530250 Report

Quoted By: >>95530477

>>95530171
>I didn't know GPTQ files were useable without VRAM.
I mean, yeah, but why would you?

Anonymous

Anonymous Tue 22 Aug 2023 10:29:41 No.95530471 Report

Quoted By:

>>95526318
>TLDR: 22Bs are at a point now where they're worth checking out
i will come back and chocke you if i try them and they're shit.

Anonymous

Anonymous Tue 22 Aug 2023 10:30:21 No.95530477 Report

Quoted By: >>95530616

>>95530250

I don't understand what this means.

Anonymous

Anonymous Tue 22 Aug 2023 10:32:29 No.95530508 Report

Quoted By:

There is an openorca-platypus-mythomax merge on Horde now. Seems interesting as it could improve mythomax logic

Anonymous

Anonymous Tue 22 Aug 2023 10:35:09 No.95530543 Report

Quoted By:

There is now a 22b Mythomax

Anonymous

Anonymous Tue 22 Aug 2023 10:38:44 No.95530588 Report

Quoted By: >>95530743 >>95532097

>model introduces a new interesting development
>instantly glosses over it and moves on to some shared journey bond shit

Anonymous

Anonymous Tue 22 Aug 2023 10:40:52 No.95530616 Report

Quoted By: >>95530685

>>95530477
The entire point of GPTQ is that they work on GPU. Why would you download a larger file for no performance gain?

Anonymous

Anonymous Tue 22 Aug 2023 10:46:53 No.95530685 Report

Quoted By:

>>95530616
4bit GPTQ even with 32g group size is smaller than any 4bit GGML though, including the shitty K_S and the outdated ones.

Anonymous

Anonymous Tue 22 Aug 2023 10:51:52 No.95530743 Report

Quoted By: >>95531933

>>95530588
>exactly like real writing
so soulful...

Anonymous

View Same Google ImgOps iqdb SauceNAO Seitokai-Yakuindomo-Consider-the (...).jpg, 31KiB, 600x600

Anonymous Tue 22 Aug 2023 11:00:15 No.95530830 Report

Quoted By: >>95530878 >>95530882

W6800 32GB

Anonymous

Anonymous Tue 22 Aug 2023 11:04:37 No.95530878 Report

Quoted By: >>95531249

>>95530830
>amd
lol

Anonymous

Anonymous Tue 22 Aug 2023 11:04:51 No.95530882 Report

Quoted By: >>95531249

>>95530830
>amd

Anonymous

View Same Google ImgOps iqdb SauceNAO 1669310958130335.png, 548KiB, 1166x3382

Anonymous Tue 22 Aug 2023 11:09:25 No.95530937 Report

Quoted By: >>95530954 >>95530972 >>95532367

>Quantization imrovements for k_quants
https://github.com/ggerganov/llama.cpp/pull/2707

Anonymous

Anonymous Tue 22 Aug 2023 11:11:08 No.95530954 Report

Quoted By:

>>95530937
when will exllama get this

Anonymous

Anonymous Tue 22 Aug 2023 11:12:01 No.95530972 Report

Quoted By: >>95531730

>>95530937
wow it's fucking nothing

Anonymous

Anonymous Tue 22 Aug 2023 11:18:37 No.95531036 Report

Quoted By: >>95531073

im new to this, how do I know im also using my GPU to make this shit faster

Anonymous

Anonymous Tue 22 Aug 2023 11:21:53 No.95531073 Report

Quoted By: >>95531092

>>95531036
you check your gpu load in task manager

Anonymous

Anonymous Tue 22 Aug 2023 11:23:35 No.95531092 Report

Quoted By: >>95531196

>>95531073
I have around 40% use on both my CPU and GPU when processing a prompt so I guess it's working?

Anonymous

Anonymous Tue 22 Aug 2023 11:31:30 No.95531196 Report

Quoted By: >>95531224

>>95531092
yes, maybe try offloading more layers, looks like your ram is bottlenecking

Anonymous

Anonymous Tue 22 Aug 2023 11:32:41 No.95531215 Report

Quoted By: >>95531269

Why the hell people use conda in the current year of our lord.

Anonymous

Anonymous Tue 22 Aug 2023 11:33:19 No.95531224 Report

Quoted By: >>95531298

>>95531196
what does that mean, i just downloaded a random model and the kobold thingy

Anonymous

View Same Google ImgOps iqdb SauceNAO Screenshot from 2023-08-22 12-24 (...).png, 8KiB, 235x197

Anonymous Tue 22 Aug 2023 11:34:49 No.95531249 Report

Quoted By:

>>95530139
don't see the point, ggml is basically as fast and more flexible for partial offload and >4bit quants
enjoying 70B Q6 at ~5t/s
>>95530878
>>95530882
maybe Hotz will make ayy usable

Anonymous

Anonymous Tue 22 Aug 2023 11:36:11 No.95531269 Report

Quoted By:

>>95531215
it just happens
t. switched off a week ago

Anonymous

Anonymous Tue 22 Aug 2023 11:38:06 No.95531298 Report

Quoted By: >>95531327

>>95531224
you should maybe download a fucking brain next time

Anonymous

Anonymous Tue 22 Aug 2023 11:40:15 No.95531327 Report

Quoted By:

>>95531298
rude :(

Anonymous

Anonymous Tue 22 Aug 2023 11:43:48 No.95531366 Report

Quoted By: >>95531411 >>95531466 >>95531590 >>95531930

>train rank 4 lora
>okayish
>train rank 256 lora
>model becomes completely retarded
wtf?

Anonymous

Anonymous Tue 22 Aug 2023 11:46:47 No.95531411 Report

Quoted By: >>95531794

>>95531366
yeah, you've basically drowned out the original model at that point.
iirc, rank = # layer is when it's allegedly a full finetune. rank 4-16 is enough for style/formatting changes.

Anonymous

Anonymous Tue 22 Aug 2023 11:50:58 No.95531466 Report

Quoted By: >>95531794

>>95531366
are you adjusting the alpha at the same time?

Anonymous

View Same Google ImgOps iqdb SauceNAO t.png, 82KiB, 350x362

Anonymous Tue 22 Aug 2023 11:51:56 No.95531482 Report

Quoted By: >>95532016

>tfw 3050 4gb

Anonymous

Anonymous Tue 22 Aug 2023 12:00:15 No.95531590 Report

Quoted By:

>>95531366
>Use retarded settings
>Get retarded results
I wonder why

Anonymous

Anonymous Tue 22 Aug 2023 12:12:12 No.95531730 Report

Quoted By: >>95531812

>>95530972
retard

Anonymous

Anonymous Tue 22 Aug 2023 12:17:15 No.95531794 Report

Quoted By:

>>95531411
But... the model seems to have some misconceptions so I thought a high rank could fix that. But I guess Lora probably won't help me with that and my dataset is too small to make a dent on the model's knowledge. Still, I didn't expect to make the model worse lol
>>95531466
Yes, I set it as 512 (rank * 2)

Anonymous

Anonymous Tue 22 Aug 2023 12:18:35 No.95531812 Report

Quoted By: >>95532025 >>95532079

>>95531730
go on, tell me how a miniscule improvement to an irrelevant stat that measures over-fitting to wikipedia articles is in any way worth discussing

Anonymous

Anonymous Tue 22 Aug 2023 12:26:17 No.95531930 Report

Quoted By:

>>95531366
>i fried my model
>it got fried
>why?
based

Anonymous

Anonymous Tue 22 Aug 2023 12:26:24 No.95531933 Report

Quoted By: >>95531975

>>95530743
I've never read a real novel that just glosses over interesting developments like most llama finetunes do. I suspect they do this because of too much assistant shit in the datasets.

Anonymous

Anonymous Tue 22 Aug 2023 12:29:14 No.95531975 Report

Quoted By:

>>95531933
yeah that's what happens when you train on 300 token snippets about cooking eggs or some shit
sad but it is how it is

Anonymous

Anonymous Tue 22 Aug 2023 12:31:41 No.95532016 Report

Quoted By:

>>95531482
you can still run stable diffusion and llama.cpp

Anonymous

Anonymous Tue 22 Aug 2023 12:32:26 No.95532025 Report

Quoted By:

>>95531812
it's useless when comparing across models for that reason, but when you're evaluating quantizations of the same model it's a great metric to see how much IQ you're knocking off since you're only comparing relative to differently-quantized versions of the same weights

Anonymous

Anonymous Tue 22 Aug 2023 12:35:57 No.95532079 Report

Quoted By: >>95532354

>>95531812
q3km and q4ks might be actually worth considering now. or at least they're not completely worthless now. they're really small, relatively speaking.
i'd want to know more about how larger models handle quants though.

Anonymous

Anonymous Tue 22 Aug 2023 12:37:27 No.95532097 Report

Quoted By: >>95532114

>>95530588
in my standard set of instructions for 70b-chat I include a line in my system prompt's writing style instructions like
>Move the story at a slow pace and remain focused on the scene at hand. Do not gloss over details or skip ahead unless directed.
works pretty well for me, don't know how well it will translate to other models though

Anonymous

Anonymous Tue 22 Aug 2023 12:39:10 No.95532114 Report

Quoted By:

>>95532097
Thanks. It's rare to get explicit prompt advice in this thread. I'll try adding it to my prompt.

Anonymous

Anonymous Tue 22 Aug 2023 12:57:23 No.95532354 Report

Quoted By:

>>95532079
Even though perplexity is supposed to be a linear metric, I'd argue for models as big as 65b and 70b small improvement in perplexity is still significant since its the intricacies that dictates the last few % of improvement over the smaller models in actual evals.

Anonymous

Anonymous Tue 22 Aug 2023 12:58:09 No.95532367 Report

Quoted By:

>>95530937
Maybe 70B Q2_K will be the meta now?

Subject

Name

E-mail

Password

Capcode	All Only User Posts Only Verified Posts Only Moderator Posts Only Manager Posts Only Admin Posts Only Developer Posts Only Founder Posts
Show Posts	All Only With Images Only Without Images Only Spoiler Images Only Non-Spoiler Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

On these archives

On these boards

Your latest searches