>>108422953"Experiments" by a schizo called DavidAU.
Look at his model collection, choose one of the older ones with a really long name, then read the model's card.
Behold the magnificence.
"expanding" (upscaling) a smaller model into a larger one then Pretraining the shit out of it, essentially using the original model as a base for a whole new model, is legit.
It's just that you need to do proper pretraining with trillions of token, not do some qlora.
Look at SOLAR 10B from back in the day. It's upscaled from mistral 7B IIRC.