Friday, March 3, 2023

Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware

This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is.

It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one tensor at a time.

I describe the rationale and technical details of the library's design in the blogpost: https://kir-gadjello.github.io/posts/zipslicer/


Comments URL: https://news.ycombinator.com/item?id=35013939

Points: 2

# Comments: 0



from Hacker News: Newest https://ift.tt/8rJ42Qk
via IFTTT

No comments:

Post a Comment

England see off USA to return to winning ways

England's Red Roses return to winning ways, but without entirely vanquishing doubts over their position at the top of the women's ga...