Friday, March 3, 2023

Show HN: Zipslicer, a library for loading LLM checkpoints on consumer hardware

This is a low-level opensource library I developed for my own use and decided to share, as it makes it possible to process large checkpoints of neural networks without renting high-RAM instances, on a regular PC. It replaces torch.load() with a custom function that produces a dictionary that materializes tensors on the fly. Compared to other solutions it doesn't require sharding or re-encoding checkpoints and uses them completely as-is.

It is a foundation to make it possible to run inference and compress language models and other large models one layer at a time - in principle, even one tensor at a time.

I describe the rationale and technical details of the library's design in the blogpost: https://kir-gadjello.github.io/posts/zipslicer/


Comments URL: https://news.ycombinator.com/item?id=35013939

Points: 2

# Comments: 0



from Hacker News: Newest https://ift.tt/8rJ42Qk
via IFTTT

No comments:

Post a Comment

Show HN: rtrvr.ai – AI Web Agent for Automating Workflows and Data Extraction

Hey HN, I'm excited to share rtrvr.ai, a Chrome extension that brings the power of AI agents to your everyday web browsing. It's de...