Скачать с ютуб Attention Sinks в хорошем качестве

Attention Sinks

In this talk Ted leads the discussion on Attention Sinks, a mechanism that solves the issues with models getting lost when earlier tokens are dropped from a model as we process more and more tokens. This appears as the model devolving into returning repetitive and/or nonsensical responses when the data being processed gets larger than the model's windows size. In this talk, Ted describes the problem in detail, why it occurs and how Attention Sink Models are able to solve the problem, and make the model able to retaining the proper information to continue to generate high quality responses even as the number of context tokens gets larger and larger. By going though this information, Ted introduces the concepts of Spectral Filters and Dark Signals which are key to understanding how to solve this issue. As usual, throughout the presentation we had a lively discussion about the concepts, issues of concern, why the changes worked, and the end results. Links Our Meetup: https://www.meetup.com/East-Bay-Tri-V... Ted's slides: https://github.com/tedkyi/talks/blob/... Efficient Streaming Language Models with Attention Sinks: https://arxiv.org/abs/2309.17453 Spectral Filters, Dark Signals, and Attention Sinks, https://arxiv.org/html/2402.09221v1 Content 00:00 Intro 01:57 Long context approaches 06:29 Attention sinks 17:40 Spectral filters 36:55 Decoder only models 52:40 SVD 56:36 Spectral filtering ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ 😊About Us West Coast Machine Learning is a channel dedicated to exploring the exciting world of machine learning! Our group of techies is passionate about deep learning, neural networks, computer vision, tiny ML, and other cool geeky machine learning topics. We love to dive deep into the technical details and stay up to date with the latest research developments. Our Meetup group and YouTube channel is the perfect place to connect with other like-minded individuals who share your love of machine learning. We offer a mix of research paper discussions, coding reviews, and other data science topics. So, if you're looking to stay up to date with the latest developments in machine learning, connect with other techies, and learn something new, be sure to subscribe to our channel and join our Meetup community today! Meetup: https://www.meetup.com/east-bay-tri-v... ▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬ #attentionsinks #spectralfilters #darksignals #machinelearning #ai #humanreasoning #generativeai #generativemodels