We are happy to announce that we demonstrated our latest work titled “VISPool: Enhancing Transformer Encoders with Vector Visibility Graph Neural Networks” at ACL 2024, Bangkok/Thailand!
In our #ACL2024 paper, we explore the powerful combination of transformers and graph neural networks (GNNs) to push the boundaries of natural language processing (NLP). While transformers have revolutionized NLP, integrating GNNs presents a promising avenue, though existing methods often struggle with limitations of transductive approach, scalability and complex text processing.
To address these challenges, we introduce a novel dynamic graph construction method for text documents, leveraging vector visibility graphs (VVGs) generated from transformer outputs. Our proposed model, the visibility pooler (VISPool), seamlessly integrates VVG convolutional networks into transformer pipelines. Notably, VISPool outperforms baselines on the GLUE benchmark with fewer trainable parameters, highlighting the effectiveness of our approach in enhancing transformers with GNNs.
For more details, check out the full paper and source code!
#ACL2024 #NLP #Transformers #GNNs #AI #MachineLearning #ACL #AI
Paper: https://aclanthology.org/2024.findings-acl.149
Code: https://github.com/koc-lab/vispool
Abstract:
The emergence of transformers has revolutionized natural language processing (NLP), as evidenced in various NLP tasks. While graph neural networks (GNNs) show recent promise in NLP, they are not standalone replacements for transformers. Rather, recent research explores combining transformers and GNNs. Existing GNN-based approaches rely on static graph construction methods requiring excessive text processing, and most of them are not scalable with the increasing document and word counts. We address these limitations by proposing a novel dynamic graph construction method for text documents based on vector visibility graphs (VVGs) generated from transformer output. Then, we introduce visibility pooler (VISPool), a scalable model architecture that seamlessly integrates VVG convolutional networks into transformer pipelines. We evaluate the proposed model on the General Language Understanding Evaluation (GLUE) benchmark datasets. VISPool outperforms the baselines with less trainable parameters, demonstrating the viability of the visibility-based graph construction method for enhancing transformers with GNNs.