We are proud to announce the GraphTeacher, which is now published in IEEE Transactions on Artificial Intelligence!<\/p>\n
GraphTeacher tackles a core NLP challenge\u2014scarce labeled data\u2014by integrating GNNs into the fine-tuning of transformer encoders to exploit unlabeled data while excluding test nodes from the graph, eliminating re-graphing and enabling inductive, single-instance inference.<\/p>\n
\ud83d\udd2c Evaluated on GLUE with BERT, DistilBERT, and RoBERTa across 5\u201350% labeled data, our approach yields up to +10% over baselines, demonstrating strong scalability, adaptability, and improved generalization\/robustness on diverse, graph-structured data\u2014offering an elegant solution to a real-world challenge.<\/p>\n
\ud83d\ude80 The contributions of GraphTeacher are summarized below:<\/p>\n
\u25cf\tUnlabeled data integration: Fine-tunes transformer encoders by leveraging unlabeled training data for real-world, low-label settings.
\n\u25cf\tNo test nodes in the graph: Excludes test nodes, avoiding re-graphing for new samples and preventing test-time information leakage.
\n\u25cf\tInductive inference: Supports efficient single-instance inference, making deployment seamless and scalable.
\n\u25cf\tScalable & robust: Works from standard to large-scale corpora and consistently improves performance.
\n\u25cf\tModel-agnostic GNN boost: Delivers gains across diverse GNN variants and with both custom and pre-built graphs.<\/p>\n
<\/p>\n
For more details and code:<\/p>\n
Paper: https:\/\/ieeexplore.ieee.org\/document\/11223233 <\/a> <\/p>\n Abstract:<\/strong><\/p>\n We present GraphTeacher for fine-tuning transformer encoders by leveraging Graph Neural Networks (GNNs) to effectively train models when fully labeled training data is unavailable. When different percentages of labeled training data exist, we study popular transformer models, DistilBERT, RoBERTa, and BERT. The proposed approach uses the underlying graph structure of a corpus by allowing the transformer encoders to incorporate GNNs into the fine-tuning process. Using latent patterns and correlations identified in unlabeled data, our method aims to enhance the model\u2019s adaptability to scarcely labeled data scenarios. Moreover, our approach excels in conducting single-instance inference, a capability not inherently possessed by models with a transductive (semi-supervised) training stage. GraphTeacher not only processes the unlabeled data effectively, as in transductive methods, but also offers an inductive inference setup for test samples. Experiments on diverse datasets and various GNN architectures show that integrating GNNs significantly enhances transformer encoders\u2019 robustness and generalization capabilities, in particular under sparsely labeled training conditions. GraphTeacher demonstrates a noteworthy improvement, achieving up to a 10% increase in performance on the GLUE benchmark dataset compared to the baselines.<\/p>\n","protected":false},"excerpt":{"rendered":" We are proud to announce the GraphTeacher, which is now published in IEEE Transactions on Artificial Intelligence! GraphTeacher tackles a core NLP challenge\u2014scarce labeled data\u2014by integrating GNNs into the fine-tuning of transformer encoders to exploit unlabeled data while excluding test nodes from the graph, eliminating re-graphing and enabling inductive, single-instance inference. \ud83d\udd2c Evaluated on GLUE […]<\/p>\n","protected":false},"author":9,"featured_media":12823,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[445],"tags":[],"_links":{"self":[{"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/posts\/12822"}],"collection":[{"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/users\/9"}],"replies":[{"embeddable":true,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/comments?post=12822"}],"version-history":[{"count":1,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/posts\/12822\/revisions"}],"predecessor-version":[{"id":12824,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/posts\/12822\/revisions\/12824"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/media\/12823"}],"wp:attachment":[{"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/media?parent=12822"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/categories?post=12822"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/umram.bilkent.edu.tr\/index.php\/wp-json\/wp\/v2\/tags?post=12822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nCode: https:\/\/github.com\/koc-lab\/graph-teacher <\/a><\/p>\n