Happy to share our latest paper titled “Graph Receptive Transformer Encoder (GRTE) for Text Classification” published in IEEE Transactions on Signal and Information Processing over Networks! Our novel approach combines graph neural networks (GNNs) with large-scale pre-trained models to address limitations in attention mechanisms of transformers for text classification. GRTE retrieves global and contextual information by representing texts as graphs, offering significant performance improvements compared to state-of-the-art models and computational savings up to ~100x.   Check out the paper for more details! #NLP #TextClassification #Transformer #GNN #GRTE #TSIPN #IEEE   Paper: https://ieeexplore.ieee.org/document/10477516 Code: https://github.com/koc-lab/grte   Abstract: By employing attention mechanisms, transformers have made great improvements in nearly all NLP tasks, including text classification. However, the context of the transformer’s attention mechanism is limited to single sequences, and their fine-tuning stage can utilize only inductive learning. Focusing on broader contexts by representing texts as graphs, previous works have generalized transformer models to graph domains to employ attention mechanisms beyond single sequences. However, these approaches either require exhaustive pre-training stages, learn only transductively, or can learn inductively without utilizing pre-trained models. To address these problems simultaneously, we propose the Graph Receptive Transformer Encoder (GRTE), which combines graph neural networks (GNNs) with large-scale pre-trained models for text classification in both inductive and transductive fashions. By constructing heterogeneous and homogeneous graphs over given corpora and not requiring a pre-training stage, GRTE can utilize information from both large-scale pre-trained models and graph-structured relations. Our proposed method retrieves global and contextual information in documents and generates word embeddings as a by-product of inductive inference. We compared the proposed GRTE with a wide range of baseline models through comprehensive experiments. Compared to the state-of-the-art, we demonstrated that GRTE improves model performances and offers computational savings up to ~100x.  

Please check our new article at the intersection of machine learning and signal processing published at IEEE Signal Processing Letters! We extend the theory of FrFT, a parametric signal transformation, by introducing it as a trainable layer in neural network architectures. We showed that the transformation parameters can be learned along with the remaining network weights through backpropagation during the training stage. Having established the theoretical foundation for this concept and validated through comprehensive experiments across distinct tasks, our work encourages the integration of numerous other parametric signal processing tools into neural networks.   Paper: https://ieeexplore.ieee.org/document/10458263 Code: https://github.com/koc-lab/TrainableFrFT   Abstract: Recently, the fractional Fourier transform (FrFT) has been integrated into distinct deep neural network (DNN) models such as transformers, sequence models, and convolutional neural networks (CNNs). However, in previous works, the fraction order a is merely considered a hyperparameter and selected heuristically or tuned manually to find the suitable values, which hinders the applicability of FrFT in deep neural networks. We extend the scope of FrFT and introduce it as a trainable layer in neural network architectures, where a is learned in the training stage along with the network weights. We mathematically show that a can be updated in any neural network architecture through backpropagation in the network training phase. We also conduct extensive experiments on benchmark datasets encompassing image classification and time series prediction tasks. Our results show that the trainable FrFT layers alleviate the need to search for suitable a and improve performance over time and Fourier domain approaches. We share our publicly available source codes for reproducibility.

Our collaborative work at UMRAM, Departments of Psychology and Electrical and Electronics Engineering of Bilkent University, and Department of Psychiatry of Hacettepe University on NLP-based approaches to understand schizophrenia is now published at Schizophrenia Research! In this study, we applied multiple natural language processing (NLP) methods for defining linguistic features of Turkish, a non-Indo-European language, in schizophrenia (SZ) to see whether the results of the NLP analyses in Turkish are language-independent or language-dependent. Our findings showed that altered linguistic features in SZ are mostly language-independent. Our results are promising to describe language patterns in schizophrenia and show that NLP measurements may allow for rapid and objective measurements of linguistic features. Paper: https://authors.elsevier.com/c/1if-S3HQGmiIHq Paper: https://www.sciencedirect.com/science/article/pii/S0920996424000628   Abstract: Natural language processing (NLP) provides fast and accurate extraction of features related to the language of schizophrenia. We utilized NLP methods to test the hypothesis that schizophrenia is associated with altered linguistic features in Turkish, a non-Indo-European language, compared to controls. We also explored whether these possible altered linguistic features were language-dependent or -independent. We extracted and compared speech in schizophrenia (SZ, N = 38) and healthy well-matched control (HC, N = 38) participants using NLP. The analysis was conducted in two parts. In the first one, mean sentence length, total completed words, moving average type-token ratio to measure the lexical diversity, and first-person singular pronoun usage were calculated. In the second one, we used parts-of-speech tagging (POS) and Word2Vec in schizophrenia and control. We found that SZ had lower mean sentence length and moving average type-token ratio but higher use of first-person singular pronoun. All these significant results were correlated with the Thought and Language Disorder Scale score. The POS approach demonstrated that SZ used fewer coordinating conjunctions. Our methodology using Word2Vec detected that SZ had higher semantic similarity than HC and K-Means could differentiate between SZ and HC into two distinct groups with high accuracy, 86.84 %. Our findings showed that altered linguistic features in SZ are mostly language-independent. They are promising to describe language patterns in schizophrenia which proposes that NLP measurements may allow for rapid and objective measurements of linguistic features.

The 34th Graduate Research Conference, organized by the Department of Electrical and Electronics Engineering (EE) and the IEEE Student Branch, took place on Thursday, January 25 in the lecture halls on the ground floor of the EE building. Bilkent EE graduate students described their state-of-the-art research projects in oral and poster presentations during this day-long event. This year’s conference included 38 oral presentations grouped into nine sessions, with three sessions occurring concurrently during each of the three time blocks. In the oral sessions, graduate students shared their research projects and answered questions from the audience. Various electrical and electronics engineering areas were covered, including artificial intelligence and machine learning, biomedical technologies, communications and networking, control and robotics, electromagnetics, electronics, integrated circuits and systems, nanotechnology, optics and photonics, and signal processing. During the two poster sessions, attendees had the opportunity to interact with the 25 presenters and learn more about research in the EE department. The conference also included some invited poster/demo presentations by EE undergraduate students. From among the presentations, the event jury selected three oral presentations for the “Best Oral Presentation” award and one poster presentation for the “Best Poster Presentation” award. The oral presentation awards went to Hasan Atakan Bedel, Burak Çatalbaş and Abdullah Erkam Arslan, while Bilal Kabaş received the poster presentation award. Prof. Süleyman Serdar Kozat of the Department of Electrical and Electronics Engineering chaired the conference. EE graduate students Ahmet Safa Öztürk, Issifu Iddrisu and Ecem Şimşek along with undergraduate students Zeynep Naz Bayındır (EE/II) and Baturalp Sönmez (CTIS/III), from the IEEE Bilkent Student Branch, took part in organizing the event. The event offered a day full of knowledge sharing and a unique opportunity for undergraduate students to experience the world of research in engineering and science. Participants had the chance to interact with researchers from other universities and professionals from some of the top engineering companies in Turkey.

The recipients of the “Elginkan Foundation Turkish Culture Research and Technology Awards”, which were launched by the Elginkan Foundation in 2006, in their 18th year, have been announced. The recipients of the awards, which will be given to contribute to studies, research and services in the fields of science, culture and arts in Turkey and to support the development of these fields, have been determined as follows. In line with the objectives of Elginkan Foundation; The eighteenth edition of the Turkish Culture Research and Technology Awards Program, which was first implemented in 2006 to support research, studies and services in the field of Turkish culture and technology, was held in 2023. One of the Elginkan Foundation 2023 Technology Awards was given to Prof. Dr. Ergin Atalar for his project “Manyetik Rezonans Görüntüleme Cihazları için Radyofrekans Verici Dizi Geliştirilmesi”.

Check out our recent IEEE Transactions on Audio, Speech, and Language Processing article on automatic sememe knowledge base construction. Sememes are the smallest and indivisible semantic units of word meaning. A predefined set of sememes is theoretically considered “the periodic table” of meaning in a natural language. However, unlike chemistry, the composition of the ultimate sememe set and optimal annotation of words with the elements of that set are open problems in language. Understanding semantics of language through the sememes is of fundamental importance to bolster the performance of a wide variety of high-level NLP applications, including today’s popular language models. Hence, linguistic experts are spending significant effort to manually determine this implicit and encompassing sememe set. In our article, we proposed the first automatic method for sememe set generation by using techniques like matrix factorization and topic modeling, utilizing tools such as Non-Negative Matrix Factorization (NMF), Latent Semantic Analysis (LSA), and Latent Dirichlet Allocation (LDA). The proposed method requires only a readily available dictionary and enhances the efficiency and scalability of generic sememe set construction. A fully computational and automatic way of devising sememe sets from readily available dictionaries can open up research directions and accelerate the existing ones where sememe knowledge is used to improve a wide range of high-level NLP tasks. Paper: https://ieeexplore.ieee.org/document/10375764 Code: https://github.com/koc-lab/mrd2skb   Abstract: Sememes are the minimum semantic units of natural languages. Words annotated with sememes are organized into Sememe Knowledge Bases (SKBs). SKBs are successfully applied to various high-level language processing tasks as external knowledge bases. However, existing SKBs are manually or semi-manually constructed by linguistic experts over long periods, inhibiting their widespread utilization, updating, and expansion. To automatically construct an SKB from Machine-Readable Dictionaries (MRDs), which are readily available, we propose MRD2SKB as an automatic SKB generation approach. Well-established MRDs exist, and their construction is much simpler than SKBs. Therefore, the proposed MRD2SKB allows for fast, flexible, and extendable generation of SKBs. Building upon matrix factorization and topic modeling, we proposed several variants of MRD2SKB and constructed SKBs fully automatically. Both quantitative and qualitative results of extensive experiments are presented to demonstrate that the performances of the proposed automatically created SKBs are on par with manually and semi-manually prepared SKBs.

Manoucher Takrimi authored an article on the eddy current losses in Magnetic Resonance in Medince. The article is visible in the offical web page. In this research, a novel method is introduced to enhance the performance of gradient array coils used in medical imaging. The approach relies on a computational electromagnetic strategy to estimate and control power losses within the cryostat, the component that houses the coil. Compared to existing methods, the new approach results in a remarkable 280% reduction in eddy power loss. The advantages are manifold: firstly, it tackles power losses within the entire cryostat body rather than just its inner surface close to the coils; secondly, it speeds up the tuning of the array coil and accurately predicts power losses or stored magnetic energies; thirdly, its accuracy matches that of commercial software; and finally, it is adaptable to array coils of any shape or even traditional coils, extending its applicability across various imaging scenarios. In simpler terms, this research provides a more efficient and versatile method to optimize the performance of medical imaging equipment, reducing energy loss and enhancing imaging quality.

Dr. Algin and his team’s new study has been published.  The results of a recent study conducted by scientists from Ankara, Bilkent, and Gazi universities were published in the scientific journal “Interventional Neuroradiology“.  In this study (quantitative radial force measurements of Woven EndoBridge devices), it was determined that the relationships between the radial forces of new generation aneurysm stents (flow disruptors) and various stent sizes were determined by Dr. Algin and colleagues. The related article can be accessed with the following citation: Quantitative radial force measurements of Woven EndoBridge devices Kutbay U, Algin O. Interv Neuroradiol. 2023; Doi: 10.1177/15910199231209072 https://pubmed.ncbi.nlm.nih.gov/37908102/

The work of Dr. Koç and his collaborators titled “RadGT: Graph and Transformer-based Automotive Radar Point Cloud Segmentation” has been published in IEEE Sensors Letters. In this study, we delve into the world of automotive radar data processing, leveraging advanced techniques such as graph-based methods and Transformers. Our research addresses critical challenges in radar point cloud segmentation, paving the way for enhanced accuracy and efficiency in autonomous driving systems.   With the rising significance of graph and transformer-based algorithms, the incorporation of these techniques into automotive radar data analysis signifies a substantial advancement in the industry. Through the fusion of these methods, we have attained exceptional outcomes, expanding the horizons of what can be achieved in radar point cloud segmentation.   The implications of our work extend far beyond the realm of autonomous vehicles. The advancements contribute to the broader landscape of computer vision and signal processing, opening new avenues for research and development in various domains. The paper can be accessed [here].   Abstract: The need for visual perception systems providing situational awareness to autonomous vehicles has grown significantly. While traditional deep neural networks are effective for solving 2-D Euclidean problems, point cloud analysis, particularly for radar data, contains unique challenges because of the irregular geometry of point clouds. This letter proposes a novel transformer-based architecture for radar point clouds adapted to the graph signal processing (GSP) framework, designed to handle non-Euclidean and irregular signal structures. We provide experimental results by using well-established benchmarks on the nuScenes and RadarScenes datasets to validate our proposed method.  

As the popularity and importance of large language models (LLMs) are soaring by the day, I am happy to announce that our recent work on gender bias and fairness in contextualized language models for natural legal language processing (NNLP) is now out at ACM Transactions on Knowledge Discovery from Data! Computational law, mostly based on natural language processing (NLP) and machine learning, has recently gained significant interest due to the technological improvements, abundance of legal texts, and increasing demand from legal professionals for technology. Since law is probably one of the most influential areas that touch upon lives, fairness and bias-free algorithm development are even more critical when we talk about artificial intelligence in law practices. In our article titled “Measuring and Mitigating Gender Bias in Legal Contextualized Language Models”,   we have introduced the first study on detecting and eliminating gender and societal bias issues present in contextualized large language models for legal NLP. This work brings the important issue of bias to the attention of computational law community, where the effects of law on people’s lives has doubled the need for studying bias. Paper: https://dl.acm.org/doi/10.1145/3628602 Code: https://github.com/koc-lab/ContextLegalBias   Abstract: Transformer-based contextualized language models constitute the state-of-the-art in several natural language processing (NLP) tasks and applications. Despite their utility, contextualized models can contain human-like social biases as their training corpora generally consist of human-generated text. Evaluating and removing social biases in NLP models has been a major research endeavor. In parallel, NLP approaches in the legal domain, namely legal NLP or computational law, have also been increasing. Eliminating unwanted bias in legal NLP is crucial since the law has the utmost importance and effect on people. In this work, we focus on the gender bias encoded in BERT-based models. We propose a new template-based bias measurement method with a new bias evaluation corpus using crime words from the FBI database. This method quantifies the gender bias present in BERT-based models for legal applications. Furthermore, we propose a new fine-tuning-based debiasing method using the European Court of Human Rights (ECtHR) corpus to debias legal pre-trained models. We test the debiased models’ language understanding performance on the LexGLUE benchmark to confirm that the underlying semantic vector space is not perturbed during the debiasing process. Finally, we propose a bias penalty for the performance scores to emphasize the effect of gender bias on model performance.

Laleh Eskandarian has presented her MSc thesis titled ”Structural connectivity alters in Pediatric Systemic Lupus Erythematosus prior to neuropsychiatric manifestations” on February 4, 2022.   ABSTRACT Systemic lupus erythematosus (SLE) is an autoimmune multi-system disorder affecting the central nervous system which may exhibit neuropsychiatric manifestations. Pediatric-onset SLE is rare and although there is an increased risk of neuropsychiatric symptoms, underlying pathological mechanisms remain poorly understood. This study explored the entire white matter network with brain structural connectivity and DTI tractography analysis to provide a better understanding of the probable connectivity alterations. Whole-brain structural connectivity and tractography of 17 pediatric-onset SLE patients without neuropsychiatric involvement (non-NPSLE) and 8 age and gender matched healthy controls were explored. To investigate the topological structure, graph theory analysis was applied. Global and nodal network features were derived by estimating structural connectivity matrices between 360 brain regions of the HCP atlas. Non-NPSLE patients demonstrated higher network characteristic path length and assortativity and lower density and global efficiency compared with healthy controls. The hubs were selected according to the strength of the nodes and their structure and distribution were changed in the patients. According to the altered hubs’ network characteristics significant modifications between HCs and non-NPSLE, along with TBSS findings, altered regions were observed in language, visual, auditory, and motor-related areas. These findings are in good agreement with WISC-IV Verbal Comprehension Index. Compared to healthy controls non-NPSLE patients showed significantly lower scores according to WISC-IV score components. To conclude, from a network and connectivity perspective, our research demonstrated an altered topological structure of the brain in non-NPSLE patients. The findings of this study provide a better understanding of the structural alterations underlying pediatric-onset non-NPSLE patients’ functional and neurocognitive abnormalities.