We present an approach to modifying Transformer architectures by integrating graph-aware relational reasoning into the attention mechanism.