Generative AI systems like ChatGPT are increasingly being used for help with decision making. However, for assistance with making tough decisions that lead to high-risk consequences (e.g., medical, judicial, business advice), these systems and the knowledge they provide have to be sufficiently trustworthy. In this PhD project, I aim to identify, formalize and capture different aspects that lead to appropriate trust (not under- or overtrust) in AI advice, to design hybrid AI systems that are a composite of data driven LLMs and knowledge driven models (Knowledge Graphs, Ontologies) that are able to adhere to these trust aspects, and to evaluate the trustworthiness of state-of-the-art AI advice, including existing software and proposed new methods, through both objective and user-centric evaluations.