Can you chat with your company database just like you do with ChatGPT?

Introduction

What if querying your database were as simple as asking ChatGPT a question? Accessing and analyzing data are essential to the success of any business. However, effective data utilization is often hindered by the need for specialized technical skills, particularly in SQL or BI, and a deep understanding of the company's data schema. This technical barrier limits access to information for a vast majority of employees, creating a bottleneck in certain decision-making processes.

There is growing interest in accessible and simple AI interfaces. We can imagine a world where corporate databases are not only accessible to a limited fraction of employees but to everyone via a simple conversational experience like with ChatGPT.

‍

‍

This use case aims to address two main objectives:

1. Democratize data access: enable all profiles, technical or otherwise, to interact easily with data.
2. Optimize team time: reduce the time spent on data extraction and manipulation so that teams can focus on high-value analysis.

These objectives are accompanied by an essential priority: ensuring secure and reliable database operation, thereby guaranteeing the protection of sensitive information. 

Generative AI performance on the BIRD benchmark 

The BIRD benchmark, published in 2023, focuses on evaluating AI performance on text-to-SQL tasks. This refers to an AI model's ability to translate a question formulated in natural language into a reliable and correct SQL query, allowing for digital interaction with a database. 

The BIRD database centralizes 95 other databases (33.4 GB) from 37 different professional domains (represented below). It contains 12,751 question-answer pairs: each question formulated in natural language is provided with its associated answer expressed in SQL. 

‍

‍

‍

The BIRD benchmark stands out for its relevance, as it reflects a level of complexity and data quality close to that encountered in a real business. Thus, it offers a realistic framework for evaluating performance. For reference, human performance on this benchmark, achieved by data engineers, is 92.9%. 

Below is the AI performance on this benchmark at the time of writing:

‍

Since 2023, a real race has been underway to match human performance, but despite spectacular advances in the field of LLMs, AI researchers have not yet managed to reach this level. 

If you would like to discover the technical approach adopted by one of the winning teams, you will find more details in the references cited in this article.

Aqsone procurement case study  

Driven by a desire to explore this innovative use case, a team from the Aqsone Lab conducted a series of experiments. Their goal: to evaluate the feasibility of conversational data exploration by leveraging existing solutions and approaches, without, however, participating directly in the BIRD benchmark.

Here are the key steps of our approach:

1. Analysis of existing solutions 

We reviewed existing solutions, ranging from off-the-shelf tools like DataGPT and DataChat to technical approaches such as natural language to SQL (text-to-SQL) or natural language to Python code generation. 

The qualitative evaluation was based on a series of criteria, including functionality, cost, data privacy, compatibility with multi-table databases, ease of use, the ability to generate data visualizations, and the overall conversational experience.

Our analysis highlighted major limitations of certain existing tools, particularly in terms of transparency regarding their operation (and pricing), the quality of their documentation, guarantees regarding data privacy and security, and their ability to handle multi-table databases.

2. Creation of a reference dataset and an evaluation set 

To best reflect real-world enterprise data, we generated an SAP procurement database. This dataset, designed to be accessible and understandable by a standard buyer, was used as the foundation for our experiments.

‍

‍

An evaluation dataset was also created. It consists of 30 question-and-answer pairs: a question formulated in natural language and an answer in SQL. 

‍

‍

3. Definition of approaches to test

We implemented and tested three different approaches based on LLMs and existing frameworks, such as PandasAI, LangChain, LlamaIndex, and Vanna AI: these tools have the advantage of being pre-configured to address our use case. Consequently, they require very little customization. 

4. Evaluation of approaches 

The "Execution Accuracy" score is calculated based on the initially created evaluation dataset. This score allows for comparing the execution results of the ground truth SQL query and the predicted SQL query against the database content. Here are the results obtained:

‍

‍

In conclusion, using a more recent language model (GPT-4) combined with an agent-based approach using metadata and few-shot prompting, or a native LangChain chain approach, yields the best results on our dataset of 30 question-answer pairs.  

2. Developing an intuitive user interface

‍

To facilitate interaction with the LLM and the database, we designed a dedicated user interface based on Streamlit. 

‍

‍

We integrated a feature that allows users to view the SQL query used to answer a question. Here is an example:

‍

‍

Conclusions

Our experiments have demonstrated the promising potential of LLMs for facilitating data access within companies. Chain-based and agent-based approaches stood out for their performance, achieving 70% execution accuracy on our experimental dataset. With the use of GPT-4, these approaches demonstrated an impressive native knowledge of SAP data and can offer a satisfactory level of explainability by displaying the generated SQL queries.

These results lead us to believe that AI could soon surpass data analysis experts in terms of speed and efficiency, specifically in the Text-to-SQL domain. The existence of the BIRD benchmark, dedicated to evaluating AI performance on text-to-SQL tasks, is a major asset for tracking progress and establishing performance standards for this use case. 

Despite these promising advances, here are the three main challenges that remain:

1. Ensuring reliability and accuracy in responses

To address this, here are some possible solutions: 

• Provide high-quality, standardized data (such as SAP) as input 
• Use prompt engineering techniques to: 
  
  • Include contextual information about the database, such as descriptions, schema, DDL, primary keys, foreign keys, attributes, metadata, jargon, business context, join types, filters, date formats, etc. The more context the LLM has about the database, the more likely the response will be accurate. 
  • Encourage the LLM to rephrase the user's question and state the steps to follow to answer it 
  • Encourage the LLM to formulate correct SQL queries that follow SQL best practices (syntax issues, etc.) 
  • Apply advanced multiple-selection methods (self-consistency), which consist of generating several responses and choosing the most consistent one
• Verify the SQL query after generation to ensure it works on the database, then ask the LLM to correct any errors if necessary.

2. Ensure a great user experience  

Interacting with the tool should be as intuitive as using ChatGPT. Here are the best practices: 

• Allow users to rephrase questions and view their interaction history. 
• Enable the LLM to ask for clarification if a question is deemed complex or incomplete.
• Improve the explainability of LLM responses by displaying either the generated SQL query or, ideally, a detailed explanation of the reasoning process.

3. Ensure database protection 

It is crucial to guarantee secure interactions, especially in an era where data—often sensitive—plays a central role in decision-making. The use of external APIs, such as OpenAI's, raises legitimate concerns regarding data security and privacy. It is therefore essential to implement robust solutions to protect this sensitive information:

• Using Azure OpenAI could offer an increased level of privacy. If necessary, using an open-source LLM locally is recommended, though this solution can be costly and sometimes limited, as the most high-performing models are rarely available as open source. 
• Ensure the security of database interactions, particularly by strictly controlling access and write permissions to preserve data integrity.

Using LLMs for conversational data exploration is a major innovation that could transform how companies use and leverage their data. The progress made to date is encouraging and paves the way for an exciting future.

References

1. BIRD Benchmark 
   
   • BIRD Paper: Link to the BIRD paper
   • BIRD Benchmark: Official BIRD benchmark website
   • Here is a summary of the work from the Distyl AI team that topped the BIRD leaderboard between July and August 2024 with 71.83% accuracy. Link to blog post , Link to paper
     
     1. Use OpenAI's GPT-4o model
     2. Iteratively fine-tune OpenAI's GPT-4o model on specific Text-to-SQL tasks using a dataset of triplets: a natural language question, the corresponding SQL query, and the associated schema elements.
     3. Maximize the model's context usage by including all possible schema information: do not apply the traditional "schema linking" method => Instead of filtering database schema information, provide the full schema to the model.
     4. Augment the context with detailed column descriptions, query hints, and information regarding expected result ordering and aggregations.
     5. Apply self-consistency methods, which involve generating multiple responses and then selecting the most consistent one.
     6. Apply a self-correction method by simulating a database administrator LLM capable of identifying and fixing errors in generated SQL queries if necessary.
   • Work from a team with a score of 72.23%: Link to paper 
2. LangChain: LangChain documentation on SQL usage
3. Vanna AI:
   
   • Vanna AI official website
   • Article on a similar experiment
4. PandasAI: PandasAI official website
5. DataDM: Devpost page
6. Similar use case created with LangChain and GPT-4: Quantmetry article
7. Comparison between LLM agent and LLM chain approaches: blog post link
8. Francesco Puppini's talk titled “SQL with LLMs: chat with your data” youtube_link