NTT ExC Partners talk about why they chose "Super RAG" and the benefits of its implementation

Project Overview

In the help desk operations of the procurement system provided by NTT ExC Partner, operators use LLMs (large-scale language models) when responding to various inquiries from customers. By incorporating Cinnamon AI's "Super RAG" into the operations, the LLM can now refer to complex documents (unstructured documents) such as manuals and specifications that contain figures and tables with a high degree of accuracy, achieving three times the accuracy of responses compared to the previous method (measured by the company).

Furthermore, by being able to instantly and accurately search large volumes of documents, the system has also succeeded in streamlining the response confirmation work of operators. In the future, NTT ExC Partners will continue to promote performance improvement using "Super RAG" in help desk operations that require tacit knowledge and experience, and will collaborate with Cinnamon AI to consider introducing "Super RAG" to other operations. In this interview, we asked Mr. Horiuchi and Mr. Kaneko of the DX Procurement Division about the reason for introducing "Super RAG," the effects it has had, and their outlook for the future.
*This interview was conducted in February 2025.

NTT ExC Partners, Inc.: https://www.nttexc.co.jp/
Cinnamon AI "Super RAG":https://cinnamon.ai/ai_model/super-rag/

"Super RAG" Introducing company person in charge

NTT ExC Partners Inc.
DX Procurement Division, Business Systems Department, Business Promotion Section Manager
Mr. Shu Horiuchi

After gaining experience at an IT venture, he participated in the launch of NTT Communications' video service. He then moved to NTT Plala. After transferring to Docomo, he was seconded to NTT ExC Partners in July 2023. He established a data analysis team for "Hikari TV," a service provided by Plala, and contributed to customer analysis and cancellation prediction. Since 2018, he has commercialized recommendations and automatic video editing using hybrid technology that combines traditional data analysis with AI, and has given numerous lectures on the results. He is currently involved in the use and promotion of AI technology within the company.

NTT ExC Partners Inc.
DX Procurement Division, Business Systems Department, Business Promotion Officer
Mr. Hirotaka Kaneko

He is promoting the introduction of generative AI using RAG technology to the support desk operations that support the NTT Group's procurement system. He is working on creating a system to improve the efficiency of inquiry response and the sophistication of the support system. In addition, based on the knowledge gained from this initiative, he is responsible for the deployment and dissemination of RAG-based generative AI throughout the company, and is acting as a promoter of AI utilization.

Q1. Please tell us the background and motivation for introducing "Super RAG"

Horiuchi: Our team has been working on improving the efficiency of support desk operations using generative AI and RAG (Retrieval-Augmented Generation * technology required for introducing generative AI into companies) since around December 2023. We conducted a PoC (proof of concept) at the support desk from January to March 2024 using RAG developed by another company, and based on the results, we began actual operation at the support desk. However, there were limitations to being text-based alone, and the challenge was that it could not handle manuals that contained many diagrams and charts.

In this situation, when we decided to use RAG, we visited AI EXPO (5th AI/Artificial Intelligence EXPO Autumn) in May 2024 to look for solutions to business issues that we thought would come to a standstill in the future. That was how we encountered Cinnamon AI. We visited many booths at AI EXPO to gather information, but later on, it was Cinnamon AI's technology that caused a buzz within the company. That's because, at the time,The term RAG came up frequently at industry exhibitions and seminars, but only two companies, including Cinnamon AI, offered a concrete solution that could solve our problem.That's why.

Q2. What are the selection criteria for the RAG system?

Horiuchi: We narrowed down our selection to two companies that offered RAG that seemed suitable for our business issues: Cinnamon AI, which was offering it at the time, and another company. In particular, OCR technology for document analysis was an important point, and we needed to have RAG properly recognize the charts and flow charts included in the support desk manual as data.

The selection criteria for RAG was based on its ability to accurately capture charts and graphs and interpret their meaning. Data analysis techniques such as vector search and chunking were also important factors. At the time, RAG was still a new technology, and companies were just starting to pay attention to it as a keyword. While there were many companies that were like, "We're going to do our best with RAG from now on!", Cinnamon AI met our requirements, and the fact that it was already in practical use was the deciding factor.

Q3. Why did NTT ExC Partners choose "Super RAG"?

Horiuchi: I tried several RAGs, but decided that Cinnamon AI's "Super RAG" was the best.Cinnamon AI was particularly outstanding in terms of chart and flow chart recognition and ease of use.

Q4. Were there any particularly memorable or challenging aspects during the "Super RAG" implementation process?

Kaneko: I was impressed by how well it could read charts and graphs. I gave the document to Cinnamon AI and itWhen I requested a demo within 2-3 days, I was surprised at how quickly I received the answer I expected.Sometimes there are scribbled notes that even humans can't read, and even Super RAG has a hard time reading them (laughs), so they need to be corrected by hand, but Super RAG can read even fairly difficult diagrams, so I'm looking forward to future updates.

Horiuchi: We didn't encounter any major issues when introducing "Super RAG". We were given a very clear answer about future scalability, which gives us peace of mind.

Q5. What issues do you expect to be resolved by introducing "Super RAG"?

Horiuchi: The introduction of "Super RAG" is a project that is positioned at the core of our AI utilization strategy.That's because the biggest keyword in our team's roadmap for 2025 is "AI agent."

"Super RAG" is a single function, so we are considering combining it with other AIs and linking them together in "AI to AI". Currently, the operator accepts the question, summarizes it, and inputs it into "Super RAG" to confirm the answer, but we would like to do this entire flow with AI. In other words, the AI summarizes the question and checks the answer. If that doesn't work, the operator will provide support. However, if there is no problem with the answer, we are thinking of having the AI answer as it is. We would like to take Copilot, which is a little convenient, and turn it into Agent, which acts as a complete proxy. We are also thinking of working in that area with Cinnamon AI.

Q6. Please tell us about the initial results and effects after implementation.

Horiuchi: Our team has three KPIs:

First"Correct answer rate"is the percentage of people who can answer correctly knowing that "Super RAG" has the correct answer, and this number exceeds 90%, meaning that if the answer is available, it can be retrieved without any problems.

Second"Hit rate"is a number that indicates how many questions a support desk can answer in its "raw" state, regardless of whether it has answers or not, that it receives in the course of normal business. It depends on the quantity and quality of documents fed into "Super RAG," but we thought this was the key to this initiative. When we tried a different RAG in May 2024, the hit rate was 21%, but the hit rates for the two support desks that introduced "Super RAG" were 80% and 60%. We believe that we can increase this to 90% in the future. In other words, the fact that "Super RAG" has significantly increased the hit rate is an easy-to-understand result in a short period of time.

Third KPI"Time saving"The current situation is that the number of users is not increasing. When they tried a different RAG, the time reduction was 31 minutes, but after introducing "Super RAG", the time reduction was only 8 minutes. The hit rate has increased, but we recognize that the time reduction is not as great as expected as an issue. We think that the reason the time reduction is not increasing as expected is that the operator creates the answer after carefully reading the AI's answer, so since "Super RAG" gives various answers, it takes time for the operator to read all of those answers.

(Response rate and hit rate seem to affect the prompts that operators create. What are your thoughts on this?)

Horiuchi: As you say, I think the accuracy of the answer depends on the operator's prompt. In the past, I did a PoC for RAG in another project, and I felt firsthand that the prompt made a difference in the accuracy of the answer. Based on that experience, I created a "prompt cheat sheet" of about seven points that make it easier to get the right answer, and shared it with the people in the field.

By the way, I would like to omit the training on how to write prompts in the future. My idea is to have an AI agent following a cheat sheet summarize the question itself, instead of an operator.

Q7. Please tell us the reactions of end users who have actually used the product.

Kaneko: The response to Super RAG has been very positive, and other departments have expressed a desire to use it as well. We plan to hold internal information sessions in the future.

especially,"Super RAG" is able to read documents containing charts and graphs that cannot be read by the RAGs used in other departments, and has been highly praised within the company.In addition, the interface of "Super RAG" is very easy to understand, so I think that even new employees will be able to start using it right away. Since the support desk is a mass of know-how, I hope that it will also shorten the training time.

Q8. Given the possibility of hallucination (so-called incorrect answers given by AI), how do you judge and manage the accuracy of answers in practice?

Horiuchi: Currently, operators do not simply accept the answers generated by the AI, but make corrections based on them and then respond. The answers generated by the AI are checked by operators, and strange answers (hallucinations) are rejected, but it seems that there are not many answers that are rejected.

Also,When the AI generates answers, the search results can be used by the operator like a dictionary, which is far more convenient than the previous task of physically searching for answers from a manual.

Q9. After this project, what would you recommend "Super RAG" to other departments or group companies?

Horiuchi: I know I've given the same answer many times, butHigh ability to recognize charts and figuresAs a company that provides IT systems, we have many manuals that explain how to operate the screens, so understanding diagrams is extremely important. "Super RAG" accurately converts the diagrams into knowledge, to the point where I'm impressed at how well it understands the intent behind the diagrams.

Surprise about the tableIt is necessary to read the items on the X and Y axes of the table (like a human), but for example, in a document like a star rating table for authority regulations, even if you ask "Who is responsible for payments up to 30 million yen?", you will get a clear answer like "The manager will make the payment!" Furthermore,It is also possible to read flow charts, which is one of the reasons why other teams using RAGs have evaluated "Super RAG" as being one step ahead.I think that this is the core of why we decided to adopt Cinnamon AI's "Super RAG". I would like to recommend it to our group companies.

Q10. How do you think "Super RAG" will help your company promote digital transformation?

Horiuchi: We are in charge of procurement systems, but we also provide a wide range of systems in the field of human capital."Inquiry" work is an inseparable task and naturally requires a considerable amount of work, so we believe that by horizontally deploying "Super RAG" to other departments, we can achieve digital transformation.

Cost-effectiveness is a key point for horizontal deployment within the company. When our division introduced AI, we placed emphasis on speed, so we put cost-effectiveness on the back burner. However, when considering internal deployment, we received many questions about reducing costs, and we need to produce solid results in the future.

Q11. How do you want to use "Super RAG" in the future?

Horiuchi: In fiscal year 2025, we will be moving towards AI agents. RAG will connect each task and turn it into an AI agent, promoting business efficiency. We believe that the core solution for this is "Super RAG."

Our ultimate vision is to create the ultimate AI to AI system that does not require human intervention in any scenario. It would be amazing if we could eliminate the need to contact an operator in the first place.

In addition, since there are many different output locations in the business world, I think the scalability of output devices will also be important. For example, if "Super RAG" had the scalability to be easily used on various SaaS, I think the value of its use would increase. Since "Super RAG" has an API, we are also considering creating something like a wrapping system. We have many ideas on this topic, so we would like to continue discussing them with everyone at Cinnamon AI.

Conclusion

NTT ExC Partners is strengthening the use of generative AI in various areas, including the support desk. The AI Co-Creation Innovation Office, established in February, is supporting the promotion of projects at a company-wide level, accelerating the introduction of cutting-edge technologies such as this project. We will be keeping a close eye on the efforts of NTT ExC Partners and Cinnamon AI, which are looking to turn "AI to AI" agents into reality in the future by utilizing the excellent document analysis functions of "Super RAG."

--Thank you for your valuable insights today.

For inquiries about "Super RAG"Click here
"Super RAG" document downloadClick here