Digitala Vetenskapliga Arkivet

Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
Experience with Large Language Model Applications for Information Retrieval from Enterprise Proprietary Data
Blekinge Tekniska Högskola, Fakulteten för datavetenskaper, Institutionen för programvaruteknik.ORCID-id: 0000-0001-5949-1375
Blekinge Tekniska Högskola, Fakulteten för datavetenskaper, Institutionen för programvaruteknik.ORCID-id: 0000-0001-7526-3727
Örebro University.
Blekinge Tekniska Högskola, Fakulteten för datavetenskaper, Institutionen för programvaruteknik.ORCID-id: 0000-0002-3646-235x
2025 (engelsk)Inngår i: Product-Focused Software Process Improvement / [ed] Dietmar Pfahl, Javier Gonzalez Huerta, Jil Klünder, Hina Anwar, Springer, 2025, Vol. 15452, s. 92-107Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

Large Language Models (LLMs) offer promising capabilities for information retrieval and processing. However, the LLM deployment for querying proprietary enterprise data poses unique challenges, particularly for companies with strict data security policies. This study shares our experience in setting up a secure LLM environment within a FinTech company and utilizing it for enterprise information retrieval while adhering to data privacy protocols. 

We conducted three workshops and 30 interviews with industrial engineers to gather data and requirements. The interviews further enriched the insights collected from the workshops. We report the steps to deploy an LLM solution in an industrial sandboxed environment and lessons learned from the experience. These lessons contain LLM configuration (e.g., chunk_size and top_k settings), local document ingestion, and evaluating LLM outputs.

Our lessons learned serve as a practical guide for practitioners seeking to use private data with LLMs to achieve better usability, improve user experiences, or explore new business opportunities. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

sted, utgiver, år, opplag, sider
Springer, 2025. Vol. 15452, s. 92-107
Serie
Lecture Notes in Computer Science (LNCS), ISSN 0302-9743, E-ISSN 1611-3349 ; 15452
Emneord [en]
AI, Artificial intelligence, Data security, Information retrieval, Large Language Model, LLM, Sandbox environment, Data privacy, Fintech, Enterprise data, Language model, Model application, Modeling environments, Privacy protocols, Security policy, Structured Query Language
HSV kategori
Identifikatorer
URN: urn:nbn:se:bth-27326DOI: 10.1007/978-3-031-78386-9_7ISI: 001423664600007Scopus ID: 2-s2.0-85211960724ISBN: 9783031783852 (tryckt)OAI: oai:DiVA.org:bth-27326DiVA, id: diva2:1923603
Konferanse
25th International Conference on Product-Focused Software Process Improvement, PROFES 2024, Tartu, Dec 2-4, 2024
Ingår i projekt
SERT- Software Engineering ReThought, Knowledge Foundation
Forskningsfinansiär
Knowledge Foundation, 20180010Tilgjengelig fra: 2024-12-28 Laget: 2024-12-28 Sist oppdatert: 2025-12-03bibliografisk kontrollert
Inngår i avhandling
1. Quality Evaluation of Generative AI Systems: Processes, Metrics, Methods, and Frameworks for Industrial Software Engineering
Åpne denne publikasjonen i ny fane eller vindu >>Quality Evaluation of Generative AI Systems: Processes, Metrics, Methods, and Frameworks for Industrial Software Engineering
2026 (engelsk)Doktoravhandling, med artikler (Annet vitenskapelig)
Abstract [en]

Generative Artificial Intelligence (GenAI) is being rapidly adopted in software engineering, introducing a paradigm shift toward human-AI co-creation. However, the non-deterministic, probabilistic, and often black-box nature of GenAI models presents challenges for traditional software quality assurance. Conventional verification and validation techniques are insufficient to handle outputs that are neither predictably correct nor incorrect, but rather stochastically plausible. This discrepancy creates an urgent need for practical processes, metrics, and new governance frameworks to evaluate and manage the quality of GenAI systems in industrial environments.This thesis examines how industrial organizations adopt GenAI, identify metrics, and evaluate system qualities in alignment with ISO quality standards. Case studies were employed to explore real-world adoption processes, identify context-specific industrial metrics, and uncover practical insights within organizations. A snowballing literature review was conducted to systematically identify, categorize, and synthesize academic metrics for evaluating the output of GenAI systems. Finally, a controlled experiment was designed to quantitatively test the efficiency (e.g., E2E generation time) and effectiveness (e.g., accuracy) of GenAI agent choices. The main contributions of this thesis are a synthesized actionable model and framework grounded in both industrial practice and quality standards. The first contribution is a four-stage adoption model, denoted as the IMRM model (Innovate → considerations, Measure → metrics, Realize → values, Manage → improvements) that integrates early-stage risk assessment (e.g., legal, security, and licensing) andquality evaluation throughout the GenAI adoption and usage.The second contribution presents a detailed framework that connects risks andmetrics to concrete decision support, justifying the business value (e.g., quality gates) and technical trade-offs of GenAI solutions. The third contribution provides a structured mapping of GenAI quality to ISO/IEC 25010, 25023, and 25059 characteristics, attempting to ground practical evaluation needs within a standardized vocabulary. This thesis concludes that a structured quality evaluation process, which prioritizes risks and context, is a valuable approach intended to support building the business confidence required to leverage GenAI for efficient and effective software engineering in industry.

sted, utgiver, år, opplag, sider
Karlskrona: Blekinge Tekniska Högskola, 2026. s. 232
Serie
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 2026:01
Emneord
Quality Evaluation, Metrics, Artificial Intelligence, AI, Generative AI, Empirical Software Engineering
HSV kategori
Identifikatorer
urn:nbn:se:bth-28958 (URN)
Disputas
2026-01-29, J1630, Karlskrona, 13:00 (engelsk)
Opponent
Veileder
Tilgjengelig fra: 2025-12-08 Laget: 2025-12-03 Sist oppdatert: 2025-12-18bibliografisk kontrollert

Open Access i DiVA

Fulltekst mangler i DiVA

Andre lenker

Forlagets fulltekstScopus

Søk i DiVA

Av forfatter/redaktør
Yu, LiangAlégroth, EmilGorschek, Tony
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric

doi
isbn
urn-nbn
Totalt: 210 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf