Conversational Decision Intelligence

1. Context and Motivation

1.1 The Problem

Decision makers and data scientists need to explore data quickly. Typical questions span a wide range: "What drove churn last quarter?", "Compare regions by revenue and show the top three," or "Summarize what these contracts say about renewal terms." Human experts rarely arrive at the correct answer in a single attempt; they form a plan, try something, examine the result, and then refine their approach or seek clarification. Naive system designs that rely on a single prompt or a rigid script ignore this reality. They fail when questions are ambiguous, when data spans multiple sources, or when the first attempt produces a schema or logic error. Users lose trust and abandon the tool.

A robust conversational decision-intelligence system must therefore address five interrelated requirements:

• Plan before acting, so that multi-step questions receive a coherent sequence of actions rather than speculative single-shot guesses.
• Reason at each step and adapt when observations contradict the plan.
• Recover from errors, such as a missing column or an empty result, rather than repeating the same mistake.
• Explain how answers were derived, surfacing assumptions, definitions, and limitations.
• Improve with use, so that repeated or similar questions benefit from past context and user corrections.

1.2 Why Conversation Flow Matters

The quality of an answer depends as much on the process used to produce it as on the underlying language model. A well-designed flow aligns with how experts operate and thus delivers three properties that matter in practice.

Efficiency. Planning reduces exploratory thrashing. Clarification (for example, "which time range?" or "which segment?") avoids wrong assumptions and repeated work. Memory and feedback shorten the path on subsequent similar questions. The system does not guess when intent is ambiguous; it asks, as a competent expert would.

Reliability. Each step is grounded in real tools and schema, or in document and API metadata for unstructured sources, reducing hallucination. When a step fails, reflection and validation allow the agent to correct course rather than repeating the error. The loop is predictable and traceable.

Validity. Reasoning is surfaced so that users can assess trustworthiness and reproduce the analysis. Assumptions, definitions, and limitations are stated explicitly. For high-stakes or verification-style questions, a critic-and-revision mechanism provides an additional quality check, in the same way that peer review does for expert work.

2. Foundations and Inspiration

The methodology described in this paper does not arise from first principles alone. It deliberately integrates a body of prior research on reasoning, tool use, and self-correction in language models, repurposing those ideas in service of a human-centred conversation flow. This section situates the methodology in that context and explains why a linear plan-and-execute design was chosen over a graph-based orchestration framework.

2.1 Frameworks Integrated

ReAct (Reason and Act)

ReAct, introduced by Yao and colleagues [1], synergizes reasoning and acting by interleaving short "thought" steps with tool calls and observations before the model decides its next action. This keeps the agent grounded in real outcomes and reduces hallucination. The execution loop described in Section 4 is explicitly ReAct-style: the model outputs a thought, then either a tool invocation or a final answer, and receives an observation. The same pattern applies across all flow variants by varying the tool set and system prompt.

Reflexion

Reflexion, introduced by Shinn and colleagues [2], gives an agent a verbal reflection after failure: a critique of what went wrong and what to try next, which then informs the next action. This paper integrates a version of that idea by running a separate, focused model call when the previous step clearly indicates failure (for example, an error message or an unexpectedly empty result). The reflection text is injected into the agent context before the next action is chosen. Reflection is triggered only on failure, so reliability gains are achieved without adding cost to every step.

Plan and Execute

Decomposing a task into a plan before executing it has a long history in both reasoning research and agent design. A planner produces an ordered list of tool invocations given the question and schema. The agent then runs a ReAct loop that can follow or deviate from that plan based on observations. The plan provides a strong prior that reduces thrashing and keeps multi-step questions coherent, while the loop retains the flexibility required for real-world recovery.

Structured Output and Schema Validation

Production agents require reliable parsing of model output. Every step, including thoughts, actions, inputs, and final answers, is validated against a fixed schema. Invalid output is caught at the boundary; the system falls back or retries instead of silently misinterpreting the reply. This is standard practice for robustness and observability, applied consistently at each step of the loop.

Pre-execution Validation and Post-answer Debate

Two targeted mechanisms extend the reflexion idea. Before executing a query after a prior failure, a lightweight validator can approve or reject the proposed query, allowing the agent to revise without re-executing. For questions that explicitly request verification ("double-check," "audit"), a critic reviews the final answer and a revision pass produces an updated response. These mechanisms draw on broader work on self-critique and debate in language models [3]. Restricting them to specific triggers keeps additional cost proportional to need.

Recursive Language Model (RLM)

When context (for example, schema and knowledge-base summaries for many sources) exceeds what a single prompt can hold reliably, an RLM-style approach is used [4]. The long context is placed in a controlled environment; the model can inspect it, slice it, and invoke a sub-model on chunks, then aggregate the sub-answers and return a final result. This trades additional steps and sub-calls for the ability to reason coherently over very large context.

Long-term Memory and Feedback

Rather than relying on a single framework, the methodology combines several established ideas: vector stores and similarity search for retrieval-augmented context, feedback labels (acceptable, unacceptable, corrected) applied at retrieval time so the system prefers the user's version of the truth, and running summaries and persona updated over time. Together these enable continuous improvement without retraining.

2.2 Why a Graph-based Framework Was Not Adopted

Graph-based orchestration frameworks such as LangGraph [7] allow agents to be built as explicit state machines with nodes, edges, cycles, and checkpointing. They are well suited to flows with many branching paths, multiple coordinating agents, or rich human-in-the-loop routing.

This methodology instead uses a linear plan-and-execute loop with conditional branches inside the loop: reflection on failure, validation before a retry query, and debate after a candidate final answer. Four reasons drove that choice.

• Flow shape. The conversation flow is fundamentally a single trajectory: plan once, then step through a reasoning loop until a final answer or a clarification request is issued. Expressing this as a graph would add nodes and edges without clarifying the control path.
• Trigger control. Precise control is needed over when reflection, validation, and debate run, triggered by simple conditions on the current step or the user message. Those checks live directly in the loop, making it straightforward to tune triggers without redefining a graph.
• Streaming and resumption. The loop streams tokens and step updates to the user and supports resumable clarification: the agent asks a question, the conversation pauses, and the loop resumes with full step history when the user replies. A linear loop handles this naturally.
• Interpretability. A loop with a fixed number of conditional branches is easy to trace and log. The current phase — planning, step N, reflection, validation, or debate — is always unambiguous. This supports debugging and audit requirements.

3. Design Principles

The methodology rests on three principles, each chosen to align with how experts think and how people typically work. Together they deliver efficiency, reliability, and validity.

Principle 1: Plan Before Acting

Experts do not issue random queries; they form a hypothesis and then execute. The system produces an ordered plan (which sources, which operations, in what sequence) given the question and the current view of data and schema, or file listings and API metadata for unstructured sources. That plan is a prior: the execution loop can follow it or deviate when observations justify a change. Planning reduces thrashing, keeps multi-step questions coherent, and makes the process efficient and predictable.

Principle 2: Reason in Steps

Humans explore iteratively: run something, examine the result, then drill down or pivot. The execution loop mirrors that pattern. At each step the model reasons briefly, chooses an action (a tool call or a final answer), and receives an observation. It then decides the next step. This reason-then-act pattern keeps the model grounded in real outcomes and allows it to correct course when something fails or when the user's intent becomes clearer.

Principle 3: Learn from Interaction

Practitioners improve with experience; they remember what worked, avoid past mistakes, and adopt the definitions and preferences of their context. The system is not stateless. It maintains long-term memory of past question-and-answer pairs and incorporates user feedback. Retrieval at query time surfaces relevant past context so that similar or follow-up questions benefit from prior learning, without requiring any retraining of the underlying language model.

4. Architectural Overview

Each user message passes through five stages. The pipeline is designed so that the path from question to answer mirrors the way a human expert would approach the same problem.

4.1 Pipeline Stages

• Context assembly. The system gathers the current conversation (recent turns), the set of connected data sources and their schema or metadata, and, when available, long-term memory and any stored persona or conversation summary. This stage applies equally whether the user is querying a database, reading documents, or calling an external API.
• Planning. A language model maps the user question and recent context to an ordered list of tool invocations. The plan is grounded in the same tools and schema that the execution layer will use.
• Execution and reasoning. An agent loop runs the plan step by step. At each step the model outputs a thought and either a tool call or a final answer. Tool results are returned as observations. The loop continues until the model produces a final answer, a clarification request is issued, or a safety limit is reached.
• Synthesis. Tool outputs are summarized into a coherent answer. A methodology note (assumptions, definitions, considerations, limitations) may be attached so the user understands how the answer was derived.
• Memory and learning. The exchange is stored in long-term memory. If the user provides feedback, it is recorded and applied at retrieval time so future context reflects the user's judgment.

4.2 The Execution Loop in Detail

Inside the execution phase, the agent repeatedly reasons and acts. The following describes the three views of the loop: the core cycle, the tool-call path (including optional reflection and validation), and the final-answer path (including optional critic and revision).

Core Cycle

Each iteration produces a thought, then an action. For a tool call, the system optionally runs reflection (when the previous step failed) and validation (before running a query after a failure), then executes the tool and receives an observation. For a final answer, an optional debate pass runs for verification-style questions, and the loop exits.

Reflection (on failure)

When the previous observation indicates failure (an error, a missing column, or an unexpectedly empty result), a short, separate model call asks what went wrong and what alternative to try. The reflection is injected into context before the next thought. This mirrors how humans recover: they pause after an error, reason about the cause, and revise their approach.

Validation (before a retry query)

When the next action is a query and the previous step failed, a lightweight validator reviews the proposed query and the prior error, returning either approval or a brief issue description. If an issue is found, the agent sees it as an observation and can revise without executing. This reduces repeated execution errors and saves time.

Critic and Revision (for verification questions)

For questions that explicitly request verification ("double-check," "audit," "validate"), a critic reviews the candidate final answer for gaps or unclear reasoning. A revision pass then produces an updated answer. This mechanism adds validity control where it matters most without adding overhead to every turn.

5. Building Blocks in Practice

5.1 The Tool-calling Model

The agent operates through a registry of tools, each with an identifier, a description, and an input schema. Invocations specify a tool name and input parameters; an executor runs the tool and returns an observation. This pattern is conceptually aligned with the Model Context Protocol (MCP) [5], which standardizes how artificial intelligence applications connect to external tools and data sources.

Tools typically include data access for structured data (SQL over relational databases, CSV files) and unstructured data (file readers, API fetches, document content), computation (Python over result sets), visualization, and an explicit "ask user" action for clarification. The same loop pattern can be specialized for a particular use case by restricting the tool set and adjusting the system prompt.

5.2 Handling Very Large Context

When the total context (schema and knowledge-base summaries across many sources) exceeds what a single prompt can hold reliably, the system adopts a recursive strategy: the long context is made available to the model in a controlled environment, the model inspects and slices it, invokes a sub-model on chunks, and aggregates sub-answers into a final response. The trade-off is additional latency and cost in exchange for coherent reasoning over very large context.

5.3 Worked Example: Happy Path

The user asks: "What were total sales last quarter?"

• Context assembly: recent turns if any, the sales database, schema summary (table: sales; columns: amount, created_at), and any relevant past question-and-answer pairs from memory.
• Planning: a single step, which is to run a SQL query summing amount where created_at falls in the previous quarter.
• Execution: the agent runs the query and receives a result. It then issues a final response with a brief methodology note (assumption: last quarter equals the previous calendar quarter; definition: total equals sum of amount).
• Memory: the exchange is stored. No reflection, validation, or debate is triggered.

5.4 Worked Example: Recovery Path

The user asks: "Show me revenue by region." The schema contains a table sales with columns region_id, amount, and date. The agent mistakenly queries a column named region that does not exist.

• Step 1: the agent queries GROUP BY region and receives an error indicating that the column does not exist.
• Reflection: the reflection pass identifies the error and suggests inspecting the schema for the correct column name.
• Step 2: the agent calls get_schema("sales") and receives the column list.
• Step 3: the agent proposes a query using GROUP BY region_id. The validator approves it. The query runs and returns five rows.
• Step 4: the agent issues a final response with the summary and an optional chart.

5.5 Worked Example: Clarification and Resumption

The user asks: "Compare performance across segments." The agent does not know which segments are intended. It issues an ask-user action: "Which segments do you mean: by product line, by region, or by quarter?" The conversation pauses. When the user replies "By quarter," the flow resumes with the same plan and step history, the agent runs queries grouped by quarter, and it returns a comparison. This illustrates how the loop supports multi-turn clarification without losing context.

6. Continuous Improvement Without Retraining

The flow is designed so that the system improves with use in the same way that human practitioners do, through experience, corrections, and accumulated context. Learning is achieved through context and retrieval policy rather than model retraining, which yields improvements in efficiency, reliability, and validity over time.

6.1 Long-term Semantic Memory

After each turn, a compact takeaway (the user question plus a brief assistant summary) is stored in a vector index keyed by workspace and optionally by user. The summary is intentionally brief so that memory captures experience and learnings rather than full analyses; the system is designed to run fresh analysis on current data when using memory for context.

At query time, the user's message is embedded and compared to past question-and-answer pairs. The most similar entries are retrieved and injected into the agent context. Follow-up or similar questions ("same as last time but for Q3") then benefit from prior context. Index size can be capped so that memory does not grow unbounded.

6.2 Feedback and Corrected Content

Users can provide feedback on any turn: acceptable, unacceptable, or corrected with replacement text. At retrieval time, entries marked as unacceptable are excluded from context; entries marked as corrected are surfaced with the user's corrected text in place of the original. Future context therefore reflects the user's version of the truth, and the system learns from corrections without changing model weights.

6.3 Running Summaries and Persona

Two higher-level structures support long-term adaptation. A running conversation summary is maintained by a language model and updated after each turn; it is injected into the agent context so the agent has a compact view of what has been discussed without re-reading every message. At the workspace or user level, a persona (role, priorities, and context inferred from interaction history) is updated periodically and used to tailor tone and emphasis. Both structures are optional and their value grows with conversation length.

6.4 Schema and Knowledge Base

Each connected data source carries a knowledge base: for structured sources, table and column lists; for unstructured sources, file listings, document indexes, or API annotations. The planner and agent see a compact view of the relevant knowledge base for each question. When retrieval is enabled, only the portion relevant to the current question is injected, keeping prompts smaller while remaining accurate. As sources are refreshed, the system's view of available data remains current, supporting reliable discovery and repeat analysis.

7. Practitioner Checklist

Teams building or evaluating conversational analytics and decision-intelligence systems can use the following questions as a design checklist. The central lens is whether the flow aligns with how experts think and how people work, and whether it delivers efficiency, reliability, and validity.

Planning and Reasoning

• Is an explicit plan produced before execution, grounded in the same tools and schema the agent uses?
• Is a reason-and-act loop used with real observations, with context size managed to avoid overflow?
• Can the agent issue a clarification request rather than guessing when intent is ambiguous?

Recovery and Quality Control

• Is reflection triggered when the previous step clearly indicates failure?
• Is a validation pass available before re-executing a query after a failure?
• Is a critic-and-revision pass available for questions that explicitly request verification?
• Is model output validated against a schema at each step?

Scale and Data Scope

• Is there a strategy for very large schema or many sources (retrieval, chunking, or recursive sub-calls)?
• Does the flow support both structured and unstructured data in a single pipeline?

Learning and Alignment

• Are past question-and-answer pairs stored in a searchable form and injected as relevant context?
• Is user feedback (acceptable, unacceptable, corrected) applied at retrieval time?
• Are running summaries or a persona maintained where they add value to long conversations?

8. Conclusion

Conversational decision intelligence requires a conversation flow that plans, reasons step by step, recovers from failure, and improves with use. This paper has outlined a methodology that combines plan-and-execute and reason-act loops for coherent, grounded behavior; reflection on failure and validation or debate where quality matters most; strict output schemas for robustness and observability; recursive or chunked strategies for very large context; and long-term memory, feedback-aware retrieval, summaries, and schema awareness for continuous improvement without retraining.

The design is built with a human-centred approach in mind. It mirrors how experts think (planning before acting, reasoning in steps, recovering from errors, documenting assumptions) and how humans typically work (iterating, clarifying when ambiguous, learning from feedback). The result is a system that delivers efficiency through fewer wasted steps and faster paths to insight; reliability through grounding in real data and recovery via reflection and validation; and validity through explicit methodology and targeted verification.

The same flow applies to structured and unstructured data, and the ideas are transferable to any assistant that answers natural language over connected data. Practitioners are encouraged to adapt these principles to their own constraints and to share experience so the field can continue to mature.

Notes and References

1. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations (ICLR). arXiv:2210.03629.
2. Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. (2023). "Reflexion: Language Agents with Verbal Reinforcement Learning." Advances in Neural Information Processing Systems (NeurIPS). arXiv:2303.11366.
3. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. (2023). "Improving Factuality and Reasoning in Language Models through Multiagent Debate." arXiv:2305.14325. Follow-on work on critic-and-revision patterns includes a range of literature on self-critique and constitutional methods.
4. "Recursive Language Models" for long-context reasoning. arXiv:2512.24601. The approach places long context in a controlled environment, allows the model to slice it, and aggregates sub-model results.
5. The tool-calling design used here is conceptually aligned with the Model Context Protocol (MCP), an open protocol for connecting AI applications to external tools and data sources. See modelcontextprotocol.io and Anthropic's MCP documentation for implementation details. This paper does not describe an MCP implementation; the flow can be realized with an in-process registry or with MCP-compatible servers.
6. Wang, L., Ma, C., Feng, X., et al. (2024). "A Survey on Large Language Model based Autonomous Agents." Frontiers of Computer Science. arXiv:2308.11432. Provides a broad synthesis of plan-and-execute and task decomposition approaches.
7. LangChain (2024). LangGraph: Build Stateful, Multi-Actor Applications with LLMs. langchain-ai.github.io/langgraph. The design trade-offs discussed in Section 2.2 are specific to the flow described in this paper and do not imply that graph-based frameworks are unsuitable for other agent architectures.

Sandeep Kavety (2026). Conversational Decision Intelligence: Flow, Reasoning, and Continuous Learning in Natural Language Systems over Structured and Unstructured Data [Working paper]. https://sandeepkavety.com/writing/conversational-decision-intelligence