Mastering Reasoning with Code: Chain of Thought Pipelines in Java

Build smarter, traceable AI workflows using LangChain4j, Quarkus, and structured step-by-step logic

Large Language Models are impressive, but unpredictable. Give them a task and they might produce brilliance or utter nonsense. If you're a Java developer integrating AI into your application, that variability can quickly become a liability. That’s where structured reasoning patterns like “Chain of Thought” (CoT) pipelines come in.

This article walks you through a full implementation of a CoT pipeline using Java, powered by Quarkus and LangChain4j. We'll break down the components, explain the reasoning model, and show how modular step-based logic helps you tame the chaos of LLMs.

Why is this even relevant?

Smaller, local language models often struggle with complex problems because they cannot process or retain enough information at once. Their context windows are typically limited to 4,000 to 8,000 tokens, which may not be enough to cover the full problem or all relevant background. Even if the context fits, these models tend to lose track of earlier information as attention shifts toward newer input. Most have not been trained extensively on tasks that require long-context reasoning or retrieval. With limited capacity and less robust memory, they often produce shallow or inconsistent answers when deeper analysis is needed.

What Is a Chain of Thought Pipeline?

A Chain of Thought (CoT) pipeline breaks down reasoning into clearly defined, sequential steps. Each step has its own role-specific prompt, its own goal, and its own validation logic. This allows you to:

• Add structure to free-form language model behavior.

• Validate intermediate results and context.

• Debug and log reasoning failures with precision.

• Improve reliability and maintainability.

Unlike a single prompt that asks the model to "solve this step by step," a CoT pipeline enforces structure by guiding the model through each stage of the reasoning process explicitly.

You can now “trace” the model's thinking, inspect intermediate outputs, and react to failures in a controlled way. It builds context incrementally, making the reasoning process transparent and easier to debug.

In contrast, a single step-by-step prompt depends entirely on the model to self-manage the process. If the result is wrong, you have no insight into which part failed or why. A pipeline gives you that control, one prompt at a time.

Why Not Just Use an Agent?

Agents are powerful but often overkill. They introduce complexity: tool calling, planning loops, action management, retry policies. For many use cases, especially domain-specific reasoning or business process automation, a deterministic chain of prompts is more than enough. It's transparent, debuggable, and easier to govern.

Let’s walk through how this works in Java.

Implementing a Chain of Thought Pipeline in Java

The full implementation includes:

• A ChainOfThoughtPipeline class that orchestrates steps.

• Modular ReasoningStep subclasses for each phase (e.g. analysis, planning, execution, verification).

• Integration with LangChain4j and a local model via Ollama.

• Context tracking and structured result handling.

• A ProblemSolverService that builds the chain and makes it available to a rest endpoint.

I did not create a fancy interface this time. Instead, you get to play with curl or just use your browser to look at the response JSON. The full code is in my Github repository. We’ll also walk through the classes on a high level only today.

Step 1: Define the Pipeline Class

The ChainOfThoughtPipeline class is a Quarkus CDI bean that implements a chain-of-thought reasoning pattern for problem-solving. It orchestrates a sequence of reasoning steps to solve complex problems by breaking them down into smaller, manageable steps.

@ApplicationScoped
public class ChainOfThoughtPipeline {
    private final ChatLanguageModel chatModel;
    private final List<ReasoningStep> steps = new ArrayList<>();
    private final Map<String, Object> context = new HashMap<>();
    private final List<StepResult> executionHistory = new ArrayList<>();

    public ChainOfThoughtPipeline(ChatLanguageModel chatModel) {
        this.chatModel = chatModel;
    }

    public ChainOfThoughtPipeline addStep(ReasoningStep step) {
        steps.add(step);
        return this;
    }

    public ChainOfThoughtResult execute(String input) {
        String currentInput = input;
        context.put("original_input", input);

        for (ReasoningStep step : steps) {
            try {
                StepResult result = step.execute(currentInput, context, chatModel);
                executionHistory.add(result);
                if (!result.isSuccess()) break;
                currentInput = result.getOutput();
            } catch (Exception e) {
                executionHistory.add(new StepResult(
                    step.getName(), currentInput, "", e.getMessage(), false
                ));
                break;
            }
        }

        boolean success = executionHistory.stream().allMatch(StepResult::isSuccess);
        return new ChainOfThoughtResult(success, currentInput, executionHistory);
    }
}

Key Components:

• ChatModel: Uses LangChain4J's chat model for AI-powered reasoning

• ReasoningStep list: Maintains an ordered sequence of steps to execute

• Context map: Stores shared data between steps

• Execution history: Tracks the results of each step for debugging/analysis

Reasoning Steps: Modular, Focused, Traceable

Each ReasoningStep subclass implements one phase of the problem-solving journey. The ReasoningStep class is an abstract base class that represents a single step in a Chain of Thought reasoning pipeline. It defines the contract for how each step should behave when executed using a language model like OpenAI or Ollama via LangChain4j.

Step 2: Analyze the Problem

class ProblemAnalysisStep extends ReasoningStep {
    public ProblemAnalysisStep() {
        super("Problem Analysis", "Decompose the problem into key facts and unknowns.");
    }

    @Override
    public StepResult execute(String input, Map<String, Object> context, ChatLanguageModel model) {
        PromptTemplate template = PromptTemplate.from("""
            Analyze this problem step by step:
            Problem: {{problem}}
            1. Key facts
            2. Unknowns
            3. Constraints
            4. Problem type
        """);

        String response = model.generate(template.apply(Map.of("problem", input)).text());
        context.put("problem_analysis", response);
        return new StepResult(name, input, response, "Analyzed problem successfully", true);
    }
}

Each step logs reasoning into the context for reuse by later steps.

Step 3: Plan the Solution

class SolutionPlanningStep extends ReasoningStep {
    public SolutionPlanningStep() {
        super("Solution Planning", "Generate a plan to solve the problem.");
    }

    @Override
    public StepResult execute(String input, Map<String, Object> context, ChatLanguageModel model) {
        String analysis = (String) context.get("problem_analysis");

        PromptTemplate template = PromptTemplate.from("""
            Based on this analysis:
            {{analysis}}
            Generate a step-by-step solution plan.
        """);

        String response = model.generate(template.apply(Map.of("analysis", analysis)).text());
        context.put("solution_plan", response);
        return new StepResult(name, input, response, "Created a solution plan", true);
    }
}

Step 4: Execute the Plan

class SolutionExecutionStep extends ReasoningStep {
    public SolutionExecutionStep() {
        super("Solution Execution", "Perform calculations and solve.");
    }

    @Override
    public StepResult execute(String input, Map<String, Object> context, ChatLanguageModel model) {
        String plan = (String) context.get("solution_plan");
        String originalProblem = (String) context.get("original_input");

        PromptTemplate template = PromptTemplate.from("""
            Problem: {{problem}}
            Plan: {{plan}}
            Execute step by step, showing calculations and intermediate results.
        """);

        String response = model.generate(template.apply(Map.of(
            "problem", originalProblem,
            "plan", plan
        )).text());

        context.put("solution_execution", response);
        return new StepResult(name, input, response, "Executed the plan", true);
    }
}

Step 5: Verify the Answer

class VerificationStep extends ReasoningStep {
    public VerificationStep() {
        super("Solution Verification", "Review and validate the answer.");
    }

    @Override
    public StepResult execute(String input, Map<String, Object> context, ChatLanguageModel model) {
        String solution = (String) context.get("solution_execution");
        String originalProblem = (String) context.get("original_input");

        PromptTemplate template = PromptTemplate.from("""
            Verify this solution:
            Problem: {{problem}}
            Solution: {{solution}}
            - Does it solve the problem?
            - Are calculations correct?
            - Is it reasonable?
        """);

        String response = model.generate(template.apply(Map.of(
            "problem", originalProblem,
            "solution", solution
        )).text());

        return new StepResult(name, input, response, "Verified solution", true);
    }
}

Declarative AI Services with LangChain4j

For this example, I am using the simple and declarative Quarkus way. The model does not get a lot of information. I am even skipping user or system prompts here.

@RegisterAiService
public interface MathProblemSolver {

    String solveWithSteps(String problem);
}

I am using this model in two ways in the ProblemSolverService

Solving the Problem in a Structured Way

The ProblemSolverService class is a Quarkus service that configures and uses the chain-of-thought pipeline to solve math problems. It acts as a high-level service that orchestrates problem-solving using a predefined sequence of reasoning steps.

This service essentially provides a "math problem solver" that follows a structured thinking process, similar to how a human might approach a complex math problem step-by-step.

@ApplicationScoped
public class ProblemSolverService {

    // The pipeline
    @Inject
    ChainOfThoughtPipeline pipeline;

    @Inject
    ChatModel mathProblemSolver;


    public ChainOfThoughtResult solve(String input) {
        pipeline = new ChainOfThoughtPipeline(mathProblemSolver)
            .addStep(new ProblemAnalysisStep())
            .addStep(new SolutionPlanningStep())
            .addStep(new SolutionExecutionStep())
            .addStep(new VerificationStep())
            .addStep(new FinalAnswerStep());
        
        return pipeline.execute(input);
    }
}

Key Components:

• @ApplicationScoped: CDI bean with application-wide scope

• ChainOfThoughtPipeline: Injected pipeline for step execution

• ChatModel: AI model specifically configured for math problem solving

Wiring it all together

The MathResource class is a REST endpoint that provides a web API for solving math problems using chain-of-thought reasoning. It exposes HTTP endpoints for users to submit math problems and get AI-powered solutions in three ways.

Key Endpoints:

• GET /ask/{problem}: Returns detailed JSON response with step-by-step reasoning process

• GET /ask/{problem}/simple: Returns only the final answer as plain text

• GET /ask/simple/{problem}: Uses a single LLM call (bypasses chain-of-thought)

@Path("/ask")
public class MathResource {

    @Inject
    ProblemSolverService problemSolverService;

    @Inject
    MathProblemSolver mathProblemSolver;

    @GET
    @Path("/{problem}")
    @Produces(MediaType.APPLICATION_JSON)
    public ChainOfThoughtResponse solveProblem(@PathParam("problem") String problem) {
        ChainOfThoughtResult result = problemSolverService.solve(problem);
        return new ChainOfThoughtResponse(problem, result);
    }

    @GET
    @Path("/{problem}/simple")
    @Produces(MediaType.TEXT_PLAIN)
    public String solveProblemSimple(@PathParam("problem") String problem) {
        ChainOfThoughtResult result = problemSolverService.solve(problem);
        return result.getFinalResult();
    }

    @GET
    @Path("/simple/{problem}")
    @Produces(MediaType.TEXT_PLAIN)
    public String solveWithOneLLMCall(@PathParam("problem") String problem) {
        return mathProblemSolver.solveWithSteps(problem);
    }
}

What Kind of Examples are solved with CoT?

Some problems are famously deceptive when tackled with intuition alone. They appear simple on the surface but contain statistical traps, conditional probabilities, or shifting perspectives that often mislead even experienced thinkers. These are exactly the kinds of problems that benefit from a structured Chain of Thought (CoT) pipeline. Rather than asking a model to blurt out a single answer, a CoT approach guides it through the logic. Step by step, subgroup by subgroup, or probability by probability

The Monty Hall Problem
One-shot: "There are 3 doors. Behind one is a car, behind the others are goats. You pick door 1. The host opens door 3, revealing a goat. Should you switch to door 2?"
CoT version: "Let's think step by step about the probabilities at each stage..."
Correct answer: Yes, you should switch. Switching gives you a 2/3 probability of winning, while staying gives you 1/3.

The Birthday Paradox
One-shot: "In a room of 23 people, what's the probability that at least two people share the same birthday?"
CoT version: "Let's calculate this step by step using the complement probability..."
Correct answer: About 50.7% (surprisingly high for most people's intuition)

Simpson's Paradox Example
One-shot: "Treatment A has a 78% success rate overall, Treatment B has 83%. Which is better?"
CoT version: "Let's examine the data by subgroups to check for confounding variables..."
Correct answer: It depends on the subgroups. Treatment A could actually be better for both severe and mild cases individually, even with a lower overall rate due to treating more severe cases.

The Base Rate Fallacy (Medical Test)
One-shot: "A disease affects 1% of the population. A test is 90% accurate. If you test positive, what's the probability you have the disease?"
CoT version: "Let's use Bayes' theorem step by step with concrete numbers..."
Correct answer: Only about 8.3% (much lower than most people guess due to false positives)

The Secretary Problem (Optimal Stopping)
One-shot: "You're interviewing 100 candidates sequentially. You must hire immediately or reject forever. What's the optimal strategy?"
CoT version: "Let's derive the optimal stopping rule using the 37% rule..."
Correct answer: Reject the first 37 candidates, then hire the next candidate who is better than all previous ones. This gives you about a 37% chance of hiring the best candidate.

Final Thoughts

Structured CoT pipelines give you surgical control over AI reasoning. Instead of one massive prompt, you build a modular process: analyze → plan → solve → verify. Each step is transparent, testable, and observable. This is critical for regulated domains, enterprise workflows, and debugging gnarly model behavior.

With LangChain4j and Quarkus, Java developers finally have the tools to build these pipelines cleanly and natively and with no Python detour.