Valid Citations, Unsupported Claims: What Fifteen Questions Revealed About Our RAG

Our citation validator accepted an answer that the cited pages did not support. In a 15-question prototype run on September 22, 2026, a patient-specific question produced a treatment recommendation with valid citation markers. The markers resolved to retrieved passages. The recommendation still went beyond those passages. We were building a professional learning platform for aesthetic medicine. Here we call the project Thalvera and its licensed reference book The Longhaven Handbook; both names are pseudonyms. This case concerns the behavior of that prototype, rather than the medical merits of its generated recommendation. We have omitted the recommendation itself.

The check we had actually implemented

The backend retrieved source fragments, assigned citation numbers and parsed the model's response. A citation was acceptable when its marker referred to a fragment supplied to the model. That established a useful structural property: the answer could not cite an arbitrary marker outside its retrieved context.

It did not establish that the fragment supported the sentence carrying that marker. Our internal result label, grounded, described the parser's outcome. Reading that label as a judgment about every claim in the answer would have given the implementation a capability it did not have.

Four questions hidden behind one citation badge

Check	What a pass establishes	What remains unresolved
Marker resolution	The marker identifies supplied context.	Whether that context supports the claim.
Retrieval relevance	The passage concerns the question's topic.	Whether it contains the requested fact.
Claim support	The passage provides evidence for the specific assertion.	Whether an answer is appropriate for the requested use.
Refusal behavior	The system declined this particular request.	Whether it consistently declines equivalent requests.

The failed response passed the first check. Its presentation made that success look much broader.

A bibliography became evidence for a recommendation

We inspected the passages behind the answer. Part of the apparent support came from a bibliography. Research titles mentioned a population and a technology; the model turned those titles into a claim about what treatment to choose. The source page existed, and the relevant words appeared on it. Their role on that page was wrong for the conclusion.

A reference list points readers toward other works. It does not, by itself, reproduce their methods or findings. Our ingestion path treated bibliographic text as ordinary page text, so retrieval did not preserve that distinction as an enforceable evidence boundary.

Other parts of the answer introduced treatments that our development record says were absent from the book. We cannot use the citation badge to explain away that addition. The prompt prohibited outside knowledge, yet that instruction did not prevent the observed response.

This also separates the case from our earlier citation UX study. That article, by Dmytro Nasyrov, examined fabricated citations in design mockups. Here a running backend returned real source locations attached to unsupported statements. Fixing placeholder content in a design would not fix this runtime failure.

What the 15 questions did and did not show

The run contained ten substantive book questions and several boundary probes. The substantive answers received citations to the expected sections in the recorded inspection. An unrelated bread-making question was rejected before a model call. A question about a topic missing from the book was rejected at retrieval as well.

There was also a successful model refusal: a request for device settings retrieved context, but the model declined to supply the requested parameters. The patient-specific question then received the unsupported recommendation. A single working refusal had not established a stable boundary between educational lookup and personalized advice.

These were selected development probes. We did not randomly sample users or independently adjudicate a representative clinical question set. Dividing the conspicuous failure by 15 would produce a number about this chosen set, not a defensible product hallucination rate.

The language probe exposed another failure upstream. The same book question scored 0.487 in Russian and 0.55 in English against a retrieval cutoff of 0.5. The Russian version was refused despite relevant material being present. Raising refusal frequency alone would therefore conceal a different defect: refusing an answer the corpus could support.

The evaluation record we needed next

For the next evaluation, the useful unit was a claim paired with the exact supporting passage. We would keep the question, retrieved context, answer span and citation mapping together, then ask an adjudicator whether the passage supported that span. Partial support would remain distinct from complete support.

A small fictional example shows the distinction without reproducing the licensed book:

Illustrative claim review, using invented text rather than a project output

Source text	Answer claim	Verdict
A bibliography entry names a study of Technique A.	Technique A should be selected for this individual.	The citation resolves, but this text does not support the recommendation.
The handbook describes a framework with four stages.	The framework contains four stages.	The narrow descriptive claim is supported.

That record would let us separate a missing passage from a generation error. It would also expose situations where a sentence combined a supported statement with an unsupported extension. Scoring the whole paragraph as either cited or uncited would lose that boundary.

Our September notes proposed stronger claim-to-source checks and a separate boundary for patient-specific questions. They do not establish that either proposal had already become a validated safeguard. An additional model could assist review, but its judgments would need their own labeled evaluation set.

A narrower contract for the citation indicator

For this prototype, a citation indicator meant that a source location was available to inspect. The engineering task was to preserve the location faithfully and make the distinction visible between original evidence and model-written text. The indicator could not substitute for claim adjudication.

Before reusing the validator as an acceptance gate, we would test bibliographies, partially supported sentences and equivalent requests with different wording. Retrieval misses and inappropriate answers need separate outcomes. The Russian question belongs in the former group; the patient-specific recommendation belongs in the latter.

That separation also shapes our test strategy for the platform: a deterministic stub can prove that citation numbers survive the API path. It cannot tell us whether the cited text supports a sentence. The same marker may pass both the stub test and the failed live answer.

Evidence scope: the September 22 prototype run and September 23 follow-up questions, summarized from our development records. The underlying licensed book and raw conversation are not redistributed. The fictional table above is explanatory, not an additional experiment. A current-version rerun and independently adjudicated claim set remain outstanding.

FAQ

Last updated: October 15, 2026

Quick answers to common questions about custom software development, pricing, process and technology.

Expand all Collapse all

Search FAQs

Type to filter questions and answers. Use Topic to narrow the list.

Showing all 14

• How long have you been in business?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has been in business since 2013, with over 13 years of experience in custom software development. During this time, we have delivered over 110 applications for 200+ clients across 28 industries, including FinTech, healthcare, crypto and e-commerce. We are rated 5/5 on Clutch based on 107 verified reviews (2026).

• What services do you provide?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production provides six core service categories: AI Development (LLM applications, AI agents, machine learning and computer vision), Blockchain and Web3 (smart contracts, DeFi, tokenization on Ethereum, Solana, TON and other chains), FinTech Development (payment platforms, digital banking, KYC and AML integration), Software Development (mobile apps, web platforms, database design, UI/UX), Security and QA (code audits, penetration testing, smart contract audits, manual and automated testing) and Consulting (architecture design, MVP validation, startup consulting).

• Where is Pharos Production located?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production is headquartered in Las Vegas, Nevada, USA (5348 Vegas Dr, Las Vegas, NV 89108), with an engineering office in Kyiv, Ukraine (44-B Eugene Konovalets Str. Suite 201, Kyiv 01133). We work with clients worldwide and provide remote collaboration across all time zones. Visit our contact page for directions and scheduling options.

• How big is your team?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has a team of 90+ engineers, including software developers, blockchain specialists, QA engineers, DevOps experts, UI/UX designers, project managers and solution architects. Our founder, Dr. Dmytro Nasyrov, holds a PhD in Artificial Intelligence and leads the technical direction of all projects.

• What types of customers do you serve?

  Copy link Copies a direct link to this answer to your clipboard.

  We serve a wide range of clients, from startups and product companies to mid-sized enterprises and large institutions. Our clients include crypto exchanges, FinTech providers (like Pleenk), healthcare organizations, sportsbook operators (like Pro Gambling), e-commerce platforms and SaaS companies. Pharos Production has worked with 200+ clients across 28 industries since 2013, adapting engagement models to match each client's stage, whether it is MVP validation for a startup or enterprise-scale development for an established business.

• How is a custom software development company different from an agency?

  Copy link Copies a direct link to this answer to your clipboard.

  A custom software development company like Pharos Production assigns one dedicated engineering team that owns your product across multiple release cycles, instead of handing it off once a contract ends. Agencies more often rotate staff between clients and optimize for fast delivery over long-term product ownership.

  Pharos Production keeps the same team on your codebase through 2-week sprints, so there is no re-onboarding cost each time you request a new feature.

• How much does custom software development cost?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom software development costs vary based on project scope and complexity. At Pharos Production, typical project ranges are: MVP development ($10,000-$25,000), suitable for startups validating a product idea; full-fledged production ($25,000-$50,000), for established businesses scaling a proven concept; and full-cycle development ($50,000-$80,000+), for complex enterprise-grade systems. These ranges include architecture design, development, QA testing and deployment. Hourly rates range from $50 to $99 depending on role and seniority. Final pricing depends on technology stack, number of integrations and engagement model (staff augmentation, dedicated team or project outsourcing).

• How long does it take to develop a custom software application?

  Copy link Copies a direct link to this answer to your clipboard.

  Development timelines depend on scope and complexity. At Pharos Production, a typical MVP takes 2-4 months, a production-ready application takes 4-8 months and a complex enterprise system can take 8-12+ months. We use an agile methodology with 2-week sprints, delivering working increments after each sprint. Every sprint includes a retrospective, progress report and planning session for the next iteration. This approach ensures transparency and allows businesses to launch faster by prioritizing high-impact features first. Get a timeline estimate for your project.

• What industries does Pharos Production serve?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production serves 28 industries: FinTech, MiCA Compliance, Banking, Insurance, Crypto, Healthcare, Fitness, E-Commerce, Food & Restaurant, On-Demand, Manufacturing, Automotive, Energy, Oil & Gas, Supply Chain, Construction, Real Estate, Gaming, Streaming, Media, Telecom, Education, Aviation, Travel, Logistics, Marketing, Agriculture and Recommender Systems. Our deepest expertise is in FinTech, blockchain and healthcare, where we have delivered compliance-ready platforms (HIPAA, PCI DSS, GDPR) and high-load systems handling thousands of concurrent users. For the latest industry insights, read our guides on FinTech trends in 2026 and the Web3 technology stack.

• Why hire a software development company instead of building in-house?

  Copy link Copies a direct link to this answer to your clipboard.

  Hiring a software development company offers faster time-to-market, lower upfront costs and access to specialized expertise without long-term employment commitments. According to Deloitte's 2024 Global Outsourcing Survey, 57% of companies outsource software development to access skills they cannot hire internally.

  Factor	In-house team	Software development company
  Time to assemble	3-6 months (recruiting + onboarding)	1-2 weeks
  Upfront cost	High (salaries, benefits, equipment)	Lower (project-based pricing)
  Specialized expertise	Limited to who you can hire locally	Access to 90+ engineers across blockchain, AI, FinTech
  Scalability	Slow (each new hire takes months)	Fast (scale up or down per sprint)
  Long-term commitment	Full-time employment contracts	Flexible engagement models
  Risk	High if key engineers leave	Company ensures continuity and knowledge transfer

  For businesses that need blockchain, AI or high-load architecture expertise, outsourcing to a specialized firm like Pharos Production reduces risk and accelerates delivery.

• What projects does Pharos Production not take on?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production focuses on mid-to-large custom software projects with budgets starting at $10,000. We do not take on template-based websites, WordPress theme customization or short-term contracts under one month. We also do not provide non-technical staffing (marketing, sales or design-only roles). Our strongest fit is blockchain, FinTech and healthcare projects where security, compliance and high-load architecture are critical. For smaller projects or MVPs under $10,000, we recommend exploring freelance platforms or no-code tools as a more cost-effective starting point.

• Why does Pharos Production use agile methodology?

  Copy link Copies a direct link to this answer to your clipboard.

  We use agile with 2-week sprints because it reduces the risk of building features that miss the mark. Each sprint ends with a working demo, a retrospective and a plan for the next iteration.

  This means clients see progress every 14 days and can adjust priorities based on real results, not assumptions. We chose this approach after years of experience showing that rigid, fixed-scope contracts lead to scope creep, missed deadlines and products that do not match market needs by launch day.

• When should you NOT hire a custom software development company?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom development is not the right choice in every situation. You should not hire a custom software company if: your problem is fully solved by an existing SaaS product (e.g. Shopify for e-commerce, Salesforce for CRM); your budget is under $10,000 and timeline is under 4 weeks; you need a simple landing page or marketing website (WordPress or Webflow is faster and cheaper); or you are still validating the idea and have not spoken to potential users yet.

  In these cases, off-the-shelf tools or no-code platforms offer better ROI. Custom development makes sense when you need unique workflows, regulatory compliance, high-load architecture or competitive differentiation that packaged software cannot provide.

• What kind of projects has Pharos Production delivered?

  Copy link Copies a direct link to this answer to your clipboard.

  Here are three anonymized examples from our recent delivery history:

  FinTech startup - payment platform (MVP)
  Scope: mobile app + backend API with bank-grade encryption. Team: 4 engineers, 1 QA. Timeline: 10 weeks. Budget: $38,000. Result: launched on schedule, processed $2M+ in transactions within the first quarter.

  Healthcare provider - patient portal (Full product)
  Scope: HIPAA-aligned web platform with EHR integration, appointment scheduling and telemedicine. Team: 6 engineers, 1 DevOps, 2 QA. Timeline: 6 months. Budget: $120,000. Result: 15,000+ active patients, zero compliance violations in two annual audits.

  Crypto exchange - trading engine (Complex)
  Scope: high-load matching engine handling 50,000+ orders per second, multi-chain wallet infrastructure on Ethereum and Solana. Team: 8 engineers, 2 QA, 1 security auditor. Timeline: 11 months. Budget: $340,000. Result: 99.97% uptime, passed three independent security audits.

  See more projects: NoMoreBets, Pulse, Sagas, Gambit Stream and Pleenk. For the full portfolio, visit our case studies. Learn more about the technology behind these projects in our guide to stablecoins and crypto infrastructure.

Dmytro Ihnatov is the Lead Backend Developer at Pharos Production. He documents backend architecture and research experiments from the team's work on a professional learning platform for aesthetic medicine.