What Our RAG Test Doubles and Synthetic Corpora Could Actually Prove

A 31.3-second audio file gave us an end bookmark at 43.1 seconds. The speech model had not processed it. Our synthetic test generator was already reporting the wrong time.

We found this while preparing known-answer recordings for a learning platform with retrieval over documents and media. Synthetic material was supposed to remove uncertainty about the reference answer. Instead, it introduced a separate component that needed checking before we could use it to judge the product.

The reference clock was wrong

On October 6, 2026, our Windows SAPI checks used Microsoft Zira Desktop. When the generator wrote a 22.05 kHz WAV, bookmark positions arrived about 1.378 times too late. The recorded byte-rate mismatch was 44,100 against 32,000 bytes per second. At 16 kHz, 16-bit mono, the final bookmark matched the WAV duration to the millisecond on that machine. We checked intermediate boundaries as well. Synthesizing only as far as selected question markers produced files whose lengths matched those marks. A final-duration assertion alone could have missed an internal timing error.

Another check found that a bookmark directly after an SSML break was reported roughly 0.9 seconds early. A requested voice that was not installed could silently fall back to the default voice. The generator therefore checked installed voices and rejected a bookmark placed at the opening of a turn after a break. Zira read both interview roles because the second planned English voice was unavailable.

Those observations apply to the recorded machine and configuration. They are not a claim that every speech synthesizer has the same clock or fallback behavior. For our tests, they were enough to disqualify unverified bookmarks as timing truth.

A passing test needs a named claim

Our backend also used offline doubles so routine tests could run without model calls. The embedding double generated deterministic bag-of-words vectors. A response stub added a citation marker when its supplied context contained the corresponding source marker. Those choices made protocol checks repeatable.

The doubles had deliberately narrower behavior than the deployed services. The in-memory vector store did not check vector length. The hashing embedder used a separate similarity threshold of 0.25; that number did not calibrate the production embedding model. One fixture even needed different wording because another seeded fragment scored 0.436 against the intended fragment's 0.385.

Which result can support which claim?

Test material	Useful evidence	Boundary
Offline doubles	DTO fields, citation transport and deterministic state transitions.	They do not measure model answer quality.
Adapter tests	Request shape, response parsing and error mapping for selected fixtures.	They do not establish all live-provider behavior.
Synthetic media	Known wording and intentionally placed facts or transitions.	The generator and its reference annotations must be verified.
Real recordings	Observed terminology and timing problems in actual speech.	Without human reference text, they do not provide a measured real-audio WER.

For the double's missing dimension check, the appropriate response was to record the gap and cover the real adapter boundary. It would have been misleading to treat a green in-memory test as evidence that the vector service would accept the same input.

Build known failures into the material

Our resulting synthetic interview lasted about 32 minutes and 24 seconds. Its script supplied a reference for transcription and contained facts assigned to known moments. The live run could then ask different questions: did retrieval find the intended moment, and did the answer cite that moment rather than an adjacent passage?

Those questions produced different results. The recorded run found the expected segment among its retrieved sources for all ten questions. The answer cited the intended moment in nine cases. The remaining answer cited a neighboring segment. A single retrieval score would have hidden that difference.

For the later synthetic video, we used 30 slides in 31 appearances, including a return to an earlier slide. Some facts existed only on the slides. Brief forward-and-back transitions and a speaker-camera region gave the segmentation code specific failure cases to encounter. This was a controlled fixture for those behaviors, rather than a sample of all webinar formats.

For scanned pages, we built image-only PDFs from public FDA documents and retained their text layers for comparison. That reference was convenient, but it contained reading-order and typography artifacts.

A mismatch against the layer could therefore reflect a reference problem rather than an OCR error. Our table-reading case shows why we checked cells separately.

The scorer also needs a failing example

A synthetic chart exposed a mistake in the expected data. The scorer labeled three axis numbers as invented because those tick labels were missing from its expected-string list. They were printed on the image. Changing the model to satisfy that verdict would have rewarded the scorer's omission.

We also found a generated video question whose wording allowed an unintended answer. Rephrasing it changed the question set. It did not demonstrate an improvement in the retrieval model. The evaluation history needs to preserve that distinction, otherwise a repaired benchmark can look like a repaired product.

For a reusable fixture, we would preserve the generator version, media checksum, expected observations and scoring configuration together. The reference should explain normalization: whether written-out numbers match digits, whether order matters and what temporal tolerance is allowed. A human should be able to inspect a disputed example without reconstructing the generator from a headline metric.

Keep protocol confidence separate from answer confidence

Our strongest offline tests established specific contracts. They could detect a malformed vector response or a lost citation field without waiting for a remote model. The synthetic corpora supplied controlled content tests. Real recordings then revealed problems the clean script had not exposed, including confidently misheard terminology.

That sequence is useful only while its evidence stays separated. The low synthetic transcription error described in our ASR evaluation case did not provide a reference transcript for the real recording. Likewise, a DOM assertion did not establish that someone had listened to an audio clip or watched a video.

For the next run, the smallest useful ledger would name the claim, the fixture that could falsify it, the observed outcome and the unsupported behavior left outside the test. Its first entry would be the generator itself: a 31.3-second file must not acquire a 43.1-second end marker.

Development evidence: audio-generator checks on October 6, live audio checks on October 7, scan checks on October 9 and video-fixture work on October 12, 2026. Results here summarize those records; the complete generators, fixtures and raw outputs are not released with this article. The software names describe the tested configuration, not a current setup guide.

FAQ

Last updated: October 15, 2026

Quick answers to common questions about custom software development, pricing, process and technology.

Expand all Collapse all

Search FAQs

Type to filter questions and answers. Use Topic to narrow the list.

Showing all 14

• How long have you been in business?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has been in business since 2013, with over 13 years of experience in custom software development. During this time, we have delivered over 110 applications for 200+ clients across 28 industries, including FinTech, healthcare, crypto and e-commerce. We are rated 5/5 on Clutch based on 107 verified reviews (2026).

• What services do you provide?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production provides six core service categories: AI Development (LLM applications, AI agents, machine learning and computer vision), Blockchain and Web3 (smart contracts, DeFi, tokenization on Ethereum, Solana, TON and other chains), FinTech Development (payment platforms, digital banking, KYC and AML integration), Software Development (mobile apps, web platforms, database design, UI/UX), Security and QA (code audits, penetration testing, smart contract audits, manual and automated testing) and Consulting (architecture design, MVP validation, startup consulting).

• Where is Pharos Production located?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production is headquartered in Las Vegas, Nevada, USA (5348 Vegas Dr, Las Vegas, NV 89108), with an engineering office in Kyiv, Ukraine (44-B Eugene Konovalets Str. Suite 201, Kyiv 01133). We work with clients worldwide and provide remote collaboration across all time zones. Visit our contact page for directions and scheduling options.

• How big is your team?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has a team of 90+ engineers, including software developers, blockchain specialists, QA engineers, DevOps experts, UI/UX designers, project managers and solution architects. Our founder, Dr. Dmytro Nasyrov, holds a PhD in Artificial Intelligence and leads the technical direction of all projects.

• What types of customers do you serve?

  Copy link Copies a direct link to this answer to your clipboard.

  We serve a wide range of clients, from startups and product companies to mid-sized enterprises and large institutions. Our clients include crypto exchanges, FinTech providers (like Pleenk), healthcare organizations, sportsbook operators (like Pro Gambling), e-commerce platforms and SaaS companies. Pharos Production has worked with 200+ clients across 28 industries since 2013, adapting engagement models to match each client's stage, whether it is MVP validation for a startup or enterprise-scale development for an established business.

• How is a custom software development company different from an agency?

  Copy link Copies a direct link to this answer to your clipboard.

  A custom software development company like Pharos Production assigns one dedicated engineering team that owns your product across multiple release cycles, instead of handing it off once a contract ends. Agencies more often rotate staff between clients and optimize for fast delivery over long-term product ownership.

  Pharos Production keeps the same team on your codebase through 2-week sprints, so there is no re-onboarding cost each time you request a new feature.

• How much does custom software development cost?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom software development costs vary based on project scope and complexity. At Pharos Production, typical project ranges are: MVP development ($10,000-$25,000), suitable for startups validating a product idea; full-fledged production ($25,000-$50,000), for established businesses scaling a proven concept; and full-cycle development ($50,000-$80,000+), for complex enterprise-grade systems. These ranges include architecture design, development, QA testing and deployment. Hourly rates range from $50 to $99 depending on role and seniority. Final pricing depends on technology stack, number of integrations and engagement model (staff augmentation, dedicated team or project outsourcing).

• How long does it take to develop a custom software application?

  Copy link Copies a direct link to this answer to your clipboard.

  Development timelines depend on scope and complexity. At Pharos Production, a typical MVP takes 2-4 months, a production-ready application takes 4-8 months and a complex enterprise system can take 8-12+ months. We use an agile methodology with 2-week sprints, delivering working increments after each sprint. Every sprint includes a retrospective, progress report and planning session for the next iteration. This approach ensures transparency and allows businesses to launch faster by prioritizing high-impact features first. Get a timeline estimate for your project.

• What industries does Pharos Production serve?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production serves 28 industries: FinTech, MiCA Compliance, Banking, Insurance, Crypto, Healthcare, Fitness, E-Commerce, Food & Restaurant, On-Demand, Manufacturing, Automotive, Energy, Oil & Gas, Supply Chain, Construction, Real Estate, Gaming, Streaming, Media, Telecom, Education, Aviation, Travel, Logistics, Marketing, Agriculture and Recommender Systems. Our deepest expertise is in FinTech, blockchain and healthcare, where we have delivered compliance-ready platforms (HIPAA, PCI DSS, GDPR) and high-load systems handling thousands of concurrent users. For the latest industry insights, read our guides on FinTech trends in 2026 and the Web3 technology stack.

• Why hire a software development company instead of building in-house?

  Copy link Copies a direct link to this answer to your clipboard.

  Hiring a software development company offers faster time-to-market, lower upfront costs and access to specialized expertise without long-term employment commitments. According to Deloitte's 2024 Global Outsourcing Survey, 57% of companies outsource software development to access skills they cannot hire internally.

  Factor	In-house team	Software development company
  Time to assemble	3-6 months (recruiting + onboarding)	1-2 weeks
  Upfront cost	High (salaries, benefits, equipment)	Lower (project-based pricing)
  Specialized expertise	Limited to who you can hire locally	Access to 90+ engineers across blockchain, AI, FinTech
  Scalability	Slow (each new hire takes months)	Fast (scale up or down per sprint)
  Long-term commitment	Full-time employment contracts	Flexible engagement models
  Risk	High if key engineers leave	Company ensures continuity and knowledge transfer

  For businesses that need blockchain, AI or high-load architecture expertise, outsourcing to a specialized firm like Pharos Production reduces risk and accelerates delivery.

• What projects does Pharos Production not take on?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production focuses on mid-to-large custom software projects with budgets starting at $10,000. We do not take on template-based websites, WordPress theme customization or short-term contracts under one month. We also do not provide non-technical staffing (marketing, sales or design-only roles). Our strongest fit is blockchain, FinTech and healthcare projects where security, compliance and high-load architecture are critical. For smaller projects or MVPs under $10,000, we recommend exploring freelance platforms or no-code tools as a more cost-effective starting point.

• Why does Pharos Production use agile methodology?

  Copy link Copies a direct link to this answer to your clipboard.

  We use agile with 2-week sprints because it reduces the risk of building features that miss the mark. Each sprint ends with a working demo, a retrospective and a plan for the next iteration.

  This means clients see progress every 14 days and can adjust priorities based on real results, not assumptions. We chose this approach after years of experience showing that rigid, fixed-scope contracts lead to scope creep, missed deadlines and products that do not match market needs by launch day.

• When should you NOT hire a custom software development company?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom development is not the right choice in every situation. You should not hire a custom software company if: your problem is fully solved by an existing SaaS product (e.g. Shopify for e-commerce, Salesforce for CRM); your budget is under $10,000 and timeline is under 4 weeks; you need a simple landing page or marketing website (WordPress or Webflow is faster and cheaper); or you are still validating the idea and have not spoken to potential users yet.

  In these cases, off-the-shelf tools or no-code platforms offer better ROI. Custom development makes sense when you need unique workflows, regulatory compliance, high-load architecture or competitive differentiation that packaged software cannot provide.

• What kind of projects has Pharos Production delivered?

  Copy link Copies a direct link to this answer to your clipboard.

  Here are three anonymized examples from our recent delivery history:

  FinTech startup - payment platform (MVP)
  Scope: mobile app + backend API with bank-grade encryption. Team: 4 engineers, 1 QA. Timeline: 10 weeks. Budget: $38,000. Result: launched on schedule, processed $2M+ in transactions within the first quarter.

  Healthcare provider - patient portal (Full product)
  Scope: HIPAA-aligned web platform with EHR integration, appointment scheduling and telemedicine. Team: 6 engineers, 1 DevOps, 2 QA. Timeline: 6 months. Budget: $120,000. Result: 15,000+ active patients, zero compliance violations in two annual audits.

  Crypto exchange - trading engine (Complex)
  Scope: high-load matching engine handling 50,000+ orders per second, multi-chain wallet infrastructure on Ethereum and Solana. Team: 8 engineers, 2 QA, 1 security auditor. Timeline: 11 months. Budget: $340,000. Result: 99.97% uptime, passed three independent security audits.

  See more projects: NoMoreBets, Pulse, Sagas, Gambit Stream and Pleenk. For the full portfolio, visit our case studies. Learn more about the technology behind these projects in our guide to stablecoins and crypto infrastructure.

Dmytro Ihnatov is the Lead Backend Developer at Pharos Production. He documents backend architecture and research experiments from the team's work on a professional learning platform for aesthetic medicine.