Our Retrieval Cutoff Worked Until the Corpus Changed

Technical reviewer: Victor Sineglazov, AI Scientific Advisor.

The intended slide ranked first with a similarity score of 0.492. Our cutoff was 0.5, so the answer pipeline discarded it. Improving the rank would not have recovered that source: the acceptance rule had already excluded the best candidate.

This October result followed an earlier calibration on English FDA documents. We had selected a stricter cutoff to reject an unrelated instruction. The same number later refused a relevant question in another language and removed a correct slide representation. The history of that cutoff explains why we stopped treating it as a portable property of the retrieval system.

The first cutoff separated the examples we had actually tried

The September 17 retrieval design specified an evaluation set of twenty-one questions against twelve FDA documents. Its document recall target was at least 0.8 within the top five retrieved results. That was a development acceptance criterion for the chosen set.

During implementation, the lowest expected-match score recorded was 0.599. Ordinary unrelated questions reached a maximum of 0.372, leaving room for a cutoff of 0.4. A final-review probe then changed the picture: an instruction asking for a fixed one-word response scored 0.422 and left seven fragments above that cutoff.

The selected threshold moved to 0.5. The YAML configuration was not its only definition. A default in the configuration record and a binding test still used 0.4 and had to be synchronized.

The rows come from separate dated checks. They do not constitute a common score distribution, and the table should not be used to infer one optimal replacement threshold.

An extra refusal can conceal a retrieval defect

In the September 22 book prototype, the same substantive question scored 0.487 in Russian and 0.55 in English. Relevant material existed in the corpus. The Russian version was nevertheless refused by the threshold calibrated on English FDA material.

A report counting only how often the system refused to answer could describe that result as stronger restraint. The reference answer and expected source showed that this particular refusal was a miss.

The development record proposed further calibration and considered query translation. Those proposals were not a measured multilingual improvement. A test would have to retain both answerable and unanswerable questions in each relevant language to establish their effect.

A new source representation changed the margin again

The October lecture checks introduced slide descriptions and transcript-derived material. For one synthetic slide question, the intended slide scored 0.492. The isolated retrieval check found nine of ten expected slides against a target of ten.

A diagnostic representation experiment combined extracted text with a meaning-oriented description. The reported score rose to 0.514, but that representation omitted the layout description, removing a chart's or photograph's visual description from search and the source supplied to the answer model. The recorded ruling also required remeasurement on the webinar. Improving one query's score did not settle the representation choice.

The later staging check added another difference: book content competed with the lecture. One answer cited the book despite finding both book and lecture material. A missing expected citation could therefore come from candidate filtering, ranking or the model's choice among supplied sources.

Our slide-detection case explains why accurate boundaries did not resolve this problem. Finding the right image interval and accepting its searchable representation are separate operations.

A deterministic test embedder has its own score scale

The test suite used a hashing embedder and in-memory vector store for deterministic checks. Its seeded examples used a cutoff of 0.25. That number belonged to the test double's representation and fixtures.

Passing those tests could establish that filtering, ordering and refusal branches behaved as specified for the seeded inputs. It could not calibrate the production embedding model. Sharing a cosine calculation did not make the two score scales interchangeable.

We described this boundary in our synthetic-corpus evaluation case. A useful double makes a behavior reproducible; it does not supply evidence about a different model's margins.

Keep the rejected candidate in the evaluation record

For the next threshold evaluation, we would preserve candidate scores before filtering, the expected source and the representation version. A record containing only the final accepted context would hide the 0.492 slide that explained the refusal.

We would distinguish whether an expected document was retrieved, whether the appropriate page or slide survived filtering, whether the answer cited it and whether that source supported the claim. A successful result at one stage would not populate the next stage's result automatically.

The source-marker field in this prototype established that a marker resolved to supplied context. It did not independently validate a clinical claim. Our citation case retains that distinction even for answers whose retrieval had succeeded.

We do not offer 0.5, 0.4 or the diagnostic 0.514 result as a setting to copy. The concrete regression is the relevant candidate just below the deployed cutoff. It belongs in the next evaluation alongside the unrelated instruction that motivated the earlier change.

Evidence: September 17 retrieval calibration and review, September 22 book-language probe, October lecture experiments and October 14 staging observations in our development records. These are bounded samples across different corpora. The diagnostic representation experiment was not a recorded production fix, and no new threshold benchmark was run for this article.

FAQ

Last updated: October 17, 2026

Quick answers to common questions about custom software development, pricing, process and technology.

Expand all Collapse all

Search FAQs

Type to filter questions and answers. Use Topic to narrow the list.

Showing all 14

• How long have you been in business?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has been in business since 2013, with over 13 years of experience in custom software development. During this time, we have delivered over 110 applications for 200+ clients across 28 industries, including FinTech, healthcare, crypto and e-commerce. We are rated 5/5 on Clutch based on 107 verified reviews (2026).

• What services do you provide?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production provides six core service categories: AI Development (LLM applications, AI agents, machine learning and computer vision), Blockchain and Web3 (smart contracts, DeFi, tokenization on Ethereum, Solana, TON and other chains), FinTech Development (payment platforms, digital banking, KYC and AML integration), Software Development (mobile apps, web platforms, database design, UI/UX), Security and QA (code audits, penetration testing, smart contract audits, manual and automated testing) and Consulting (architecture design, MVP validation, startup consulting).

• Where is Pharos Production located?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production is headquartered in Las Vegas, Nevada, USA (5348 Vegas Dr, Las Vegas, NV 89108), with an engineering office in Kyiv, Ukraine (44-B Eugene Konovalets Str. Suite 201, Kyiv 01133). We work with clients worldwide and provide remote collaboration across all time zones. Visit our contact page for directions and scheduling options.

• How big is your team?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production has a team of 90+ engineers, including software developers, blockchain specialists, QA engineers, DevOps experts, UI/UX designers, project managers and solution architects. Our founder, Dr. Dmytro Nasyrov, holds a PhD in Artificial Intelligence and leads the technical direction of all projects.

• What types of customers do you serve?

  Copy link Copies a direct link to this answer to your clipboard.

  We serve a wide range of clients, from startups and product companies to mid-sized enterprises and large institutions. Our clients include crypto exchanges, FinTech providers (like Pleenk), healthcare organizations, sportsbook operators (like Pro Gambling), e-commerce platforms and SaaS companies. Pharos Production has worked with 200+ clients across 28 industries since 2013, adapting engagement models to match each client's stage, whether it is MVP validation for a startup or enterprise-scale development for an established business.

• How is a custom software development company different from an agency?

  Copy link Copies a direct link to this answer to your clipboard.

  A custom software development company like Pharos Production assigns one dedicated engineering team that owns your product across multiple release cycles, instead of handing it off once a contract ends. Agencies more often rotate staff between clients and optimize for fast delivery over long-term product ownership.

  Pharos Production keeps the same team on your codebase through 2-week sprints, so there is no re-onboarding cost each time you request a new feature.

• How much does custom software development cost?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom software development costs vary based on project scope and complexity. At Pharos Production, typical project ranges are: MVP development ($10,000-$25,000), suitable for startups validating a product idea; full-fledged production ($25,000-$50,000), for established businesses scaling a proven concept; and full-cycle development ($50,000-$80,000+), for complex enterprise-grade systems. These ranges include architecture design, development, QA testing and deployment. Hourly rates range from $50 to $99 depending on role and seniority. Final pricing depends on technology stack, number of integrations and engagement model (staff augmentation, dedicated team or project outsourcing).

• How long does it take to develop a custom software application?

  Copy link Copies a direct link to this answer to your clipboard.

  Development timelines depend on scope and complexity. At Pharos Production, a typical MVP takes 2-4 months, a production-ready application takes 4-8 months and a complex enterprise system can take 8-12+ months. We use an agile methodology with 2-week sprints, delivering working increments after each sprint. Every sprint includes a retrospective, progress report and planning session for the next iteration. This approach ensures transparency and allows businesses to launch faster by prioritizing high-impact features first. Get a timeline estimate for your project.

• What industries does Pharos Production serve?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production serves 28 industries: FinTech, MiCA Compliance, Banking, Insurance, Crypto, Healthcare, Fitness, E-Commerce, Food & Restaurant, On-Demand, Manufacturing, Automotive, Energy, Oil & Gas, Supply Chain, Construction, Real Estate, Gaming, Streaming, Media, Telecom, Education, Aviation, Travel, Logistics, Marketing, Agriculture and Recommender Systems. Our deepest expertise is in FinTech, blockchain and healthcare, where we have delivered compliance-ready platforms (HIPAA, PCI DSS, GDPR) and high-load systems handling thousands of concurrent users. For the latest industry insights, read our guides on FinTech trends in 2026 and the Web3 technology stack.

• Why hire a software development company instead of building in-house?

  Copy link Copies a direct link to this answer to your clipboard.

  Hiring a software development company offers faster time-to-market, lower upfront costs and access to specialized expertise without long-term employment commitments. According to Deloitte's 2024 Global Outsourcing Survey, 57% of companies outsource software development to access skills they cannot hire internally.

  Factor	In-house team	Software development company
  Time to assemble	3-6 months (recruiting + onboarding)	1-2 weeks
  Upfront cost	High (salaries, benefits, equipment)	Lower (project-based pricing)
  Specialized expertise	Limited to who you can hire locally	Access to 90+ engineers across blockchain, AI, FinTech
  Scalability	Slow (each new hire takes months)	Fast (scale up or down per sprint)
  Long-term commitment	Full-time employment contracts	Flexible engagement models
  Risk	High if key engineers leave	Company ensures continuity and knowledge transfer

  For businesses that need blockchain, AI or high-load architecture expertise, outsourcing to a specialized firm like Pharos Production reduces risk and accelerates delivery.

• What projects does Pharos Production not take on?

  Copy link Copies a direct link to this answer to your clipboard.

  Pharos Production focuses on mid-to-large custom software projects with budgets starting at $10,000. We do not take on template-based websites, WordPress theme customization or short-term contracts under one month. We also do not provide non-technical staffing (marketing, sales or design-only roles). Our strongest fit is blockchain, FinTech and healthcare projects where security, compliance and high-load architecture are critical. For smaller projects or MVPs under $10,000, we recommend exploring freelance platforms or no-code tools as a more cost-effective starting point.

• Why does Pharos Production use agile methodology?

  Copy link Copies a direct link to this answer to your clipboard.

  We use agile with 2-week sprints because it reduces the risk of building features that miss the mark. Each sprint ends with a working demo, a retrospective and a plan for the next iteration.

  This means clients see progress every 14 days and can adjust priorities based on real results, not assumptions. We chose this approach after years of experience showing that rigid, fixed-scope contracts lead to scope creep, missed deadlines and products that do not match market needs by launch day.

• When should you NOT hire a custom software development company?

  Copy link Copies a direct link to this answer to your clipboard.

  Custom development is not the right choice in every situation. You should not hire a custom software company if: your problem is fully solved by an existing SaaS product (e.g. Shopify for e-commerce, Salesforce for CRM); your budget is under $10,000 and timeline is under 4 weeks; you need a simple landing page or marketing website (WordPress or Webflow is faster and cheaper); or you are still validating the idea and have not spoken to potential users yet.

  In these cases, off-the-shelf tools or no-code platforms offer better ROI. Custom development makes sense when you need unique workflows, regulatory compliance, high-load architecture or competitive differentiation that packaged software cannot provide.

• What kind of projects has Pharos Production delivered?

  Copy link Copies a direct link to this answer to your clipboard.

  Here are three anonymized examples from our recent delivery history:

  FinTech startup - payment platform (MVP)
  Scope: mobile app + backend API with bank-grade encryption. Team: 4 engineers, 1 QA. Timeline: 10 weeks. Budget: $38,000. Result: launched on schedule, processed $2M+ in transactions within the first quarter.

  Healthcare provider - patient portal (Full product)
  Scope: HIPAA-aligned web platform with EHR integration, appointment scheduling and telemedicine. Team: 6 engineers, 1 DevOps, 2 QA. Timeline: 6 months. Budget: $120,000. Result: 15,000+ active patients, zero compliance violations in two annual audits.

  Crypto exchange - trading engine (Complex)
  Scope: high-load matching engine handling 50,000+ orders per second, multi-chain wallet infrastructure on Ethereum and Solana. Team: 8 engineers, 2 QA, 1 security auditor. Timeline: 11 months. Budget: $340,000. Result: 99.97% uptime, passed three independent security audits.

  See more projects: NoMoreBets, Pulse, Sagas, Gambit Stream and Pleenk. For the full portfolio, visit our case studies. Learn more about the technology behind these projects in our guide to stablecoins and crypto infrastructure.

Dmytro Ihnatov is the Lead Backend Developer at Pharos Production. He documents backend architecture and research experiments from the team's work on a professional learning platform for aesthetic medicine.