How to evaluate AI submittal review tools
AI submittal review tools are new, and published benchmarks are one of the few public reference points. They help, but they test a narrow slice of the job, and most vendor scores are self-reported. The most reliable test is a side-by-side trial on a submittal your team has already reviewed, against your own plans and specs, judged on both the findings and the workflow around them.
Last reviewed October 29, 2026.
What published benchmarks miss
Benchmarks are useful for comparing general capability under fixed conditions. The public AEC-Bench paper (arXiv 2603.29199) is one reference. A few limits show up in that kind of test:
- Task sets are often small, and often product data only, with no shop drawings. The overall review action is not always scored.
- Answer keys are written by the benchmark authors, and reasonable reviewers can disagree.
- Automated scoring often matches keywords or exact clause numbers. That can miss a correct answer worded differently, and it may not penalize extra flags.
- Many scores are self-reported, run once, and published without the underlying outputs.
- Benchmarks usually do not score markup, citations, stamps, or integrations.
Run your own trial
- Pick two to four past submittals your team has already reviewed. Include one approved, one returned for revise and resubmit, and one rejected. Mix product data with at least one shop drawing.
- Use your own plans and specs, including Division 01 and any addenda in effect.
- Give every tool the same files and the same instructions.
- Have the reviewer who handled the original submittal compare each output with the actual review record.
- Time the whole process, from upload to a returnable submittal, not only the model run.
Judge the findings and the workflow
- Real issues caught or missed, compared with your review record.
- False positives, and how long each one takes to dismiss.
- Whether each finding cites a spec paragraph, sheet, or detail a reviewer can open quickly.
- Whether markup lands on the submittal sheets.
- Whether the response uses your cover sheet and stamps.
- Whether the tool fits your Procore or Autodesk Forma Build (formerly Autodesk Build) process. Your licensed reviewer still makes the final call.
How Specset approaches this
Specset places markup on the submittal sheets, cites the source of every finding, and imports your cover sheet and stamps. We would rather you test it on a submittal you already know, against your own plans and specs, next to any other tool you are considering.
Questions to ask before you rely on a score
01Are vendor benchmark scores comparable?
Only when every tool runs the same tasks the same way and the outputs are published. Most vendor scores today are self-reported, run once, and published without outputs, so treat them as a starting point, not a verdict.
02What submittal should we test with?
One your team has already reviewed, so you know the right answer. Include an approved one, one returned for revise and resubmit, and one rejected, plus at least one shop drawing if you can.
03Should we test more than one tool?
Yes. Give every tool the same files and the same instructions, and have the same reviewer judge them side by side.
04How should we count false positives?
Count every flag your reviewer dismisses and time how long each one takes. A tool that flags everything can look thorough and still cost you time.
05Is accuracy all that matters?
No. Check whether each finding cites a spec paragraph or sheet you can open quickly, whether markup lands on the sheets, and whether the response uses your own cover sheet and stamps. Also check that the tool fits your Procore or Autodesk Forma Build (formerly Autodesk Build) process. Your licensed reviewer still makes the final call.