Benchmark Guide
7 executable engineering studies now include 3 comparisons with real measurements from DOI-linked primary papers. Every study retains its model declaration, discretization history, fixed acceptance rule, provenance, limitations, and unedited outcome.
NOT_ESTABLISHED.How to read this guide
Reference
Solution targets come from public standard reproductions or a DOI-linked benchmark paper. Experimental targets come from two CC BY primary papers and one versioned CC BY dataset; source values and hashes are recorded under benchmarks/data/.
Numerics
Every run records formulation, mesh, computed response, target, error, discrete equation residual, and environment in benchmarks/results.json.
Decision
Only the named selected run determines PASS/FAIL. Coarser runs expose discretization behavior; they are not averaged into a favorable result. No benchmark earns physical-validation credit.
NAFEMS T3 — one-dimensional transient heat transfer
PASSExternal target reproduction and spatial-convergence study. Reference: The Standard NAFEMS Benchmarks, TNSB Rev. 3 (October 1990), T3.
Model
0.1 m bar; 0.01 m by 0.01 m cross-section
k=35 W/(m degC), rho=7200 kg/m^3, cp=440.5 J/(kg degC)
T(0,t)=0; T(0.1,t)=100 sin(pi t/40) degC; lateral adiabatic
Acceptance
20 field_line2 elements, dt=0.05 s
absolute relative error in the published target quantity; threshold 2%. The selected discrete equation residual must also be at most 1.0e-08.
Project qualification threshold fixed at 2% before execution; tighter than the 3.0% error reported for Abaqus's fine linear-element result on the public source page.
Reference target
T(0.08 m, 32 s) = 36.60 degC
The original NAFEMS target is cited through the publicly accessible Abaqus reproduction; the proprietary NAFEMS source was not consulted.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=field_line2, elements=5, nodes=6, characteristic size m=0.02 | 40.99657 | 36.6 | 12% | 1.036677e-13 |
| element=field_line2, elements=10, nodes=11, characteristic size m=0.01 | 37.44812 | 36.6 | 2.32% | 9.848766e-14 |
| element=field_line2, elements=20, nodes=21, characteristic size m=0.005 | 36.79691 | 36.6 | 0.538% | 9.442162e-14 |
Known deviations and limits
- This qualifies one response quantity for this benchmark domain; it does not establish accuracy for arbitrary transient thermal models.
- The time step is held fixed during the spatial study; separate time-step verification remains necessary for production models.
NAFEMS FV16 — cantilevered thin square plate
PASSExternal six-mode frequency comparison and mesh study. Reference: The Standard NAFEMS Benchmarks, TNSB Rev. 3 (October 1990), FV16.
Model
10 m by 10 m square plate, thickness 0.05 m
E=200 GPa, nu=0.3, rho=8000 kg/m^3
Along x=0: transverse displacement uz=0 and rotation about y, ry=0; membrane DOFs are absent from the plate formulation.
Acceptance
12 by 12 MITC4 mesh
maximum absolute relative error across modes 1 through 6; threshold 5%. The selected discrete equation residual must also be at most 1.0e-08.
A 5% envelope was fixed before execution for all six modes; individual published Abaqus variants exceed this on higher modes, so failures are retained rather than mode-selected away.
Reference target
f1..f6 = [0.421, 1.029, 2.582, 3.306, 3.753, 6.555] Hz
The original NAFEMS targets are cited through the public Abaqus reproduction.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=mitc4, elements=16, nodes=25, divisions per side=4 | 0.421756, 1.04514, 2.93961, 3.59581, 4.21423, 7.51004 | 0.421, 1.029, 2.582, 3.306, 3.753, 6.555 | 14.6% | 6.375112e-13 |
| element=mitc4, elements=64, nodes=81, divisions per side=8 | 0.419062, 1.02976, 2.65639, 3.35688, 3.84666, 6.77714 | 0.421, 1.029, 2.582, 3.306, 3.753, 6.555 | 3.39% | 2.135346e-13 |
| element=mitc4, elements=144, nodes=169, divisions per side=12 | 0.418462, 1.02668, 2.6041, 3.31121, 3.77971, 6.63564 | 0.421, 1.029, 2.582, 3.306, 3.753, 6.555 | 1.23% | 1.641217e-13 |
Known deviations and limits
- physicsbase uses a 4-node MITC plate; the public Abaqus table reports S8R/S8R5/STRI65 shell formulations.
- Only transverse plate kinematics are represented, so the published in-plane restraints have no corresponding DOFs.
- Mode-by-mode comparison assumes no mode crossing; the reported sequence was also inspected by frequency order only, not by a modal assurance criterion.
- Passing frequencies do not independently qualify stresses or forced response.
NAFEMS FV52 — simply supported solid square plate
PASSExternal seven-mode solid-element comparison. Reference: The Standard NAFEMS Benchmarks, TNSB Rev. 3 (October 1990), FV52.
Model
10 m by 10 m square solid plate, thickness 1.0 m
E=200 GPa, nu=0.3, rho=8000 kg/m^3
uz=0 on all four bottom-face edges (z=-0.5 m), matching the published problem statement
Acceptance
10 by 10 by 1 hex8i mesh (finest of the 6/8/10 convergence series)
maximum absolute relative error across the first seven elastic modes; threshold 10%. The selected discrete equation residual must also be at most 1.0e-08.
A 10% all-mode envelope was fixed before execution. The formulation now matches the published incompatible-mode elements; the finest mesh in the fixed convergence series meets the envelope. Neither the targets nor the threshold were changed.
Reference target
elastic modes = [44.092, 106.66, 106.66, 156.23, 193.58, 200.13, 200.13] Hz after three RBMs
The original NAFEMS targets are cited through the public Abaqus reproduction.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=hex8i, elements=36, nodes=98, in plane divisions=6, through thickness divisions=1 | 46.6384, 121.701, 121.701, 188.665, 199.221, 212.108, 212.108 | 44.092, 106.66, 106.66, 156.23, 193.58, 200.13, 200.13 | 20.8% | 9.203206e-16 |
| element=hex8i, elements=64, nodes=162, in plane divisions=8, through thickness divisions=1 | 45.6326, 115.038, 115.038, 176.386, 196.773, 209.566, 209.566 | 44.092, 106.66, 106.66, 156.23, 193.58, 200.13, 200.13 | 12.9% | 1.416452e-15 |
| element=hex8i, elements=100, nodes=242, in plane divisions=10, through thickness divisions=1 | 45.1277, 112.059, 112.059, 170.997, 195.635, 208.341, 208.341 | 44.092, 106.66, 106.66, 156.23, 193.58, 200.13, 200.13 | 9.45% | 2.177294e-15 |
hex8i brick — the same formulation as the reference Abaqus C3D8I result, independently verified by a distorted-mesh constant-strain patch test (~1e-13) and a slender cantilever recovering Euler-beam tip deflection to 0.3% — brings the worst mode within the unchanged 10% envelope at the finest mesh of the fixed 6/8/10 convergence series (20.8% → 12.9% → 9.5%). No target and no tolerance were changed.Known deviations and limits
- physicsbase uses hex8i incompatible-mode bricks, matching the formulation of the reference Abaqus C3D8I table result (element verified independently by a distorted-mesh constant-strain patch test to ~1e-13 and a slender cantilever recovering Euler-beam tip deflection to 0.3%).
- One element is used through thickness; convergence is shown in-plane over 6/8/10 divisions.
- Frequency-order matching does not resolve repeated-mode basis rotations; a modal assurance criterion is not yet implemented.
- A formulation-specific pass for hex8 cannot be generalized to tet4, tet10, or hex20.
Scordelis–Lo cylindrical roof
PASSPublished shell benchmark and mesh-convergence study. Reference: R. H. MacNeal and R. L. Harder, A Proposed Standard Set of Problems to Test Finite Element Accuracy, Finite Elements in Analysis and Design 1 (1985) 3–20, Scordelis–Lo roof.
Model
quarter model; R=25, L=50, t=0.25, 40 degree half-angle
E=4.32e8, nu=0
symmetry and rigid-diaphragm constraints used by the established quarter-model benchmark
Acceptance
12 by 12 quarter mesh
absolute relative error in free-edge displacement; threshold 2%. The selected discrete equation residual must also be at most 1.0e-08.
The pre-existing project qualification threshold is 2%; this study exposes the full 4/8/12 mesh history rather than only the accepted endpoint.
Reference target
free-edge vertical displacement = 0.3024
Direct DOI-linked publication.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=shell4, elements=16, nodes=25, quarter mesh=4 by 4 | 0.2981037 | 0.3024 | 1.42% | 1.293113e-12 |
| element=shell4, elements=64, nodes=81, quarter mesh=8 by 8 | 0.299183 | 0.3024 | 1.06% | 6.034572e-12 |
| element=shell4, elements=144, nodes=169, quarter mesh=12 by 12 | 0.3013398 | 0.3024 | 0.351% | 1.657247e-11 |
Known deviations and limits
- physicsbase shell4 is its own implementation; no Abaqus result is used as the target.
- This benchmark is sensitive to load discretization and boundary-condition conventions; those choices are recorded above.
- One displacement target cannot qualify shell stress recovery or arbitrary distorted meshes.
Loh et al. cantilevers - static-to-dynamic cross-response
PASSPredict measured modal frequency using modulus inferred from the paper's separate static-deflection response. Reference: Loh et al., An Integrated Approach for the Determination of Young's Modulus of a Cantilever Beam Using Finite Element Analysis and the Digital Image Correlation (DIC) Technique, Electronics 11 (2022) 2826, Table 4, steel/aluminum/brass.
Model
three 600 mm effective-length rectangular cantilevers, 19 mm wide and 3.3 mm thick
Table 4 static moduli Es=[175.5, 66.7, 82.0] GPa and densities=[8195, 2700, 8170] kg/m^3
ideal fixed clamp at x=0; free at x=0.600 m
Acceptance
16 beam2d elements per specimen
maximum absolute relative error across all three measured first frequencies; threshold 8%. The selected discrete equation residual must also be at most 1.0e-08.
An 8% all-material engineering-screening envelope was fixed before executing the engine. It was not derived from the computed errors and is not a formal uncertainty bound.
Reference target
measured f1 = [7.0, 7.2, 4.8] Hz
Geometry, densities, static-test moduli, and measured frequencies are transcribed from the DOI-linked paper; the paper is CC BY 4.0.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=beam2d, elements per specimen=4, nodes per specimen=5 | 6.85283, 7.36015, 4.69139 | 7, 7.2, 4.8 | 2.26% | 2.444871e-17 |
| element=beam2d, elements per specimen=8, nodes per specimen=9 | 6.85262, 7.35993, 4.69125 | 7, 7.2, 4.8 | 2.27% | 1.09006e-17 |
| element=beam2d, elements per specimen=16, nodes per specimen=17 | 6.85261, 7.35991, 4.69124 | 7, 7.2, 4.8 | 2.27% | 1.126368e-16 |
Known deviations and limits
- The engine uses an Euler-Bernoulli beam; the paper's frequency extraction used DIC on the physical beams.
- Clamp compliance, dimensional tolerances, and density uncertainty are not reported and are therefore not fitted.
- The static modulus and modal frequency come from the same specimens, so this is cross-response evidence rather than an independent material blind prediction.
- The paper reports three DIC repetitions but not the individual observations or a measurement-uncertainty budget.
- The acceptance rule was recorded in this implementation session, not in an independently timestamped preregistration; this study earns no physical-validation credit.
Turkay single-profile wood cantilever modes
PASSCompare an uncalibrated literature-property beam model with five measured modes. Reference: Turkay, Determination of Dynamic Characteristics of Composite Cantilever Beams Using Experimental and Analytical Methods, Buildings 15 (2025) 1608, single-profile wood specimen W, Tables 3 and 5.
Model
1.00 m long rectangular wood profile, 30 mm wide by 10 mm high
longitudinal E1=10,100 MPa and density=600 kg/m^3 from the paper's initial analytical model
ideal fixed clamp at one end; free at the other
Acceptance
20 beam2d elements
maximum absolute relative error across measured modes 1 through 5; threshold 10%. The selected discrete equation residual must also be at most 1.0e-08.
A 10% five-mode engineering-screening envelope was fixed before engine execution to accommodate the explicitly unmodeled connection and orthotropic/shear detail. It is not a measurement-uncertainty interval.
Reference target
measured modes 1-5 = [6.410, 41.090, 114.700, 221.100, 353.600] Hz
Dimensions, literature properties, test method, and modal frequencies are transcribed from the CC BY 4.0 primary paper.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=beam2d, elements=5, nodes=6 | 6.62783, 41.5561, 116.718, 230.574, 382.696 | 6.41, 41.09, 114.7, 221.1, 353.6 | 8.23% | 2.614341e-17 |
| element=beam2d, elements=10, nodes=11 | 6.62774, 41.5367, 116.33, 228.119, 377.688 | 6.41, 41.09, 114.7, 221.1, 353.6 | 6.81% | 3.075201e-17 |
| element=beam2d, elements=20, nodes=21 | 6.62774, 41.5354, 116.302, 227.916, 376.802 | 6.41, 41.09, 114.7, 221.1, 353.6 | 6.56% | 1.088192e-16 |
Known deviations and limits
- The engine uses a slender Euler-Bernoulli beam and the reported longitudinal modulus; the paper's SAP2000 model declared a fuller orthotropic material.
- The experimental L-bracket/screw attachment is idealized as a perfect clamp because its compliance was not identified.
- The paper tested one specimen of each beam type; this benchmark uses only specimen W and cannot estimate specimen-to-specimen variability.
- No numerical measurement uncertainty or identified support stiffness is available in the source.
- The acceptance rule was not independently preregistered, so the comparison is an experimental benchmark and not physical validation.
Tomanek and Stutts Al6061 transient cooling - held-out repeats
PASSApply run-1 fitted properties unchanged to two held-out physical repeats and predict 3,010 temperature observations. Reference: Tomanek and Stutts, Data on the Validation to Determine the Material Thermal Properties Estimation Via a One-Dimensional Transient Convection Model, Data in Brief 40 (2022) 107632, Al6061-T6, nominal 100 C tests 100al1/2/3.
Model
150 mm long, 3.175 mm diameter Al6061-T6 cylindrical rod
rho=2767.99 kg/m^3, cp=896 J/(kg K), and run-1 fitted k=148.7127419669693 W/(m K)
measured TC1 history prescribed at x=0; distributed lateral Robin convection h=139.008276375768 W/(m^2 K) to zero excess temperature; adiabatic far end
Acceptance
60 field_line2 elements, dt=1 s, 301 samples per repeat
maximum relative L2 history error across TC2-TC6 in each of two held-out repeats; threshold 10%. The selected discrete equation residual must also be at most 1.0e-08.
A 10% per-sensor held-out history envelope was fixed before engine execution. It is a screening criterion, not a combined experimental/model uncertainty interval.
Reference target
TC2-TC6 excess-temperature histories, t=0..300 s, in 100al2 and 100al3
The CC BY 4.0 Mendeley dataset supplies the raw CSV files and the official workbook supplies the run-1 fitted h and k. The curated held-out CSV is SHA-256 checked at runtime.
| Discretization | Computed | Target | Max/reference error | Equation residual |
|---|---|---|---|---|
| element=field_line2, elements=15, nodes=16, dt s=1.0 | 57.672, 40.8424, 28.9324, 20.4968, 14.5249, 58.8936, 41.6963, 29.5258, 20.9151, 14.82 | 59.39, 42.08, 30.35, 22.07, 14.2, 60.63, 42.95, 31.02, 22.56, 14.59 | 7.64% | 9.012978e-12 |
| element=field_line2, elements=30, nodes=31, dt s=1.0 | 57.7528, 40.9502, 29.044, 20.6028, 14.6189, 58.9648, 41.803, 29.6413, 21.0235, 14.916 | 59.39, 42.08, 30.35, 22.07, 14.2, 60.63, 42.95, 31.02, 22.56, 14.59 | 7.14% | 2.474939e-11 |
| element=field_line2, elements=60, nodes=61, dt s=1.0 | 57.7729, 40.9766, 29.0718, 20.629, 14.6422, 58.982, 41.8295, 29.6698, 21.0504, 14.9398 | 59.39, 42.08, 30.35, 22.07, 14.2, 60.63, 42.95, 31.02, 22.56, 14.59 | 7.02% | 4.851276e-11 |
Known deviations and limits
- The engine uses linear finite elements and Crank-Nicolson rather than the source's 100-term Fourier-series implementation.
- The measured TC1 history is a time-varying boundary condition, so TC1 itself is not scored.
- Run 1 was fitted using the source authors' workflow and runs 2-3 are held out here, but all runs share the same apparatus and test series.
- The source does not publish a complete thermocouple, wind-speed, dimensional, or model-form uncertainty budget.
- The initial field uses the six measured t=0 thermocouple values and the boundary uses measured TC1; only TC2-TC6 are predictions.
- The split and acceptance rule were not independently preregistered before access to the dataset, so this earns no physical-validation credit.
Reproduce
python -m benchmarks.suite
The command intentionally exits nonzero while any selected qualification study fails. For report generation after a known failure, the JSON result is still written first.