appendix j - ENHANCEMENT Evaluations
CDTC developed a basis for evaluating candidates for funding
under the Transportation Enhancements Program (TEP). The basis for Round Two evaluation of
proposals from within the Capital District reflects several changes to the
Round One methodology.
In evaluating proposals to Round One of the TEP, a team of
CDTC and New York State Department of Transportation (NYSDOT) Region One staff
used a methodology approved by the CDTC Planning Committee on
¨
Benefit to Enhancement Region and Environment
¨
Enhancement of Transportation Plans, Projects
¨
Relationship To/Support for Other Plans,
Projects
¨
Size of Matching Share, Assurance of
Availability
¨
Direct User Benefits to Immediate Proposal Area
and Environment
¨
Innovation/Creativity/Mix of Activities
¨
Supportiveness of Master Planning in Recognized
Areas of Special Significance
¨
Level of Community, Regional Support
While the Guidebook-based
methodology was helpful in providing a framework for objective assessment of
the merits of each proposal, discussions after the completion of Round One
raised several concerns with this basis, particularly the following:
¨
the criterion set seemed to place too much
weight on regional rather than local benefits
¨
on a related note, trails or other sorts of
"tourism-oriented" proposals tended to be the only ones which could
score well on most or all of the criteria
¨
safety benefits seemed not to get adequate
weight
¨
partly related to the safety point, many
important projects in urban areas did not see their true benefits reflected in
point scores
¨
there was no explicit opportunity to consider
cost-effectiveness in rankings
As a result, while there were 12 types of projects
eligible for TEP funding, most proposals faring well in the rankings were for
trail developments or extensions. In
addition, higher-cost proposals tended to fare better than lower-cost
proposals.
For Round One of the TEP, the
CDTC/NYSDOT evaluation team needed to base its evaluations on the Guidebook, for applicants developed
their proposals based on the Guidebook's
listing of criteria. That is, as the Guidebook contained the first
indications of what the basis for proposal evaluation would be, the team did
not have the opportunity to introduce additional criteria, for it would
arguably have been unfair to applicants to have their proposals evaluated based
on a different set of criteria from those presented in the Guidebook.
For Round Two, a CDTC document discussing the evaluation
methodology to be applied to Capital District proposals was provided to
potential applicants at the informational workshop held at the start of the Round
Two TEP effort and thereafter, with the CDTC document positioned as a companion
document to the Guidebook. The CDTC document noted that while the
CDTC evaluations would be more rigorous, considering criteria beyond those set
forth in the Guidebook, applicants
would not be compelled to do any more
work in preparing proposals than would be expected based strictly on the Guidebook.
CDTC and NYSDOT Region One staff met to discuss their
concerns with the Round One approach and to identify possible changes to the
evaluation methodology which would ensure that evaluations of Round Two
proposals would be based on broader opportunities for success. The group identified a series of
modifications to the Round One evaluation process dealing with criteria and process. The CDTC Planning Committee discussed and
concurred with these possible modifications at its
1. Presentation
of evaluation findings and preliminary rankings to the Planning Committee by
Enhancements project category, to facilitate Committee consideration of
prioritized candidate lists reflecting a wider range of project types.
2. Addition
of a new sub-criterion within the "Benefits" criterion group (see
Modification 4 for a discussion of criterion groups) dealing with safety
impacts. (In addition, as will
be detailed later on in this document, the descriptions of the existing
criteria in this group have been modified for purposes of clarity and
consistency.)
3. Reallocation
of maximum point scores to reduce bias toward very large projects.
4. The
aggregation of criteria into three criterion groups within which
evaluators would have a defined degree of flexibility in allocating points.
5. Provision
to the Planning Committee of indications of how the proposals fare under supplemental
screening criteria employing "A" through "C" grades
for feasibility
and cost-effectiveness.
6. Application
of model-based
or otherwise quantitative assessments of potential proposal impact wherever
possible (e.g., using the bicycle and/or pedestrian versions of the CDTC
regional travel model). These
assessments would be inputs to evaluator consideration of how proposals fare
for level of benefit and cost-effectiveness.
7. Provision
to the Planning Committee of evaluators' "overall impression" rankings for
each project. These rankings may differ
from point score-based rankings; in cases where these differences are
significant, reviewers could provide one-sentence descriptions of why in their
estimations the discrepancies exist.
8. Addition
of new
parties to the evaluation team and solicitation of feedback from the Bicycle and
Pedestrian Issues Task Force during the evaluation period as a source
of "reality checks."
The Round Two Transportation Enhancements Program (TEP)
candidate evaluation methodology applies eleven criteria:
¨
Environmental
Benefit (worth up to 10 points, excluding bonus)
¨
Economic
Benefit (up to 10 points, excluding bonus)
¨
Access/Patronage
Benefits (up to 10 points, excluding bonus)
¨
Transportation
System Enhancement (up to 10 points, excluding bonus)
¨
Local
Benefit and Community Enhancement (up to 15 points, excluding bonus)
¨
Safety
Benefits (up to 15 points, excluding bonus)
¨
Relationship
To/Support for Other Plans, Projects (up to 10 points, excluding bonus)
¨
Size of
Matching Share (up to 5 points, excluding bonus)
¨
Level of
Community, Regional Support -- Letters of Support, Resolutions, Endorsements
(up to 5 points, excluding bonus)
¨
Innovation/Creativity
(up to 5 points, excluding bonus)
¨
Mix of
Eligible Enhancements (up to 5 points, excluding bonus)
The “ranging structure” prepared by CDTC staff presents
details on how scores would be determined for each of these criteria, for
example, what would merit fifteen points for "benefits to immediate
proposal area and environment" as opposed to nine, or none. The structure gives sample indications of
what might garner a proposal different point scores on individual criteria, and
allows the evaluator to review guiding language from the Round One TEP
Guidebook and CDTC interpretations and key in scores accordingly. The structure is reflected in a spreadsheet
template, a completed version of which can be printed out and kept on file for
each proposal. In addition, narrative
rationales for scores may be included in the printouts.
Following presentation of the ranging structure, a sample
application of the new structure to a series of proposals previously considered
by CDTC for both Round One of the TEA-21 TEP and the final round of the ISTEA
TEP is presented, to illustrate the types of changes to evaluation outcomes
which could result from use of the new methodology.
For four criteria, the point scores are direct functions of
some quantifiable attribute:
¨
points for "Relationship to/Support for
Other Plans, Projects" would be assigned based on how many such plans or
projects are cited and/or known (and can validly be considered to be supported
by the proposal)
¨
points for "Size of Matching
Share/Assurance of Availability" would be assigned based on where the
indicated matching share falls into a series of percentage ranges
¨
points under the
"Innovation/Creativity" criterion would be assigned based on
definable unique features and "model project" potential
¨
point assignments under the “mix” criterion
would be based on how many TEP-eligible activities were incorporated into the
proposal
For the remaining criteria, the ranging structure sets forth
four illustrative "levels of success" that a proposal might achieve,
with narrative descriptions of each:
zero percent (no success); 20 percent (low success), 60 percent
("high medium" success), or 100 percent (high success). These levels would correspond to zero, two,
six and ten point scores for the ten point criteria; zero, three, nine and
fifteen point scores for the fifteen point criteria; and zero, one, three and
five point scores for the five point criteria.
Again, these are illustrative;
evaluators would award whatever point scores within the maximums were deemed
appropriate based on individual proposal attributes.
In examining the ranging structure, it should be borne in
mind that by such measures as transportation benefit or economic development,
most TEP proposals would be seen at best as only having "low success"
(that is, two points out of ten) potential compared to activities such as
highway construction or the development of a new office building. It is arguably not appropriate to consider
the potential benefits of Enhancement-type projects against the reference of
all possible investments. Thus, the
maximum potential (100%) “level of success” will be
based on what is possible for TEP-type
projects in the Capital District.
This determination will require a combination of staff knowledge of
existing TEP-type projects (including completed projects which were not funded
under TEP but would have been eligible) and what the theoretical "best
case" benefit of an Enhancement project could be. Staff would document the bases for all point
scorings, and would have this supporting information available if needed when
it presents the results of its reviews to the Planning Committee for approval
before transmission to the statewide Transportation Enhancements Advisory
Committee (TEAC).
|
|
Project
Name |
|
||||
|
Project
Sponsor |
|
|||||
|
Project
Number: |
01-R1-0##-CDTC |
|||||
|
|
SCORE |
0 |
|
|||
|
SCORE-BASED
RANK |
|
|
||||
|
|
|
|
|
|
||
|
"OVERALL
IMPRESSIONS" RANK: |
|
|
||||
|
|
|
|
|
|
||
|
FEASIBILITY
(A/B/C): |
|
(A=HIGH/B=MEDIUM/C=LOW) |
||||
|
|
|
|
|
|
||
|
COST-EFFECTIVENESS
(A/B/C): |
|
(A=HIGH/B=MEDIUM/C=LOW) |
||||
|
|
|
|
|
|
||
|
|
|
|
this |
|
||
|
|
|
max |
proj |
|
||
|
|
|
Score |
Score |
|
||
|
“Benefits” Criterion Group |
|
|||||
|
SUBTOTAL
(Max 70): |
0 |
|
||||
|
|
|
10 |
0 |
B1.
Environmental Benefit |
||
|
|
|
10 |
0 |
B2.
Economic Benefit |
||
|
|
|
10 |
0 |
B3.
Transportation Benefit 1: Enhancement
of Access/Patronage |
||
|
|
|
10 |
0 |
B4.
Transportation Benefit 2: Enhancement
of Transportation System |
||
|
|
|
15 |
0 |
B5.
Local Benefit and Community Enhancement |
||
|
|
|
15 |
0 |
B6.
Safety Benefits |
||
|
"Support" Criterion
Group |
|
|||||
|
SUBTOTAL
(Max 20): |
0 |
|
||||
|
|
|
10 |
0 |
S1.
Relationship To/Support for Other Plans, Projects |
||
|
|
|
5 |
0 |
S2.
Size of Matching Share |
||
|
|
|
5 |
0 |
S3.
Level of Community, Regional Support:
Letters of Support, Resolutions, Endorsements |
||
|
“Innovation” Criterion Group |
|
|||||
|
SUBTOTAL
(Max 10): |
0 |
|
||||
|
|
|
5 |
0.0 |
I1. Innovation/Creativity: Project is innovative or could serve as a
model for similar enhancement projects. |
||
|
|
|
5 |
0.0 |
I2. Mix of Activities: Project encompasses two or more eligible
transportation enhancement activities. |
||
|
|
TOT |
100 |
0 |
|
||
Note: For each
criterion, space will be provided to enter information to clarify the basis for
assigning a particular score. To save
space, this is not represented in the criterion discussions.
Also note that the examples provided for low/medium/high
potentials under each criterion should not be taken to be the only examples with regard to benefiting
groups or project types which could qualify for points at the indicated level
of success. Furthermore, the
"medium" and "high" determinations could be the results of
some cumulative consideration, e.g., the achievement of more than one type of
"low-level" benefit.
|
Benefits Criterion Group |
|
|||
|
|
sample |
max |
this |
|
|
|
raw |
wtd |
proj |
|
|
|
score |
score |
score |
|
|
|
|
10.0 |
0 |
B1. Environmental Benefit |
|
|
|
|
|
|
|
|
|
|
|
Criterion Definition: The extent to which the project would
preserve or positively influence natural, cultural or historic resources,
scenic quality, air or water quality, wildlife habitat or migration. |
|
|
0 |
0.0 |
|
NONE
(project not likely to produce ANY environmental benefit) |
|
|
1 |
2.0 |
| |