APPendix j - ENHANCEMENT Evaluationsappendix j

 

 

Background

 

CDTC developed a basis for evaluating candidates for funding under the Transportation Enhancements Program (TEP).  The basis for Round Two evaluation of proposals from within the Capital District reflects several changes to the Round One methodology.

 

In evaluating proposals to Round One of the TEP, a team of CDTC and New York State Department of Transportation (NYSDOT) Region One staff used a methodology approved by the CDTC Planning Committee on May 5, 1999 which applied criteria set forth in the NYSDOT TEP Guidebook.  The methodology, as detailed in Appendix J of the CDTC’s 2001-06 TIP, provided for point scoring of proposals based on eight criteria:

 

¨      Benefit to Enhancement Region and Environment

¨      Enhancement of Transportation Plans, Projects

¨      Relationship To/Support for Other Plans, Projects

¨      Size of Matching Share, Assurance of Availability

¨      Direct User Benefits to Immediate Proposal Area and Environment

¨      Innovation/Creativity/Mix of Activities

¨      Supportiveness of Master Planning in Recognized Areas of Special Significance

¨      Level of Community, Regional Support

 

While the Guidebook-based methodology was helpful in providing a framework for objective assessment of the merits of each proposal, discussions after the completion of Round One raised several concerns with this basis, particularly the following:

 

¨      the criterion set seemed to place too much weight on regional rather than local benefits

¨      on a related note, trails or other sorts of "tourism-oriented" proposals tended to be the only ones which could score well on most or all of the criteria

¨      safety benefits seemed not to get adequate weight

¨      partly related to the safety point, many important projects in urban areas did not see their true benefits reflected in point scores

¨      there was no explicit opportunity to consider cost-effectiveness in rankings

 

As a result, while there were 12 types of projects eligible for TEP funding, most proposals faring well in the rankings were for trail developments or extensions.  In addition, higher-cost proposals tended to fare better than lower-cost proposals. 

 

 

Summary of Modifications from Round One

 

For Round One of the TEP, the CDTC/NYSDOT evaluation team needed to base its evaluations on the Guidebook, for applicants developed their proposals based on the Guidebook's listing of criteria.  That is, as the Guidebook contained the first indications of what the basis for proposal evaluation would be, the team did not have the opportunity to introduce additional criteria, for it would arguably have been unfair to applicants to have their proposals evaluated based on a different set of criteria from those presented in the Guidebook.

 

For Round Two, a CDTC document discussing the evaluation methodology to be applied to Capital District proposals was provided to potential applicants at the informational workshop held at the start of the Round Two TEP effort and thereafter, with the CDTC document positioned as a companion document to the Guidebook.  The CDTC document noted that while the CDTC evaluations would be more rigorous, considering criteria beyond those set forth in the Guidebook, applicants would not be compelled to do any more work in preparing proposals than would be expected based strictly on the Guidebook.

 

CDTC and NYSDOT Region One staff met to discuss their concerns with the Round One approach and to identify possible changes to the evaluation methodology which would ensure that evaluations of Round Two proposals would be based on broader opportunities for success.  The group identified a series of modifications to the Round One evaluation process dealing with criteria and process.  The CDTC Planning Committee discussed and concurred with these possible modifications at its March 7, 2001 meeting.  The new approach reflects eight key changes:

 

1.      Presentation of evaluation findings and preliminary rankings to the Planning Committee by Enhancements project category, to facilitate Committee consideration of prioritized candidate lists reflecting a wider range of project types.

2.      Addition of a new sub-criterion within the "Benefits" criterion group (see Modification 4 for a discussion of criterion groups) dealing with safety impacts.  (In addition, as will be detailed later on in this document, the descriptions of the existing criteria in this group have been modified for purposes of clarity and consistency.)

3.      Reallocation of maximum point scores to reduce bias toward very large projects.

4.      The aggregation of criteria into three criterion groups within which evaluators would have a defined degree of flexibility in allocating points.

5.      Provision to the Planning Committee of indications of how the proposals fare under supplemental screening criteria employing "A" through "C" grades for feasibility and cost-effectiveness.

6.      Application of model-based or otherwise quantitative assessments of potential proposal impact wherever possible (e.g., using the bicycle and/or pedestrian versions of the CDTC regional travel model).  These assessments would be inputs to evaluator consideration of how proposals fare for level of benefit and cost-effectiveness.

7.      Provision to the Planning Committee of evaluators' "overall impression" rankings for each project.  These rankings may differ from point score-based rankings; in cases where these differences are significant, reviewers could provide one-sentence descriptions of why in their estimations the discrepancies exist.

8.      Addition of new parties to the evaluation team and solicitation of feedback from the Bicycle and Pedestrian Issues Task Force during the evaluation period as a source of "reality checks."

 

 

Ranging Structure for Project Evaluation

 

The Round Two Transportation Enhancements Program (TEP) candidate evaluation methodology applies eleven criteria:

 

¨      Environmental Benefit (worth up to 10 points, excluding bonus)

¨      Economic Benefit (up to 10 points, excluding bonus)

¨      Access/Patronage Benefits (up to 10 points, excluding bonus)

¨      Transportation System Enhancement (up to 10 points, excluding bonus)

¨      Local Benefit and Community Enhancement (up to 15 points, excluding bonus)

¨      Safety Benefits (up to 15 points, excluding bonus)

¨      Relationship To/Support for Other Plans, Projects (up to 10 points, excluding bonus)

¨      Size of Matching Share (up to 5 points, excluding bonus)

¨      Level of Community, Regional Support -- Letters of Support, Resolutions, Endorsements (up to 5 points, excluding bonus)

¨      Innovation/Creativity (up to 5 points, excluding bonus)

¨      Mix of Eligible Enhancements (up to 5 points, excluding bonus)

 

The “ranging structure” prepared by CDTC staff presents details on how scores would be determined for each of these criteria, for example, what would merit fifteen points for "benefits to immediate proposal area and environment" as opposed to nine, or none.  The structure gives sample indications of what might garner a proposal different point scores on individual criteria, and allows the evaluator to review guiding language from the Round One TEP Guidebook and CDTC interpretations and key in scores accordingly.  The structure is reflected in a spreadsheet template, a completed version of which can be printed out and kept on file for each proposal.  In addition, narrative rationales for scores may be included in the printouts.

 

Following presentation of the ranging structure, a sample application of the new structure to a series of proposals previously considered by CDTC for both Round One of the TEA-21 TEP and the final round of the ISTEA TEP is presented, to illustrate the types of changes to evaluation outcomes which could result from use of the new methodology.

 

 

Structure Basis

 

For four criteria, the point scores are direct functions of some quantifiable attribute:

 

¨      points for "Relationship to/Support for Other Plans, Projects" would be assigned based on how many such plans or projects are cited and/or known (and can validly be considered to be supported by the proposal)

¨      points for "Size of Matching Share/Assurance of Availability" would be assigned based on where the indicated matching share falls into a series of percentage ranges

¨      points under the "Innovation/Creativity" criterion would be assigned based on definable unique features and "model project" potential

¨      point assignments under the “mix” criterion would be based on how many TEP-eligible activities were incorporated into the proposal

 

For the remaining criteria, the ranging structure sets forth four illustrative "levels of success" that a proposal might achieve, with narrative descriptions of each:  zero% (no success); 20% (low success), 60% ("high medium" success), or 100% (high success).  These levels would correspond to zero, two, six and ten point scores for the ten point criteria; zero, three, nine and fifteen point scores for the fifteen point criteria; and zero, one, three and five point scores for the five point criteria.  Again, these are illustrative; evaluators would award whatever point scores within the maximums were deemed appropriate based on individual proposal attributes.

 

In examining the ranging structure, it should be borne in mind that by such measures as transportation benefit or economic development, most TEP proposals would be seen at best as only having "low success" (that is, two points out of ten) potential compared to activities such as highway construction or the development of a new office building.  It is arguably not appropriate to consider the potential benefits of Enhancement-type projects against the reference of all possible investments.  Thus, the maximum potential (100%) “level of success” will be based on what is possible for TEP-type projects in the Capital District.  This determination will require a combination of staff knowledge of existing TEP-type projects (including completed projects which were not funded under TEP but would have been eligible) and what the theoretical "best case" benefit of an Enhancement project could be.  Staff would document the bases for all point scorings, and would have this supporting information available if needed when it presents the results of its reviews to the Planning Committee for approval before transmission to the statewide Transportation Enhancements Advisory Committee (TEAC).

 


Ranging Structure Summary Sheet

 

 

Project Name

 

Project Sponsor

 

Project Number:

01-R1-0##-CDTC

 

SCORE

0

 

SCORE-BASED RANK

 

 

 

 

 

 

 

"OVERALL IMPRESSIONS" RANK:

 

 

 

 

 

 

 

FEASIBILITY (A/B/C):

 

(A=HIGH/B=MEDIUM/C=LOW)

 

 

 

 

 

COST-EFFECTIVENESS (A/B/C):

 

(A=HIGH/B=MEDIUM/C=LOW)

 

 

 

 

 

 

 

 

this

 

 

 

max

proj

 

 

 

Score

Score

 

“Benefits” Criterion Group

 

SUBTOTAL (Max 70):

0

 

 

 

10

0

B1. Environmental Benefit

 

 

10

0

B2. Economic Benefit

 

 

10

0

B3. Transportation Benefit 1:  Enhancement of Access/Patronage

 

 

10

0

B4. Transportation Benefit 2:  Enhancement of Transportation System

 

 

15

0

B5. Local Benefit and Community Enhancement

 

 

15

0

B6. Safety Benefits

"Support" Criterion Group

 

SUBTOTAL (Max 20):

0

 

 

 

10

0

S1. Relationship To/Support for Other Plans, Projects

 

 

5

0

S2. Size of Matching Share

 

 

5

0

S3. Level of Community, Regional Support:  Letters of Support, Resolutions, Endorsements

“Innovation” Criterion Group

 

SUBTOTAL (Max 10):

0

 

 

 

5

0.0

I1.  Innovation/Creativity:  Project is innovative or could serve as a model for similar enhancement projects.

 

 

5

0.0

I2.  Mix of Activities:  Project encompasses two or more eligible transportation enhancement activities.

 

TOT

100

0

 

 


Detailed Discussions of Criteria and Scores

 

Note:  For each criterion, space will be provided to enter information to clarify the basis for assigning a particular score.  To save space, this is not represented in the criterion discussions.

 

Also note that the examples provided for low/medium/high potentials under each criterion should not be taken to be the only examples with regard to benefiting groups or project types which could qualify for points at the indicated level of success.  Furthermore, the "medium" and "high" determinations could be the results of some cumulative consideration, e.g., the achievement of more than one type of "low-level" benefit.

 

Benefits Criterion Group

 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

10.0

0

B1. Environmental Benefit

 

 

 

 

 

 

 

 

 

Criterion Definition:  The extent to which the project would preserve or positively influence natural, cultural or historic resources, scenic quality, air or water quality, wildlife habitat or migration.

 

0

0.0

 

NONE (project not likely to produce ANY environmental benefit)

 

1

2.0

 

LOW (project likely to have some minimal environmental benefit)

 

3

6.0

 

MEDIUM (project likely to have modest but significant environmental benefit)

 

5

10.0

 

HIGH (project likely to provide substantial environmental benefit)

 

7.5

15.0

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

In general, this category focuses on the "natural" environment; however, the Guidebook notes that the rater has broad discretion to interpret and define these terms.  For example, the rater will form an opinion on "what is a cultural resource?" (there are many answers: one interpretation might be "areas of historical or archeological significance", while another could be "areas where human social interactions may occur").   Examples of indicators might be:

 

 

 

 

·         Natural resources conserved or protected

 

 

 

 

·         Cultural resources conserved or protected

 

 

 

 

·         Historic resources preserved or enhanced

 

 

 

 

·         Scenic quality preserved or enhanced

 

 

 

 

·         Air and/or water quality directly improved as a result

 

 

 

 

·         Wildlife habitat/migration areas are preserved, restored, created, or otherwise enhanced


 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

10.0

0

B2.  Economic Benefit

 

 

 

 

Criterion Definition:  The extent to which the project would improve the quality of life through job creation, increased tourism, economic development, balanced distribution of funds and other socio-economic factors.  Should be considered in the context of what is possible in these areas for an Enhancements-level project.

 

0

0.0

 

NONE (project not likely to produce ANY economic benefit)

 

1

2.0

 

LOW (project likely to have some minimal economic benefit)

 

3

6.0

 

MEDIUM (project likely to have modest but significant economic benefit)

 

5

10.0

 

HIGH (project likely to provide substantial economic benefit)

 

7.5

15.0

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

This category focuses on the potential for positive economic impacts resulting from an enhancement project.  Examples follow:

 

 

 

 

·         Additional jobs are created in the community

 

 

 

 

·         Existing jobs will be retained within the community

 

 

 

 

·         Tourism and visitor revenues will be enhanced through:

 

 

 

 

·         Additional hotel occupancy, increased restaurant and retail sales

 

 

 

 

·         Potential for "return-trips" increased

 

 

 

 

·         Economic Development potential (e.g. marketability of the community) is enhanced through:

 

 

 

 

·         Improved community aesthetics

 

 

 

 

·         Perception of a higher "quality of life"

 

 

 

 

·         Economically challenged individuals are assisted.

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

10.0

0

B3. Transportation Benefit 1:  Enhancement of Access/Patronage

 

 

 

 

 

 

 

 

 

Criterion Definition:  The extent to which the project would increase or improve access to activity centers (business, school, recreation, shopping, etc.).  Access is to be considered both in a general sense and for particular groups such as people with disabilities.

 

0

0.0

 

NONE (project not likely to produce ANY transportation access/patronage benefit)

 

1

2.0

 

LOW (project likely to have some minimal transportation access/patronage benefit)

 

3

6.0

 

MEDIUM (project likely to have modest but significant transportation access/patronage benefit)

 

5

10.0

 

HIGH (project likely to provide substantial transportation access/patronage benefit)

 

7.5

15.0

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

The focus of this category should be on the enhanced mobility (especially with non-traditional modes) of persons or on significant improvement in the quality of the trip experience (improved access to sites, etc.).  Examples here are best expressed in the form of questions:

 

 

 

 

·         What activity centers will be connected?  Are the connections genuinely enhanced?

 

 

 

 

·         What is the current level of connectivity/access (i.e. how dramatic are effects of the proposed improvements)?

 

 

 

 

·         Is user safety/security a current issue?

 

 

 

 

·         Is access guaranteed to all individuals (e.g. ADA, private ownership are examples of issues)?


 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

10.0

0

B4. Transportation Benefit 2:  Enhancement of Transportation System

 

 

 

 

Criterion Definition:  The extent to which the project would build, extend or connect local and regional transportation systems for the purpose of facilitating non-motorized and/or intermodal travel.

 

0

0.0

 

NONE (project not likely to produce ANY transportation system enhancement)

 

1

2.0

 

LOW (project likely to have some minimal transportation system enhancement)

 

3

6.0

 

MEDIUM (project likely to have modest but significant transportation system enhancement)

 

5

10.0

 

HIGH (project likely to provide substantial transportation system enhancement)

 

7.5

15.0

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

This category concentrates on the development of the intermodal transportation system envisioned by the ISTEA legislation and reinforced in TEA-21.  Whereas the previous category looked at how the proposed project meets user "demand", this category looks at the "supply" aspects of the transportation equation.  Examples include:

 

 

 

 

·         Transportation modes being connected (e.g. bikes and pedestrians, bikes and buses, bikes and autos, trains and pedestrians, etc.).  Also, projects identified in transportation plans; a part of continuing or ongoing transportation programs.

 

 

 

 

·         System deficiencies being addressed   (e.g.  Pedestrian circulation systems, bikeway systems, etc.). 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

15.0

0

B5.  Local Benefit and Community Enhancement

 

 

 

0

 

 

 

 

 

Criterion Definition:  The extent to which the project would provide or increase recreational or transportation opportunities for its immediate neighbors, and would be a neighborhood amenity rather than a source of local consternation.

 

0

0.0

 

NONE (project not likely to produce ANY direct user benefits of these sorts)

 

1

3.0

 

LOW (project likely to have some minimal direct user benefits of these sorts)

 

3

9.0

 

MEDIUM (project likely to have modest but significant direct user benefits of these sorts)

 

5

15.0

 

HIGH (project likely to provide substantial direct user benefits of these sorts)

 

7.5

22.5

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

The aim of this criterion is to encourage consideration of the positive impacts a project will have at the most local level.  It is important because some projects can have significant regional benefit while having little benefit -- if not in fact being a nuisance -- to their immediate neighbors. Among the considerations for this criterion will be the extent to which the project would do the following:

 

 

 

 

·         Create or enhance some sort of opportunity which is genuinely likely to be used by its neighbors.

 

 

 

 

·         Preserve community resources (e.g. neighborhoods, cultural facilities, gathering areas, etc.).

 

 

 

 

·         Enhance neighborhood ambiance or safety


 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

15.0

0

B6.  Safety Benefits

 

 

 

0

 

 

 

 

 

Criterion Definition:  The extent to which the project would enhance safety, particularly the safety of cyclists or pedestrians.  Can also consider the benefit to all travelers attendant to remedying a known hazardous condition such as a poor line of sight or narrow roadway.

 

0

0.0

 

NO safety benefit expected (proposal is for an effort which would in itself have no discernible impact on any kind of safety problem, e.g., a planning or historic preservation project)

 

1

3.0

 

MODEST safety benefit expected (proposal includes improvements which will call attention to or otherwise provide guidance regarding a safety problem, but will not substantially resolve the problem, e.g., cautionary signage, crosswalks or other non-capital improvements)

 

3

9.0

 

MEDIUM safety benefit expected (proposal includes elements which will substantially resolve but not eliminate a safety problem,  e.g., bikeable shoulder/bike lane construction)

 

5

15.0

 

HIGH safety benefit expected (proposal includes a hazard remediation or development of a new facility which would effectively eliminate a known safety problem)

 

7.5

22.5

 

EXTRAORDINARY  (150% of maximum)


 

Support Criterion Group

 

 

sample

max

this

 

 

Raw

wtd

proj

 

 

Score

score

score

 

 

 

10.0

0

S1. Relationship To/Support for Other Plans, Projects:

 

 

 

 

 

 

 

 

 

Criterion Definition:  The extent to which the project would implement goals in regional plans or other federal, state or local plans. 

 

0

0.0

 

NONE (project not likely to further any local, regional or state plan goals)

 

1

2.0

 

LOW (project likely to further goals in one cited or known plan)

 

3

6.0

 

MEDIUM (project likely to further goals in two or three cited or known plans)

 

5

10.0

 

HIGH (project likely to further goals in more than three cited or known plans)

 

7.5

15.0

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

The linkage to existing plans is critical.  This is particularly true for projects within urbanized areas under the jurisdiction of a Metropolitan Planning Organization (MPO).  Under the law, MPOs must not only approve projects for programming in their Transportation Improvement Program (TIP), the projects must relate to a Long Range Plan. Remember that a formal action from a MPO (e.g. resolution) represents the full support and approval of all of the member governments and participants in the metropolitan region.  If a project is known to be consistent with, or actually may implement some aspect of various plans, ordinances, local master plans, etc., it is appropriate to make note of that fact.

 

sample

max

this

 

 

Raw

wtd

proj

 

 

Score

score

score

 

 

 

5.0

0

S2.Size of Matching Share

 

 

 

 

 

% MATCH:

 

 

Criterion Rationale:  A 20% minimum match is required; the provision of a match in excess of 20% benefits the overall program as it allows federal funds to be used for additional enhancement projects.

 

0

0.0

 

<20% match (ineligible)

 

1

1.0

 

20-24.99% match

 

3

3.0

 

25-34.99% match

 

5

5.0

 

35% or greater match

 

7.5

7.5

 

EXTRAORDINARY  (150% of maximum)

 

 

 

 

NYSDOT Bonus Category Cutoffs (key bonus score into green box at left)

 

 

 

0.0

20%

 

 

 

1.0

21-30%

 

 

 

2.0

31-40%

 

 

 

3.0

41-50%

 

 

 

4.0

51-60%

 

 

 

5.0

60% or greater


 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

5.0

0

S3.Level of Community, Regional Support:  Letters of Support, Resolutions, Endorsements

 

 

 

 

 

 

 

 

 

Criterion Definition:  Letter(s) of support from elected officials; endorsement action from local governments (resolutions, etc.); endorsement action from local governments (resolutions, etc.);letters of support/endorsement actions from interest groups (e.g. Chambers of  Commerce, advocacy groups, neighborhood associations, etc.)  

 

 

 

 

This is a critical category in that it represents the level of community and political support for the project.  While transportation projects are often delayed (or terminated) as a result of significant opposition, projects that have the full backing of community groups and  leaders/elected officials have a higher completion rate. Projects that demonstrate evidence of a combination of both "grass roots" support and support from the appropriate officials are more favorable than those that do not.  The degree of support is also critical: letters from individuals are good, but resolutions, petitions, or other formal actions of support by groups of people are better.

 

0

0.0

 

NO evidence of support provided

 

1

1.0

 

SOME (LOW) support (e.g., one or two letters from individual citizens included in the proposal)

 

3

3.0

 

MEDIUM support (e.g., one or two letter(s) from the parties described under the "Criterion Definition"  section)

 

4

4.0

 

HIGH support (e.g., letter(s) from the parties described under the "Criterion Definition"  section, plus some official support, e.g., a resolution)

 

5

5.0

 

HIGHEST support (outstanding indication of support, e.g., considerable quantity of letters/resolutions, indication of plans by outside parties to provide assistance with project implementation)

 

7.5

7.5

 

EXTRAORDINARY  (150% of maximum)


 

Innovation Criterion Group

 

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

Score

 

 

 

5.0

0

I1.  Innovation/Creativity:  Project is innovative or could serve as a model for similar enhancement projects.

 

 

 

 

 

 

 

 

 

Criterion Definition:  The level of "innovativeness" or the suitability of the project as a "model" for other projects.  Unique design or application, new technologies, development of public/private partnerships and multi-jurisdictional projects, are all good examples.

 

0

0.0

 

·         Project is routinely organized, designed, planned

 

1

1.0

 

·         Project has a couple of unique characteristics

 

3

3.0

 

·         Project has unique characteristics / some model potential

 

5

5.0

 

·         Project is extremely unique / definitely a model

 

7.5

7.5

 

·         EXTRAORDINARY  (150% of maximum)

 

sample

max

this

 

 

raw

wtd

proj

 

 

score

score

score

 

 

 

5.0

0

I2. Mix of Activities:  Project encompasses two or more eligible transportation enhancement activities.

 

 

 

 

 

 

0

0.0

0

·         1 eligible activity

 

3

3.0

0

·         2-3 eligible activities

 

5

5.0

0

·         4-5 eligible activities

 

7.5

7.5

0

·         EXTRAORDINARY (6+ eligible activities)

 

 

 

0

<-- DOT "multiple activities" score (no need to key in -- will be copied to DOT sheet)

 

 

 

 

Many transportation enhancement project proposals may technically encompass two or more eligible activities.  If they do, the TEAC will consider this fact in their rating.  However, each individual aspect of the proposal should "stand alone" in the sense; if the project were split by category, each would qualify on its own merits: (e.g. landscaping might be only a side-effect to the development of scenic overlook and probably would not receive extra credit).