Loading...
The URL can be used to link to this page
Your browser does not support the video tag.
Home
My WebLink
About
0010_3_Community Opinion Survey - Exhibit B Proposal
THIS PAGE INTENTIONALLY LEFT BLANK Table of ContentsiCity of Lake Elsinore True North Research, Inc. © 2016. . . . . . . . . . . . . . . . T ABLE OF CONTENTS Table of Contents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Cover Letter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Sets True North Apart?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Qualifications & Experience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Personal vs. Firm-Level Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Municipal Surveys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Additional Riverside County Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Project Organization & Staffing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Resumes and Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Survey Approach & Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Trade Secret Notification: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Innovative Methods for a Changing Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 4 Questionnaire. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Pre-Test & Programing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Three-Pronged Recruiting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Mixed-Method Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Data Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Analysis & Toplines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 8 Survey Report. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Benchmarking & Statistical Significance Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Presentation & Consulting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Time Line. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Quality Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Compensation/Payment Schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Fixed-Fee Cost Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Insurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Standard Agreement & Term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Contract. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Proposal Term . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Cover LetterTrue North Research, Inc. © 2016 1City of Lake Elsinore . . . . . . . . . . . . . . . . C OVER LETTER True North Research, Inc. (True North) is pleased to present this proposal to the City of Lake Elsinore to design and conduct a statistically reliable survey of adult residents in the city. Based on the RFQ, as well as our experience conducting hundreds of similar surveys for California cit- ies, True North understands that the City has two main research interests for the proposed study. The first is to use the survey as a community needs assessment and performance mea- surement tool. That is, the survey is an opportunity to profile residents’ needs and priorities, measure how well the City is performing in meeting these needs through existing services and facilities, and gather data on a variety of quality-of-life, issue, and policy-related matters. Like many of True North’s municipal clients, however, the City of Lake Elsinore also recognizes the value of a survey for developing a long-term strategic plan. What are residents’ long-term aspirations and hopes for Lake Elsinore’s future? How would they prioritize among a variety of strategic initiatives the City could pursue? In what areas do residents feel that the City can improve the way it communicates and engages with the community? Answers to questions like these are just as important as service or policy-related questions to helping the City meet resi- dents’ evolving expectations and needs. By analyzing the results of the 2016 Community Opinion Survey in conjunction with past survey results (if available) and other secondary data, the proposed study will provide Council and staff with information that can be used to make sound, strategic decisions in a variety of areas— including prioritization, planning, performance measurement, policymaking, community engagement, and budgeting. CHALLENGES To successfully meet the challenges of the proposed study, the City will need a consultant team that has a unique set of skills and experiences. First and foremost, the team should possess a highly sophisticated understanding of survey methodology to ensure that the resulting data are valid and reliable measures of respondents' opinions and that the results are representative of the population of adult residents in the City of Lake Elsinore—including hard to reach subgroups. This expertise should extend to all aspects of conducting surveys, including sampling design, questionnaire design, data collection, data processing, weighting, statistical analysis, and presentation. In addition to being experts in survey methodology, the consultant team should also have a sound understanding of public administration, including the purpose, processes, and challenges associated with providing public services and how to use survey research to guide service improvements. And, of course, the consultant should have extensive experience working closely with city staff, council and other consulting firms to translate research results into clear and actionable recommendations. WHAT SETS TRUE NORTH APART? The True North team has the expertise and experi- ence needed to successfully overcome the challenges of the proposed study. Although we pro- vide more details in the body of this proposal, we thought it would be helpful to highlight at the outset what distinguishes True North from other research firms. Unmatched Experience & Expertise Conducting Community Opinion Surveys for CA Cities True North’s President (Dr. Timothy McLarney) and Principal Researcher (Richard Sarles) have personally designed and conducted more surveys to assist California cities in meeting their per- Cover LetterTrue North Research, Inc. © 2016 2City of Lake Elsinore . . . . . . . . . . . . . . . . formance goals than any other researchers in the State. To date, our team has conducted over 900 surveys for public agencies, including more than 300 community opinion surveys for Cali- fornia cities. The advantage of working with True North is that our experience will not only allow us to be a very active, insightful collaborator with the City of Lake Elsinore when designing the survey, it will also enable us to provide reliable benchmarking information and meaningfully interpret (and provide context to) the results of the City’s survey. Who You See is Who You Get Unlike most in the industry, True North firmly believes that the best way to conduct your project is to have the people in our organization with the most exper- tise and experience roll-up their sleeves and actually do the work. Whereas other firms typically follow the bait-and-switch model for managing projects where most of the actual work is dele- gated to low-level staff, a key advantage of selecting our team is that who you see is who you get. True North’s President (Dr. McLarney) will personally handle 80% of the research tasks for the proposed study. Innovative Methods for a Changing Environment Much has changed during the past seven years in terms of how the public receives information, the accessibility of residents through tra- ditional recruiting methods, and their willingness to participate in community surveys. Tech- niques that traditionally worked well for surveying residents in the past (such as random digit dial—RDD) are no longer nearly as effective at producing reliable samples due to factors such as call-screening and the growing percentage of households that rely solely on cell phones. More- over, relying solely on telephone-based data collection methodologies lowers participation rates, introduces response bias, and ultimately lowers the reliability of survey results. The bottom line is in order to continue producing statistically reliable results, the research methodologies that cities use for community surveys must to adapt to these new challenges. At True North, we have been at the forefront in developing innovative survey methodologies to overcome these new challenges and ensure statistically reliable results. For the City of Lake Elsinore, we recommend using a mixed-method survey design in which sampled residents will be recruited to participate in the survey using a using a combination of mailed letters, email and telephone calls. We will also use a mixed-method data collection approach that will provide sam- pled residents with the flexibility to participate by telephone or online at a secure, password pro- tected web site. Collectively, our recommended approach will increase participation rates, reduce response bias, and increase the overall accuracy and reliability of the City’s survey results when compared to using telephone data collection only. The research team at True North is eager to partner with the City of Lake Elsinore on this impor- tant study. We think that you will find our qualifications, our attention to client service, and our interest in this study to be exceptional. Should you have any questions about this proposal, please do not hesitate to contact me by email at mclarney@tn-research.com or by phone at 760.632.9900. I will serve as the City’s point of contact during the selection process and as the Project Manager for the duration of the study, if selected. Cover LetterTrue North Research, Inc. © 2016 3City of Lake Elsinore . . . . . . . . . . . . . . . . Sincerely, Timothy McLarney, Ph.D. President True North Research 1061 Neptune Avenue Encinitas CA 92024 760.632.9900 mclarney@tn-research.com Qualifications & ExperienceTrue North Research, Inc. © 2016 4City of Lake Elsinore . . . . . . . . . . . . . . . . Q UALIFICATIONS & EXPERIENCE True North is a full-service survey research firm that is dedicated to providing California cities and public agencies with a clear understanding of the opinions, perceptions, priorities and con- cerns of their residents and customers. Through designing and implementing scientific surveys, focus groups and one-on-one interviews, as well as expert interpretation of the findings, True North helps its clients to move with confidence when making strategic decisions in a variety of areas—such as planning, policy evaluation, performance management, establishing budget pri- orities, passing revenue measures, and developing effective public outreach campaigns. Designing and implementing custom survey research studies that adhere to rigorous method- ological standards is both the purpose and the passion of True North Research. In addition to being nationally recognized experts in survey research methodology,1 Dr. McLarney and Mr. Sarles have also personally designed and conducted more research studies to assist Cali- fornia cities in meeting their performance goals than any other individual researchers in the State. The advantage of working with True North is that this experience is particularly help- ful not only in the design of the research, it also enables our team to provide reliable benchmark- ing information and meaningfully interpret (and provide context to) the results of the surveys. This includes the ability to compare the City of Lake Elsinore’s performance to those of other municipalities in the region, throughout California, and/or across the nation,2 if desired. True North Research was co-founded in 2002 as a two-person consulting firm by Dr. McLarney and Mr. Sarles and is an S-Corporation in good standing with the State of California. Although we have had many opportunities to expand our business during the past 14 years, we have chosen to maintain True North as a two-person firm. Dr. McLarney and Mr. Sarles still manage every aspect of every True North project. We are an extremely efficient, well-oiled, collaborative team of two—and we have chosen to keep it that way so that our clients continue to receive exception- ally high quality customer service and research insights. Dr. McLarney and Mr. Sarles are commit- ted to working with the City of Lake Elsinore for the duration of this project—there will be no staff turnover or changes. PERSONAL VS. FIRM-LEVEL EXPERIENCE Before discussing our experience in more detail, we feel it is important to note the difference between personal experience and firm-level experience. It is very common for research firms to provide a long list of projects in their propos- als in an effort to demonstrate that they are both well-qualified and highly experienced in con- ducting surveys of this type. The reality, however, is that this firm-level experience is often a game of smoke-and-mirrors. What they don’t tell you is that their list of projects was managed by dozens of different project managers over many years, most of whom no longer work for the firm and only one of whom will be assigned to your project (along with lower-level support staff). 1. For example, Dr. McLarney has served as an expert witness in survey research methodology for legal cases, has published research methodologies in peer-reviewed academic journals, and on the basis of his research has been honored as a Visiting Scholar at U.C. Berkeley’s Institute for Governmental Studies. 2. True North sponsored a national survey to gather reliable data on how residents view the performance of their local governments across the country. Qualifications & ExperienceTrue North Research, Inc. © 2016 5City of Lake Elsinore . . . . . . . . . . . . . . . . The bottom line is that it doesn’t matter how many projects or clients a firm has worked with over the years. Nor does the experience of the Principals or others at the firm matter if they aren’t going to be the ones rolling-up their sleeves to work on your study. What matters is the experience and expertise of the specific individuals who will be working directly on your project and handling each of the key research tasks. For this reason, we take a different approach. The experience discussed throughout this pro- posal—and the projects referenced below—are the personal experience of the individuals who will be working on all aspects of the study for the City of Lake Elsinore: Dr. McLarney and Mr. Sar- les. We encourage the City to take this difference into account when reviewing firms and propos- als. MUNICIPAL SURVEYS During their careers, Dr. McLarney and Mr. Sarles have designed and conducted over 900 survey research studies for public agencies—including more than 300 community surveys for California municipalities that had research goals similar to those expressed by the City of Lake Elsinore. Below is a sampling of the California cities for which Dr. McLarney and Mr. Sarles have personally designed and conducted community surveys. ADDITIONAL RIVERSIDE COUNTY EXPERIENCE In addition to the cities noted above, True North has also conducted dozens of surveys for other public agencies in Riverside County, including WRCOG, I-15 Interregional Partnership, Murrieta Valley USD, Temecula Valley USD, Moreno Valley USD, Corona-Norco USD, Perris SD, Val Verde USD, Desert Sands USD, and College of the Desert. Agoura Hills Encinitas Lompoc Riverside Aliso Viejo Folsom Long Beach Rocklin Anaheim Fontana Los Angeles San Carlos Apple Valley Fremont Los Gatos San Clemente Atherton Gilroy Malibu San Diego Azusa Glendale Manhattan Beach San Gabriel Banning Glendora Mission Viejo San Jose Beverly Hills Grand Terrace Moorpark San Marcos Brea Half Moon Bay Murrieta San Marino Burbank Hesperia Norwalk Santa Clarita Campbell Huntington Beach Oxnard Santa Monica Cerritos Indian Wells Palmdale Temecula Chino Irvine Petaluma Thousand Oaks Chula Vista La Canada-Flintridge Pico Rivera Truckee Claremont Laguna Beach Placentia Upland Clayton Laguna Niguel Port Hueneme Ventura Costa Mesa Lake Forest Rancho Cucamonga Westlake Village Dana Point La Mesa Rancho Palos Verdes Watsonville Del Mar Lathrop Rancho Santa Margarita West Hollywood Diamond Bar La Verne Redlands Whittier El Cajon Lemon Grove Redondo Beach Yorba Linda Project Organization & StaffingTrue North Research, Inc. © 2016 6City of Lake Elsinore . . . . . . . . . . . . . . . . P ROJECT ORGANIZATION & STAFFING Most research firms manage survey projects through division of labor and a lot of delegation. Although the Principals may appear for the interview and for client meetings, their involvement beyond that point is generally negligible. The Project Manager’s role is to be the point of contact and help design the study, but even they only handle about 20% of the workload. The unfortu- nate reality is that most of the important research tasks—data processing, data analysis, report writing, presentation building—are delegated to junior-level staff. The division of labor and delegation model is popular among research firms because it is scal- able and built for quantity. Any firm that has just two or three Principals, yet boasts multiple offices and several hundred studies every year, is certainly relying heavily on division of labor and delegation to get the work done. However, it isn’t a model that is consistent with high qual- ity research, careful attention to a client’s needs, or insightful strategic advice. The problem with division of labor and delegation is that it creates a disconnect in the research process. The people doing the heavy lifting not only lack the experience and expertise needed to analyze the data appropriately and provide insightful recommendations, they also do not have a first-hand understanding of a client’s needs because junior-level staff generally have little or no client interaction throughout a project. Moreover, under this model the Project Manager attempts to provide recommendations without having personally analyzed the data, written the report, or even prepared the Powerpoint summary. We do things differently at True North. In fact, one of the main reasons we founded True North 14 years ago was because we recognized that there is a much better model for managing a research project—have the person with the most experience and expertise in the firm roll up his/ her sleeves and actually do the work! At True North, who you see is who you get. Dr. McLarney will not only serve as the Project Manager and the day-to-day contact for the District, he will per- sonally manage 80% of the research tasks on the survey project including all design, analysis, report writing, and presentations. We have purposely kept True North as a single office, two-per- son consulting firm because of our commitment to this project management model and the many benefits it provides to our clients. FIGURE 1 PROJECT ORGANIZATION Figure 1 presents the organizational chart for the proposed study. True North Research will serve as prime contractor at the direc- tion of the City of Lake Elsinore. Dr. Timothy McLarney (President of True North) will be the Project Manager for the study and will be assisted by Richard Sarles (Principal Researcher at True North). The telephone interviews will be conducted by the same high quality, professional phone bank that True North has teamed with to complete more than 300 resident and community sur- veys—MWRC. Dameon Travis (Senior City of Lake Elsinore True Nort h Research, Inc. MWRC Timothy McLarney, Ph.D. Richard Sarles, M.S. Dameon Travis, B.A. Project Organization & StaffingTrue North Research, Inc. © 2016 7City of Lake Elsinore . . . . . . . . . . . . . . . . Account Manager at MWRC) will manage the day-to-day interviewing schedule and quality control procedures in close consultation with Mr. Sarles and Dr. McLarney. RESUMES AND ROLES For the City’s reference, the following paragraphs provide addi- tional information about Dr. McLarney’s, Mr. Sarles’ and Mr. Travis’ respective roles on this proj- ect, as well as their education and research backgrounds. Project Manager Timothy McLarney, Ph.D., will serve as the Project Manager for the pro- posed survey and the City’s point of contact on all matters related to the study. Dr. McLarney will lead all design, analysis, reporting and presenta- tion tasks. Approximately 80% of the research tasks will be managed directly by Dr. McLarney. As President of True North, Dr. McLarney is responsible for the design, management and analysis of True North's qualitative and quantitative research projects, including those that address community needs assess- ments, customer satisfaction, revenue measure feasibility, organiza- tional development, public policy and strategic planning issues. Dr. McLarney's work to date has provided California cities, counties, special jurisdictions, transportation planning agencies, councils of government, school districts, corporations, and political campaigns with research to address their often complex marketing, planning and/or performance measurement needs. During his career, Dr. McLarney has occupied a key role in over 900 research studies, including more than 300 resident sat- isfaction surveys for California cities. Dr. McLarney is a nationally recognized expert in survey research meth- odology, sampling theory, weighting and the use of statistical methods to generalize survey results. His research has been recognized at numer- ous national and state conferences, has been published in academic journals, and has earned him honors including the title of Visiting Scholar at the Institute of Governmental Studies at UC Berkeley. He has also served as an independent expert witness in survey research meth- odology for California legal cases. Prior to co-founding True North Research in 2002, Dr. McLarney was the Director of Research at Godbe Research and a consultant for Quest, which provides organizational development training and consulting. Dr. McLarney holds a Ph.D. and M.A. in Government from Cornell University with an emphasis in survey methodology, sampling theory and public opinion, as well as a Bachelor's degree in Politics from the University of California, Santa Cruz. Principal Researcher Richard Sarles, M.S., will assist Dr. McLarney with managing data collec- tion, data processing, and graphics production for the report. Mr. Sarles will manage approximately 20% of the research tasks. As Principal Researcher and co-founder of True North Research, Mr. Sarles applies his Project Organization & StaffingTrue North Research, Inc. © 2016 8City of Lake Elsinore . . . . . . . . . . . . . . . . expertise in statistics, sampling, weighting and analysis to a wide variety of research studies. A veteran of over 700 research studies, public and private clients throughout the country rely on Mr. Sarles' research to help them meet their performance and revenue goals. Data Collection Manager Dameon Travis, B.A., is the Senior Account Manager of MWRC and will serve as the Data Collection Manager for this study. As he has for more than 200 of True North’s survey projects, Mr. Travis will manage the data collection process and interviewing team (telephone interviews) from MWRC’s offices in Pocatello, Idaho, and will report directly to Dr. McLar- ney and Mr. Sarles. Mr. Travis has extensive experience in managing all aspects of telephone interviewing, including sampling designs, CATI sta- tions and programming, interviewer training, daily reporting procedures, and quality control processes. In the past five years alone, Mr. Travis has managed more than 300 studies such as the one proposed here. With a B.A. in Communications and Economics and more than seven years expe- rience at MWRC working closely with Dr. McLarney and Mr. Sarles on sim- ilar studies, Mr. Travis will be an invaluable asset to the project. ReferencesTrue North Research, Inc. © 2016 9City of Lake Elsinore . . . . . . . . . . . . . . . . R EFERENCES The following are brief descriptions and reference information for similar community surveys conducted by True North’s team of Dr. McLarney (lead) and Mr. Sarles (support). We take great pride in going the extra mile for our clients, in providing exceptional client service, and in pro- viding research and deliverables that are the best in the industry—but please don’t just take our word for it. We encourage the City to contact our references and ask about the quality of work performed by True North. In addition to the references listed below, we can supply dozens more if the City is interested for our work on resident satisfaction surveys. City of Temecula Dr. McLarney and Mr. Sarles have personally designed and conducted eight research studies for the City of Temecula since 2000, including res- ident satisfaction surveys in 2000, 2007, 2011 and 2014; an extensive library needs assessment which included a telephone survey, one-on-one interviews with council members, staff, and community leaders, and sev- eral community workshops that helped in the successful application for Proposition 14 funding for a new library; a survey of voters and property owners to assess the feasibility of passing a revenue measure for parks and open space protection in 2002; a resident survey in 2004 related to cable television services; and a survey of Old Town visitors in 2008. Contact: Aaron Adams, City Manager 43200 Business Park Dr., Temecula CA 92590 Phone: 951.694.6419 Email: Aaron.Adams@cityoftemecula.org Projects: Resident Satisfaction Surveys (‘00, ‘07, ‘11, ‘14) Old Town Visitor Survey (‘08) Library Needs Assessment (‘01) Open Space and Parks Survey (‘02) Cable Television Survey (‘04) City of Lake Forest Dr. McLarney and Mr. Sarles designed and implemented both resident and business satisfaction surveys for the City of Lake Forest in 1998, 2000, 2002, 2004, 2006, 2008, 2010, 2012, and 2014. The focus of the surveys was on measuring the City’s performance in meeting the needs and priorities of its residents and the business community. Contact: Debra Rose, Director of Management Services 25550 Commercentre Dr., Lake Forest, CA 92630 Phone: 949.461.3414 Email: DRose@ci.lake-forest.ca.us Projects: Business Surveys (‘98, ‘00, ‘02, ‘04, ‘06, ‘08, ‘10, ‘12, ‘14) Resident Surveys (‘98, ‘00, ‘02, ‘04, ‘06, ‘08, ‘10, ‘12, ‘14) ReferencesTrue North Research, Inc. © 2016 10City of Lake Elsinore . . . . . . . . . . . . . . . . City of La Mesa True North has designed and conducted three community priorities and satisfaction surveys for the City of La Mesa, using a sample of 800 resi- dents to allow for analysis within geographic subregions of the City. Contact: Yvonne Garrett, Assistant City Manager 8130 Allison Avenue, La Mesa CA 91942 Phone: 619.667.1311 Email: ygarrett@ci.la-mesa.ca.us Projects: Resident Satisfaction Survey (‘06, ‘11, ‘13) City of San Marcos True North has assisted the City of San Marcos with three surveys in recent years, including a General Plan Update survey in 2010, Resident Satisfaction Survey in 2014, as well as an Economic Development Survey of local businesses in 2014. Contact: Lydia Romero, Deputy City Manager 1 Civic Center Dr., San Marcos CA 92069 Phone: 760.744.1050 ext. 3114 Email: LRomero@san-marcos.net Projects: Resident Planning Survey (‘10) Resident Satisfaction Survey (‘14) Economic Development Survey of Businesses (‘14) City of Thousand Oaks True North has designed and conducted three community priorities and resident satisfaction surveys for the City of Thousand Oaks, as well as survey of employees and a revenue measure feasibility study. Contact: Andrew Powers, Assistant City Manager 2100 Thousand Oaks Blvd., Thousand Oaks, CA 91362 Phone: 805.449.2148 Email: APowers@toaks.org Projects: Resident Satisfaction Survey (‘09, ‘13, ‘15) Employee Survey (‘15) Revenue Measure Survey (‘09) City of Westlake Village True North designed and conducted community visioning and satisfac- tion surveys for the City of Westlake Village in 2005, 2009, and 2015. The surveys, which were conducted via telephone, Internet and mail, addressed many key topics of interest and are being used to guide Coun- cil's future service, planning, and policy decisions. Contact: Audrey Brown, Assistant City Manager 31200 Oak Crest Dr., Westlake Village CA 91361 Phone: 818.706.1613 Email: Audrey@wlv.org Projects: Resident Survey (‘05, ‘09, ‘15) ReferencesTrue North Research, Inc. © 2016 11City of Lake Elsinore . . . . . . . . . . . . . . . . City of Brea True North has conducted four surveys for the City of Brea, including a survey to gauge community sentiment regarding the proposed extension of the Olinda Alpha Landfill, two resident satisfaction surveys, and a sur- vey to gauge the feasibility of a general sales tax measure on the Novem- ber 2008 Ballot. Contact: Cindie Ryan, Communications & Marketing Manager 1 Civic Center Circle, 2nd Floor, Brea CA 92821 Phone: 714.990.7600 Email: CINDIER@ci.brea.ca.us Projects: Resident Satisfaction Surveys (‘08, ‘11) Revenue Measure Survey (‘08) Landfill Extension Survey (‘06) City of Aliso Viejo True North designed and conducted the City of Aliso Viejo’s Community Satisfaction Survey in 2013, which had a special emphasis on prioritizing capital improvements projects the City is considering. True North also conducted a revenue measure feasibility survey for the City in 2002. Contact: Glenn Yasui, Director of Administrative Services 12 Journey, Suite 100, Aliso Viejo CA 92656 Phone: 949.425.2500 Email: gyasui@cityofalisoviejo.com Projects: Community Survey (‘02, ‘13) City of Manhattan Beach Dr. McLarney and Mr. Sarles have designed and conducted eight surveys for the City of Manhattan Beach since 1999, including four resident satis- faction surveys and three revenue measure feasibility surveys. The bian- nual surveys of 400 residents were conducted for customer satisfaction, strategic planning and performance measurement purposes. Contact: Bruce Moe, Director of Finance 1400 Highland Ave, Manhattan Beach, CA 90266 Phone: 310.802.5000 Email: bmoe@citymb.info Projects: Resident Satisfaction Surveys (‘00, ‘02, ‘04, ‘06) General Plan Survey (‘00) Revenue Measure Feasibility Surveys (‘99, ‘14, ‘16) ReferencesTrue North Research, Inc. © 2016 12City of Lake Elsinore . . . . . . . . . . . . . . . . City of San Jose Since 2006, Dr. McLarney has conducted annual customer satisfaction surveys for the City of San Jose. The surveys are conducted using a mixed-methodology which allows customers the flexibility to participate by telephone or online at a secure, password protected website hosted by True North. Contact: Page Benway, Senior Analyst 200 East Santa Clara St., San Jose CA 95113 Phone: 408.535.7887 Email: Page.Benway@sanjoseca.gov Projects: Customer Surveys (every year ‘06 to ‘16) City of San Carlos True North assisted the City of San Carlos in conducting Community Opinion Surveys in 2011, 2013 and 2016, which measured community needs and priorities as they relate to municipal services, gauged resi- dents’ opinions of the City’s performance in providing services, and gathered additional performance and policy-related information. True North also assisted San Carlos with voter research related to a potential revenue measure in 2008 and 2009. Contact: Jeff Maltbie, City Manager 600 Elm Street, San Carlos CA 94070 Phone: 650.802.4228 Email: JMaltbie@cityofsancarlos.org Projects: Citizen Survey (‘11, ‘13, ‘16) Voter Surveys (‘08, ‘09) City of Dana Point True North has conducted three surveys for the City of Dana Point, including a resident satisfaction survey to measure community priorities and satisfaction with respect to municipal services, a revenue measure feasibility survey, and a resident survey regarding the City’s vacation rental policies. Contact: Doug Chotkevys, City Manager 33282 Golden Lantern, Dana Point CA 92629 Phone: 949.248.3513 Email: DChotkevys@DanaPoint.org Projects: Resident Satisfaction Survey (‘09) Revenue Measure Survey (‘10) Vacation Rental Resident Survey (‘12) Survey Approach & MethodologyTrue North Research, Inc. © 2016 13City of Lake Elsinore . . . . . . . . . . . . . . . . S URVEY APPROACH & METHODOLOGY Trade Secret Notification: This section of the proposal contains information and a description of services and project approaches and procedures that are trade secrets as defined by Govern- ment Code Section 6254.7. This proposal contains a description of a unique process, approach and procedures that are known only to True North Research and which are not known or used by other competitors. True North requests that this proposal not be duplicated, distributed, dis- cussed, or shown to parties other than the individuals at the City of Lake Elsinore who are involved with the Project—in whole or in part—without the expressed written consent of True North Research, Inc. INNOVATIVE METHODS FOR A CHANGING ENVIRONMENT If the ultimate goal of this study is to enable the City of Lake Elsinore to make sound, strategic decisions based upon the data that is collected, it is critical that the data be collected according to rigorous method- ological standards. Indeed, although it may be easy to conduct a 'survey', it is quite challenging to design and implement a survey that A) produces valid3 and reliable4 measures of the opinions and behaviors of interest, and B) produces data that is representative5 of the population of indi- viduals that is the focus of a study. Only by satisfying each of these conditions will the proposed study produce data that is trustworthy and thus a sound basis for making decisions. Meeting the aforementioned conditions has become more difficult in recent years. Much has changed in terms of how the public receives information, the accessibility of residents through traditional recruiting methods, and their willingness to participate in community surveys. Whereas conventional RDD (random digit dial) telephone surveys worked well for surveying resi- dents in the past, they are no longer nearly as effective at producing reliable samples due to fac- tors such as call-screening and the increasing percentage of households that rely solely on cell phones. Nor do they provide the flexibility to respondents in terms of how they participate that is required to achieve acceptable participation rates. The bottom line is that in order to continue producing statistically reliable measures of their residents’ opinions, priorities and behaviors, the research methodologies that California cities employ for community surveys must adapt to these new challenges. The methodology that we recommend in this section is designed to do just that—adapt to these new challenges by employing innovative techniques that will ensure statistically reliable results. As discussed in greater detail later in this section, we propose to use a stratified sampling methodology that ensures a representative sample of Lake Elsinore households regardless of whether they use land lines or cell phones. We also recommend using a three-pronged recruit- ing strategy of mailed letters, email invitations, and telephone calls to recruit participation in the survey and a mixed-method data collection methodology that allows sampled respondents 3. Validity refers to the degree to which a question actually measures the underlying attitude or construct that it was designed to measure. For example, asking people whether they own “stock” to measure their financial savings behavior is sufficiently ambiguous such that it has low validity. Research has shown that people in farming communities are more likely to interpret “stock” to mean livestock—not financial stocks. 4. Reliability refers to the degree to which the observed results for a survey question are free of measurement errors and thus accurately measure the true value among the survey participants. 5. In this context, representative refers to the degree to which the results found among the survey participants match what would have been found if all adults in the City participated in the survey. A probability-based sample free from the many possible sources of selection bias is the key to obtaining representative results. Survey Approach & MethodologyTrue North Research, Inc. © 2016 14City of Lake Elsinore . . . . . . . . . . . . . . . . the flexibility of participating by telephone or online at a password protected web site. Designed, tested and refined by True North over the past six years, our recommended survey approach has proven to increase participation rates, reduce response bias, and increase the overall accuracy and reliability of citizen survey results when compared to the methods generally used by other polling firms. The proposed methodology is similar to that True North employed for the City of Temecula for its most recent Community Opinion Survey. The following sections describe the main elements of the proposed methodology, which includes several value-added methodological and project management features likely not found in the proposals submitted by other firms. COMMUNICATION At the outset of this study, True North recommends having a kick-off meeting with all individuals who will be actively involved in the study (or will be an important stakeholder) in attendance. The kick-off meeting is a great forum for talking about past research conducted and the City’s needs, expectations, and concerns for the upcoming study and deliver- ables. The kick-off meeting will also provide an opportunity for Dr. McLarney to clarify any ques- tions about the proposed methodology, ask questions of city staff, and identify changes to the methodology (as proposed) that may better serve the City’s interests. In short, the purpose of the kick-off meeting is to make sure that everyone is in agreement about the details of the study, identify areas that need further clarification, share materials, and establish a more detailed work plan for the project. After the kick-off meeting, Dr. McLarney will keep in regular contact with the Project Manager identified by the City through telephone, email, and in-person meetings. QUESTIONNAIRE Dr. McLarney will work closely with city staff and/or a subcommittee in designing and revising the questionnaire. The questionnaire design and revision process will begin at the kick-off meeting and will take about two weeks to complete. Dr. McLarney will take the lead in creating an initial draft after the kick-off meeting and receiving input that may be sub- mitted by staff. City staff will have the opportunity to then review and comment on the draft questionnaire. It usually works best if the City assigns a single person to gather the comments of staff and then discuss them with Dr. McLarney, who will make changes based on the comments. This process is repeated, as needed, until all parties are satisfied with the questionnaire. Throughout the design process, Dr. McLarney will work closely with Lake Elsinore to develop a questionnaire that covers the topics of interest to the City and avoids the many possible sources of systematic measurement error, including position-order effects, wording effects, response- category effects, scaling effects and priming. PRE-TEST & PROGRAMING Once the City has approved the draft survey instrument, it will be pre-tested internally and with at least 20 Lake Elsinore households prior to actually field- ing the survey to ensure that respondents can easily understand the point of each question, the response options match what respondents want to answer, the skip patterns and CATI logic are correct, etc. After any final adjustments are made in consultation with the City, the instrument will be programmed for both telephone (CATI) and online versions. Survey Approach & MethodologyTrue North Research, Inc. © 2016 15City of Lake Elsinore . . . . . . . . . . . . . . . . SAMPLING The sampling design for this study is in many respects the most important stage of the survey research project. Only through carefully designing the sample to meet the many theoretical and logistical challenges associated with conducting community surveys will the results of the interviews be representative of the estimated 37,980 adult residents in the City.6 As noted previously, telephone-based sampling techniques (such as RDD) that in past years worked well for generating representative samples of residents are no longer effective on their own. Note only do they fail to account for the growing number of households—especially younger households—that have given up their land lines in favor of cell phones, the prevalence of caller ID and other similar technologies has led to a substantial rise in call screening behav- iors. In combination, these factors create a situation in which a significant percentage of house- holds are simply unreachable if one relies solely on telephone-based sampling and recruiting techniques. Rather than choose phone numbers at random, our recommended solution is to use a compre- hensive database of Lake Elsinore households (listed by physical address/location) and stratified sampling methods to select a random sample of Lake Elsinore households. This will ensure that all households in Lake Elsinore have an equal probability of being selected for the survey—not just those that have land lines or published cell phone numbers. It also enables True North to recruit households using an effective combination of mailed invitation letters, email and tele- phone calls, as discussed below (see Three-Pronged Recruiting on page 16). SAMPLE SIZE By using the aforementioned stratified random sampling design with adjust- ments as needed, True North will ensure that the sample is representative of the City’s adult population. The results of the sample can then be used to estimate the opinions of all adults in the City. Because not all residents will participate in the survey, however, the results will have what is known as a statistical margin of error due to sampling. The margin of error refers to the difference between what is found from the sample of respondents and what would have been found had all residents been surveyed. This is where sample size becomes important. The larger the sample, all other things being equal, the smaller the margin of error and the more confidence one can have that the survey results accurately reflect all adults in the City. To provide the City with a sense for how the statis- tical margin of error is related to sample size, Figure 2 plots the relationship between sample size and the maximum7 statistical margin of error for the proposed survey. As illustrated in the figure, beyond a sample size of 400 the statistical margins of error begin to flatten-out such that incremental units of additional sample result in smaller and smaller reductions to the statistical margins of error. For this reason, True North generally recommends a sample of 400 residents as it best balances the need for reliability with the costs associated with data collection. A sam- ple of 400 completed interviews will have a maximum margin of error at the 95% confidence level of 4.88% (plus or minus). 6. This figure is based on Census 2010 data and California Department of Finance projections for 2015. 7. The margin of error will vary depending on the distribution of responses to the question. The maximum margin of error occurs when the results are evenly split (50-50) between two response options. Survey Approach & MethodologyTrue North Research, Inc. © 2016 16City of Lake Elsinore . . . . . . . . . . . . . . . . FIGURE 2 MAXIMUM STATISTICAL MARGIN OF ERROR DUE TO SAMPLING IN CITY OF LAKE ELSINORE Although 400 is the minimum guaranteed sample size for the study, it is also likely that we will deliver a larger sample size. One of the advantages of using our proposed recruiting strategy (see Three-Pronged Recruiting on page 16) and offering respondents the option to participate online as well as by telephone (see Mixed-Method Data Collection on page 17) is that if the response to the mailed invitations is greater than our conservative estimate, the resulting sam- ple size will be larger than 400 completed interviews. The additional completed surveys will be provided to the City at no additional cost. THREE-PRONGED RECRUITING Once households have been selected for the sample, each will be mailed a personalized letter and (when available) an email that invites an adult mem- ber of the household to participate in the survey either online or via telephone. The letters will include a unique password for each household to ensure that only Lake Elsinore residents are able to take the survey and limit the number of completed interviews per household to one.8 To maximize the response rate and resident candor, we recommend that the envelope and letter be produced on city stationary or using the city logo, but clearly state that the survey is being con- ducted by an independent research firm to ensure the objectivity and anonymity of the study.9 Households that do not respond to the initial mailed invitation or email by participating in the survey will receive follow-up telephone calls for the purposes of soliciting their participation and conducing the interviews. Using this three-pronged recruiting method (mailed letters, emailed invitations, and telephone calls) allows us to reach a substantially higher percentage of residents when compared to using only one recruiting method. 8. Once a password has been used, it can’t be used again. However, if a respondent should be interrupted before completing the survey, they can use the password to access the survey site again and it will pick up where they left off. 9. We will provide recommended language for the invitation letter to the City for review and approval. Sample of 400 Adult s ± 4.88% 0% 2% 4% 6% 8% 10% 12% 14% 0 100 200 300 400 500 600 700 800 900 1000 Sample Size (Number of Respondents)Margin of Error Survey Approach & MethodologyTrue North Research, Inc. © 2016 17City of Lake Elsinore . . . . . . . . . . . . . . . . MIXED-METHOD DATA COLLECTION The reliability of survey results depends, in part, on having a high participation rate. One of the most productive ways of encouraging participa- tion (and minimizing response bias) in a survey is to make it as convenient as possible for an individual to participate. Although some individuals find it most convenient to participate via telephone, others may have responsibilities and schedules that are simply not conducive to spending up to 20 minutes on the phone at a particular time. For this reason, we recommend a mixed-method design that will allow respondents maximum flexibility in choosing when and how they participate in the study. The mixed-method approach involves allowing respondents the option to participate in the sur- vey by telephone or online through a secure, password protected web-based application designed and hosted by True North. Respondents can access the online survey from a computer, tablet or smart phone, and the screen will scale accordingly. By providing respondents with flex- ibility as to when and how they participate in the survey, a mixed-method approach (especially in combination with a three-pronged recruiting method) will deliver higher participation rates, min- imize response bias, and ultimately deliver more reliable results than using telephone-only based data collection methodologies. Telephone Interviewing Quality telephone interviewing will be an important element of the proposed survey. Using a ‘bargain’ phone facility is rarely a bargain, as the integrity of the data and the accuracy of the results depends on the knowledge, experience and professionalism of the interviewers and supervisors. In the end, it is critical that the research team and the City can trust the data that were collected. For this reason, True North will team with MWRC, a profes- sional interviewing company that has partnered with Dr. McLarney and Mr. Sarles on more than 400 community survey projects in the past and has a proven track-record of professional inter- viewers, quality data collection, and competitive pricing. Located in a university town with few alternative employment opportunities, MWRC interviewers tend to have higher education levels and are more motivated and professional than most facilities. Interviewing for a resident survey is typically conducted in the evenings (5:30PM to 9:00PM PST) during weekdays, and 10:00AM to 6:00PM on weekends. True North works closely with MWRC to manage quality control procedures during data collection, which will include a training session with interviewers and supervisors to ensure that telephone interviewers understand the purpose of all of the questions in the survey, thoroughly testing the CATI and web programs prior to fielding, active and remote supervisor monitoring of interviews, quality control checks on a ran- dom selection of interviews to identify any discrepancies that may occur (and need adjusting), and daily monitoring of the sample characteristics to ensure that data collection is adhering to the sampling plan (for more details on our quality-control processes, see Quality Control on page 19). DATA PROCESSING Once the interviewing is complete, True North will process the data. This consists of checking the data for errors or inconsistencies, coding and recoding responses, categorizing open-end responses, and preparing frequency analyses and cross-tabulations. Survey Approach & MethodologyTrue North Research, Inc. © 2016 18City of Lake Elsinore . . . . . . . . . . . . . . . . ANALYSIS & TOPLINES Once the data processing is complete, True North will analyze the results and prepare an initial topline report for the City’s review. The topline report consists of the aggregate percentage responses for each question asked in the survey. SURVEY REPORT The survey report prepared by True North Research for the City will include a thorough question-by-question analysis of the survey findings, an executive summary of the key findings and recommendations, a detailed discussion of the methodology used to complete the study, a copy of the questionnaire, full-color graphics of the overall results and those of among specific subgroups (e.g., by length of residence, age, presence of children in the home, gender, etc.), and a complete set of crosstabulations for reference that display the results broken down by various subgroups of respondents. True North believes that the quality of the final report and presentation reflects directly upon the quality of the study as a whole as well as True North Research, so Dr. McLarney and Mr. Sarles take great care in developing a report which is thorough (200+ pages cover to cover)10, insightful, free of mistakes, and presents the infor- mation in an eye-catching, full-color display. True North's reports are, without question, the most thorough and insightful in the industry. We don’t just report the results—we put the findings into context and explain their meaning based upon related questions in the survey, the City’s goals, relevant secondary information, the results of prior surveys, as well as our experience working with California municipalities on hun- dreds of similar studies. BENCHMARKING & STATISTICAL SIGNIFICANCE TESTING Because the proposed survey may include some questions that were asked in previous surveys administered to Lake Elsinore residents including potential key performance measures for the City, there is an obvious interest in benchmarking against prior surveys and analyzing how the responses to these ques- tions may have changed over time. It is important to recognize, however, that survey results can change over time for a number of reasons—only some of which are meaningful from a planning or performance measurement perspective. The goal is thus to isolate those changes that likely reflect actual changes in public opinion from those that are due to methodological artifacts such as measurement error, sample design effects, and chance associated with independently selected random samples. The former two methodological sources must be addressed in the design stages of the study to ensure an apples-to-apples comparison between the respective studies. The latter source—chance associated with independently selected random samples—is addressed through conducting statistical significance tests. One advantage in partnering with True North is that we recognize that the appropriate statistical tests vary depending on the parameter being estimated (mean, total or percentage), the relation- ship of the respective sampling procedures (related or independent), as well as the distribution of the data on the selected question. As part of the report, True North will conduct the appropri- ate tests of statistical significance for those questions and performance measures that can be meaningfully compared across the City’s resident surveys, if applicable. 10.For those readers who don’t have the time or the interest to review the entire report, we present a concise executive summary of the key findings and conclusions. Survey Approach & MethodologyTrue North Research, Inc. © 2016 19City of Lake Elsinore . . . . . . . . . . . . . . . . PRESENTATION & CONSULTING Once the final report has been completed, True North will prepare a PowerPoint presentation of the survey results and deliver the presentation to City staff and Council, as desired. Moreover, after the study is complete, we will continue to be a good partner for the City. As issues or questions arise for which the survey may provide some guidance, we will be happy to consult with and advise the City at no additional charge. TIME LINE True North will work with the City of Lake Elsinore to establish a time line that meets the City’s needs. Using the May 10, 2016 start date referenced in the RFQ, the schedule in Figure 3 shows each of the major tasks needed to complete the survey, as well as the proposed start date and duration for each task. Dr. McLarney and Mr. Sarles have the availability and resources needed to meet this time line, as well as the flexibility to adjust this schedule, as needed, to best meet the City’s needs. For the reader’s convenience, we have color-coded the schedule shown in Figure 3. The green bars denote periods where the City will be reviewing materials and deliverables. All other tasks can be managed by True North without need for City staff or resources. FIGURE 3 PROPOSED TIME LINE QUALITY CONTROL Although we have commented on various aspects of our quality con- trol procedures throughout this proposal, in this section we elaborate on the quality control pro- cedures we employ to ensure that the City receives accurate, reliable survey results, analyses and deliverables. True North ensures the integrity of its data through a series of steps that begins with the ques- tionnaire design process. As noted previously, we work closely with our clients during the ques- tionnaire design and revision process to ensure that the questionnaire covers the topics they are interested in while avoiding the many possible sources of measurement error, including posi- tion-order effects, wording effects, response-category effects, scaling effects and priming. The questionnaire will be designed and thoroughly reviewed internally by Dr. McLarney and Mr. Sar- les before being shared with city staff. It will then be jointly reviewed by Dr. McLarney and city staff, content revisions identified, and a new paper/electronic draft will be created and proofed internally again by Dr. McLarney prior to being distributed to city staff for review. This process is Activity Name Duration (Days)Start Date Finish Date 15222951219263101724317 May 16 Jun 16 Jul 16 Aug 16 15222951219263101724317 Design Survey Kickoff 1.00 5/16/16 5/16/16 Questionnaire Draft 10.00 5/16/16 5/27/16 City Review Questionnaire 5.00 5/30/16 6/3/16 Revise & Finalize Questionnaire 3.00 6/6/16 6/8/16 CATI & Web Program 5.00 6/9/16 6/15/16 Sample Design 9.00 6/3/16 6/15/16 Administer Survey Produce & Send Mail & Email Invitations 4.00 6/16/16 6/21/16 Conduct Interviews 10.00 6/22/16 7/5/16 Data Processing 4.00 7/6/16 7/11/16 Analysis, Reporting & Presentation Topline Summary 1.00 7/13/16 7/13/16 Draft Report 9.00 7/14/16 7/26/16 City Review Report 3.00 7/28/16 8/1/16 Finalize Report 3.00 8/3/16 8/5/16 Develop Powerpoint Summary 3.00 8/5/16 8/9/16 Presentation of Findings 1.00 8/11/16 8/11/16 Survey Approach & MethodologyTrue North Research, Inc. © 2016 20City of Lake Elsinore . . . . . . . . . . . . . . . . repeated, as needed, until all parties are satisfied with the questionnaire. To avoid version-con- trol issues, we do not share the Master File but rather have city staff review and comment on a pdf version. This ensures that small, unintentional changes or mistakes are not made during the review process that find their way into the final questionnaire. Once the City has approved the questionnaire, it will be programmed by True North into a pro- prietary CATI (Computer Assisted Telephone Interviewing) system that guides interviewing staff when conducting phone interviews, and a customized web interface to allow online participa- tion. Both systems automatically navigate skip patterns in the survey, randomize question items as desired, and prevent most types of keypunching mistakes made by an interviewer or cus- tomer. And in studies that allow respondents the option of participating by telephone or online, the CATI and online systems are linked so that the responses are recorded to the same data file. This feature eliminates errors that may arise when merging data from separate files with sepa- rate layouts. Once developed, the CATI and web programs are thoroughly tested by Mr. Sarles and Mr. Travis to ensure they are free of errors, limit input to valid data ranges for each question, verify certain responses with the interviewer or the respondent in real-time and, when appropriate, accurately cross reference data gathered earlier in the survey or previously collected from the customer data on file. This hands-on testing process is conducted by repeatedly working through the CATI and web interview, entering combinations of data, testing skip patterns, etc. Once hands-on testing of the CATI and online systems is complete, True North employs separate testing software (also proprietary) that generates a mock data file based on acceptable types and ranges of data as permitted by the CATI and online programs. This resulting test data, out- putted in SPSS format, allows True North to review and test a data file that matches the eventual final study data file with respect to its column layout, data structure, data labels, variable names, and valid data ranges. In studies where there may be an interest in tracking the results of prior surveys, it is at this point of the testing where special consideration is made to accommodate the particulars of those past data files so the layouts are consistent, the variable and label names identical, and the data types and ranges an exact match. This test file can also be provided to city staff for review and possible revisions, if desired. Once we are fully satisfied that the CATI and web programs are perfect, we will also provide city staff with the opportunity to review the programs online (if desired). This involves giving staff a link to access to our CATI and online web system in test mode. City staff can then review the pro- grams in a live environment, answering questions and working through the interview. In our experience, this stage of the review rarely uncovers programming errors—but it occasionally produces additional content changes as our clients have an opportunity to experience the inter- view first-hand and often come away with an idea or two that we incorporate into the question- naire using the same process described above. Once True North and city staff are satisfied with the CATI and the data file structure, the study will be pre-tested with at least 20 residents. The objective of the pre-test is to ensure that respondents can easily understand the point of each question, the response options match what respondents want to answer, the skip patterns and CATI logic are correct, etc. Again, we will review the results of the pre-test with city staff and make recommended changes, if warranted. Survey Approach & MethodologyTrue North Research, Inc. © 2016 21City of Lake Elsinore . . . . . . . . . . . . . . . . We know good interviewing starts with good interviewers, which is why we team with MWRC. Each interviewer is carefully screened and trained before they start dialing. Our training process provides us with an articulate, skilled and diverse pool of employees who are capable of over- coming respondent objections, converting refusals, and handling complex questionnaires. Initially, we screen all prospective interviewers by administering written tests to identify skills such as knowledge of current affairs, grammar, spelling and work habits. We then train our inter- viewers in a formal classroom environment. The training covers a basic orientation to the indus- try, specific interviewing mechanics (skip patterns, probing, etc.), and telephone etiquette. Following the classroom training, each interviewer participates in practice interviews, playing the role of both the interviewer and the respondent. Supervisors tape these interviews and the roles of both the interviewer and the respondent are evaluated during a re-training and diagnostic ses- sion. Separate exercises in probing are also conducted. After completing their training, the inter- viewers enter a two-week probationary period where they conduct live interviews under the close supervision of training personnel. Supervisor personnel continue to monitor each interviewer’s progress on a regular basis. If an interviewer fails to meet the interviewing standards, he or she is removed from the project and re-trained. Interviewers are thoroughly briefed before beginning a new study. The briefing is designed to familiarize the interviewers with the study objectives, the questionnaire and CATI program, the sample and the respondent type, as well as any special considerations or instructions related to the project (e.g., alert supervisor if respondent insists on knowing the sponsor of a survey at the outset of the interview, which can affect the integrity of the interview). Briefings cover the follow- ing points: • Background of the study • Any unique problems that may arise • The importance of precisely following the sample design • A question-by-question review of the questionnaire and appropriate ways to complete cer- tain questions • Working with the questionnaire and CATI system • Handling the specific nuances of the questionnaire (including asking questions to obtain potentially sensitive information) • Following prescribed, specific survey protocols and procedures • Working with difficult to handle and non-English speaking respondents While reviewing the specifics of the study, the supervisor encourages the interviewers to ask per- tinent questions. Our clients are also welcome to participate in the briefings remotely or in per- son. A high supervisor-to-interviewer ratio is maintained in all of our interviewing centers to ensure that the lessons learned in training are applied correctly in the field. Supervisors unobtrusively monitor at least 15 percent of all live interviews via telephone and computer links. Interviewers are evaluated on several areas, including the following: Survey Approach & MethodologyTrue North Research, Inc. © 2016 22City of Lake Elsinore . . . . . . . . . . . . . . . . • Reading verbatim • Correct pronunciation • Appropriate speed • Avoiding bias • Telephone etiquette • Accurate and thorough probing • Controlling the conversation • Correct and complete data entry Managing the sample in the field is made easy using MWRC’s sophisticated CATI and online sys- tems. All respondent records are flagged with their appropriate strata, and quotas are pro- grammed into the system for strata as well as relevant demographic characteristics. Once the number of interviews in a particular subgroup has been met, that group will be automatically “closed” and the dialing/interviewing will continue for strata and subgroups that have yet to be filled. The sample is also reviewed nightly as data collection proceeds to ensure that is within certain tolerances and meets the desired demographic distributions. At True North, we take additional steps to ensure that the sample is representative beyond sim- ply managing the sample on strata and quotas. For resident surveys, we recommend conducting interviews only during certain time windows (5:30 PM to 9:00 PM midweek; 10:00 AM to 5:00 PM weekends) when a balanced mix of residents are most likely to be home. If we are unable to reach a resident on our initial attempt, callbacks are automatically scheduled at different times of the day and on subsequent days to ensure that we maximize our chances of reaching the respondent. Only after eight attempts are made to reach the respondent (twice the four-call stan- dard used by most polling firms) will we consider that resident unreachable and replace them with another resident with a similar profile. Once the interviews are collected, we process the data and once again review the data file to confirm that it is a clean file, free of mistakes, and has the proper variable names, labels and assignment. Additional variables may be added to the file at this point as we code open-ended responses, create derived variables, and merge additional secondary information on the file for analysis. For this reason, we thoroughly test the data file again at the conclusion of data collec- tion and pay special attention to any new variables that are added to the file. Before submitting the final data file to the City, we conduct a variable-by-variable analysis of data frequencies and distributions, and test the logical relationships between variables, to ensure that the data makes sense and there are no odd patterns. True North’s management model is also a key factor in maintaining quality control throughout a project—and especially when producing analyses, reports and recommendations (see Project Organization & Staffing on page 6). Whereas most firms manage projects by division of labor and delegating most of the work to low-level staff, at True North we firmly believe that the best way to conduct your study is to have the people in our organization with the most expertise and experience roll-up their sleeves and actually do the work. Rather than rely on division of labor, our entire management model is designed to integrate project management, design, implemen- tation, analysis, and reporting. A key advantage of selecting True North is that who you see is Survey Approach & MethodologyTrue North Research, Inc. © 2016 23City of Lake Elsinore . . . . . . . . . . . . . . . . who you get. Dr. McLarney and Mr. Sarles will not just advise on your project or be a “manager”— they will personally handle all key research tasks. True North’s management approach has obvious advantages when it comes to developing the final deliverables for a project, including the report and presentation. Because Dr. McLarney and Mr. Sarles collaborate throughout a project on all aspects of the study, they have a first-hand understanding of the client’s research objectives, the data, and the study findings, and are thus in a great position to develop a report that A) captures the motivation for the study, B) avoids unnecessary clutter and instead focuses on study findings that speak to the important research issues/questions of interest, and C) provides insightful strategic recommendations and action- able takeaways for the client. Our process for developing the report and presentation for the City will begin with a conversa- tion with city staff to identify any special needs or desires, as well as identify the appropriate audiences for the report. Dr. McLarney and Mr. Sarles will then jointly discuss the general layout and content. Graphics will be produced and reviewed by Mr. Sarles for inclusion in the report based on direction from Dr. McLarney. Once the graphics are developed, they will be reviewed for accuracy based on comparison with SPSS output and other related documents including topline results and crosstabs. Once we are satisfied that the numbers are all correct, the graph- ics will then be formatted to ensure consistency in layout, look and feel, narrative will be added to the report, and additional animation if desired for the Powerpoint. Dr. McLarney will take the lead in developing the narrative, and both Mr. Sarles and Dr. McLarney will proof the report and Powerpoint separately before providing the files to the City for review and discussion. After fol- low-up discussions with the City, any changes that are requested will be made in accordance with the same process identified above until the report and Powerpoint are given the final stamp of approval by the City. Compensation/Payment ScheduleTrue North Research, Inc. © 2016 24City of Lake Elsinore . . . . . . . . . . . . . . . . C OMPENSATION/PAYMENT SCHEDULE The following table presents True North’s fixed-fee cost estimate to design and conduct the sur- vey as described in this proposal. The costs shown are inclusive—there will be no additional charges for travel, incidentals, meetings, post-project advising, or any other matter.11 If selected as the City’s research partner for this study, our payment terms are simple: we request full pay- ment at the conclusion of the study. No interim or up-front payments are needed. When comparing our costs to those of other bidders, please note that our costs include value- added services that may not be provided by other firms. Our value-added services include: •Using a stratified sampling methodology that ensures a representative sample of house- holds regardless of whether they use land lines or cell phones. •Using a three-pronged recruiting strategy of mailed letters, email invitations, and tele- phone calls to recruit participation in the survey and a mixed-method data collection meth- odology that allows sampled respondents the flexibility of participating by telephone or online at a password protected web site. This approach will increase participation rates, reduce response bias, and provide more reliable results than possible if one uses telephone- only recruiting and data collection. •Providing comparison benchmarks on key performance measures from a selection of the hundreds of resident satisfaction surveys that True North has conducted for California cit- ies, if desired by the City. •Conducting statistical significance testing to identify those changes in residents’ opinions between prior Lake Elsinore resident surveys (if applicable) and the proposed 2016 survey that are statistically significant versus those that are likely due to chance associated with independent random sampling. • Providing the most thorough and insightful report in the industry. We don’t just report the results—we put the findings into context and explain their meaning based upon related questions in the survey, the City’s goals, relevant secondary information, the results of prior surveys, as well as our experience working with California municipalities on hundreds of similar resident studies. Although some of these services are likely not included in the proposals submitted by other bid- ders (and we could similarly follow suit and offer a lower cost), our experience is that these addi- tional services are necessary to ensure that the City receives statistically reliable survey results and the maximum return for its investment in this study. FIXED-FEE COST OPTIONS The cost for a survey is based in part on the sample size and the length of time it takes to administer the interview. Because the questionnaire has yet to be developed, we present 15 and 20 minute length options in Table 1 on the next page and their associated costs for the City’s consideration assuming a sample size of at least 400 interviews. The actual cost of the survey will be pro-rated accordingly if the length falls somewhere in between 15 and 20 minutes. 11.These costs assume two in-person meetings/presentations and unlimited conference calls throughout the study, which is standard for a community survey project. If additional in-person meeting are desired by the City, they will be billed at $175 per hour including travel time. Compensation/Payment ScheduleTrue North Research, Inc. © 2016 25City of Lake Elsinore . . . . . . . . . . . . . . . . Please also note that we have been conservative in estimating the response rate to the initial mailed invitation letters. If the response rate is higher than our conservative estimate, we will provide additional completed interviews beyond 400 at no additional cost to the City. The costs shown in Table 1 also include all expenses associated with producing and mailing the invitation letters—a value-added feature likely not included in other firms proposals. TABLE 1 COSTS FOR 15 AND 20 MINUTE SURVEY OF AT LEAST 400 RESIDENTS 15 Min. 20 Min. Sample 800$ 800$ CATI Program/Host 1,250$ 1,250$ Invitation Letters 2,250$ 2,250$ Data Collection 10,000$ 12,000$ Data Processing 2,000$ 2,000$ Research Fee 7,500$ 7,500$ Project Management & Meetings 1,750$ 1,750$ Miscellaneous Expenses 250$ 250$ TOTAL 25,800$ 27,800$ Task Survey Length InsuranceTrue North Research, Inc. © 2016 26City of Lake Elsinore . . . . . . . . . . . . . . . . I NSURANCE True North carries the insurance coverages requested in the RFQ in amounts that meet or exceed the requested limits. Standard Agreement & TermTrue North Research, Inc. © 2016 27City of Lake Elsinore . . . . . . . . . . . . . . . . S TANDARD AGREEMENT & TERM CONTRACT True North has reviewed the City of Lake Elsinore’s Standard Agreement for Professional Services and does not take exception to any aspect of the agreement. If selected as the City’s research partner, we are prepared to sign the agreement without modifications. PROPOSAL TERM This proposal is binding for a period of 120 days from the date of sub- mittal to the City.