Organizational Network Analysis: MEASURE Evaluation’s Experience 2010-2014

Organizational Network Analysis: MEASURE Evaluation’s Experience


Executive Summary
Health care organizations usually provide a narrow range of services that address a specific health need or the focus of a particular funding source.For many organizations, it is too costly or complex to provide all the health services necessary to address the full range of their clients' needs.However, when an organization can see that it is part of a network of organizations providing the full range of services, and when it learns to collaborate within that network, it can gain access to and make use of information, skill sets, commodities, materials, and resources better than it can as a solitary organization.From a client's perspective, the network is better able to meet his or her health needs.Well-coordinated networks can result in more synergies, less duplication, saved expenses, more thorough provision of services, and better health outcomes.
Research on organizational networks in health systems was initially conducted in developed countries.At MEASURE Evaluation, we sought to extend the application by creating a practical organizational network analysis (ONA) approach for resource-poor settings.The purpose of this document is to share that approach, the related tools, and insights from implementing ONA in three countries: Ethiopia, Thailand, and Malawi.MEASURE Evaluation implemented the ONA approach in these countries from 2009 to 2014 to better understand how to improve service coordination in a community, mainly through client referral, and thereby improve client access to needed services.In Ethiopia, we assessed linkages between home and community based care and family planning networks.In Malawi, we applied the approach to links between USAID-funded NACS (Nutrition Assessment Counseling and Support) clinical sites, and community services for economic strengthening, livelihoods, and food security.In Thailand, the organizations studied provided services for men-who-have-sex-with-men (MSM) and transgender people.To examine the effects on clients served by these networks, we also collected data on their service experiences.In Ethiopia, we conducted an intervention to strengthen the network and followed it with a second ONA to evaluate changes in the network and client service experiences.
The information yielded by the ONA that proved to be most useful to the organizations included: (1) the identification of organizations in the network and a resulting directory of organizations and services offered; (2) the visualization of client referrals between the organizations that could guide new linkages; and (3) clients' reports of their health needs and constraints to obtaining referrals.The dissemination meetings helped build relationships between the organizations and establish a common agenda among them.In Ethiopia, subsequent meetings served as a platform for strengthening the network and improving clients' service experiences.
vi In these three applications, we found that ONA can result in a network perspective among organizations, guide more efficient and effective collaboration among them, and improve the client care experience.In the future, we will apply ONA to additional networks, develop a rapid ONA approach, develop measures of network effectiveness, and further demonstrate links between strengthened networks and improved client outcomes.
The audience for this report includes a broad range of professionals seeking applied, practical, and sustainable solutions to improve coordination and collaboration among service delivery organizations.

Introduction
Organizations are generally structured to address specific health needs that either represent a care niche and/or are driven by the funding sources.Moreover, for many organizations, it is too costly or complex to provide all the health services necessary to address the needs of their clients.For example, people living with HIV have many needs.They need to access HIV testing to find out their status.Once their status is known, they need access to care and treatment services.They will also need counseling to protect their partners from also acquiring HIV.They may need prevention, diagnosis, and treatment for other diseases such as tuberculosis.Health and social needs will depend on clients' sex, types of sexual relationships, age, and life stage, among other factors.
Organizations that are linked together in network can be more effective to provide health services and address health problems.A group of organizations working together will have greater access to information, skill sets, commodities, materials, and resources than a single organization.Organizations that work together may be better able to impact health status, particularly if they can effectively refer clients to needed services.Networks that are made up of local organizations may be more sustainable because they are connecting local actors and avoiding dependency on a central donor or international non-governmental organization (NGO) (Bloom, Reeves, Sunseri & Nyahn-Jones, 2008).Networks may facilitate learning and innovation through sharing ideas and adapting approaches.Networks bring together voices that can be a platform for advocacy and change.For these reasons, MEASURE Evaluation has developed an organizational network analysis (ONA) protocol and data collection tool and has been testing the approach in different countries.A sample of the organizational data collection tool is located in the appendix.

Purpose and Audience
The purpose of this document is to share the tools, approaches, experiences, and lessons learned implementing and organizational network analysis approach in three countries, Ethiopia, Thailand, and Malawi.From the synthesis of those experiences we draw insight into how ONA may be best applied moving forward and suggest innovations and adaptations of ONA for future testing.
The audience for this report includes a broad range of professionals seeking applied, practical, and sustainable solutions to improve coordination and collaboration among service delivery organization and ultimately to improve health outcomes.

ONA Background
Research related to organizational networks has mainly been in developed country health systems, although it is acknowledged the potential that strengthened networks can have on improved health systems (Bloom et al., 2008;Blanchet & James, 2011).At MEASURE Evaluation, we sought to fill a gap by testing an ONA approach that would be practical, applied, and replicable in developing country settings.The ONA approach as we have applied it was largely informed by work in the domestic mental health field and treatment of sexually transmitted infections (STIs) (see, for example, work by Morrissey and colleagues [Morrissey, Rdgely, Goldman & Bartko, 1994]; Provan and colleagues [Provan & Milward, 1995;and Provan, Veazie, Staten & Teufel-Shone, 2005]; and Thomas and colleagues [Thomas, Isler, Carter & Torrone, 2007;Thomas, Torrone, Levandowski & Isler, 2008]).Those experiences have led to the development of tools that describe the types of linkages or relationships between organizations (e.g., shared resources, clients, information, etc.), the frequency of those linkages (e.g., regular, occasional), quality of the relationship (e.g., informal, formal), confirmation of linkages (e.g., reported relationships confirmed by partner organization or memorandums of understanding [MOUs]).
Organizational network analysis is based in social network analysis, but the difference is that the actors in the network are organizations instead of individuals.Similar to social network analysis, one of the advantages of ONA is the power to visualize relationships between organizations.Network analysis metrics are also useful, including network density (the number of links as a proportion of all possible), centrality (the degree to which a high proportion of links are with a single or few organizations), reciprocity (the proportion of mutual ties, e.g., A refers clients to B and B refers clients to A) (Wasserman & Faust, 1994), in-degree (e.g., referrals received) and out-degree connections (e.g., referrals sent) (Freeman, 1979), and the tendency for homophilic or heterophilic connections (with similar or dissimilar organizations) (Newman, 2003).
Descriptive information can be used to understand how similar or dissimilar organizations are on certain characteristics such as catchment area size, services offered, number of clients, types and number of clinical and non-clinical staff, types of linkages with other organizations, perceptions about networking with other organizations.This information can also be useful to identify gaps and redundancies in services.
ONA is also consistent with systems approaches because it considers the connections among different components, the implications of these connections, and the active engagement with stakeholder on the ground to facilitate change (Leischow & Milstein, 2006)

Process of Conducting an ONA
Health care networks consist of government agencies and nongovernmental organizations (NGOs) providing a range of essential health services to meet a common health goal.However, members of networks are often not aware that they belong to a network or they only know part of the network.This limits the flow of information, resources and clients throughout the network, and generally limits the ability of providers to meet the needs of their clients.An organizational network analysis is a systematic and scientific approach that can help organizations see the network and make better use of the resources within it to the benefit of their clients.
The following is a summary of tasks to carrying out an ONA: 1. Engage stakeholders.Throughout the planning, implementation, analysis, and communication, work with those who have an interest in the performance of the network.This is an ongoing process.See the section "Ensure findings are actionable," below.
2. Define the network: Apply specific criteria to define the network.The network is essentially a cluster of organizations -government entities, NGOs, community-based organizations (CBOs) and others providing the services of interest within the geographic area of interest.
3. Enumerate the network members: Generate an initial list of all network members in the area of interest by identifying community gatekeepers and interviewing them to build a list of all service providers within the network (an "enumeration period").

Interview clients: Collect information directly from clients through client interviews to
understand their perceived quality of care, their met or unmet needs, and some of their behaviors (e.g., adherence to antiretroviral therapy or ART).
6. Map the organization locations: Collect geographic coordinates of organizational network members as well as important landmarks, points of interest, and transportation hubs.This information will be useful to visualize the network members, their location in relation to others, and can help illuminate any access barriers.
7. Analyze the data: Analyze data collected using qualitative and quantitative methods.Using a network analysis software (such as UCINET or Gephi) analyze relevant data by utilizing measures of centrality, density, and reciprocity of network ties.
8. Share the findings with the organizations: Conduct a dissemination meeting so the participating organizations have an opportunity to see the network they are a part of, validate the quantitative data, discuss how to improve the network and linkages, and understand how improving the network will benefit them.9. Share the findings with other stakeholders: Conduct a stakeholder meeting to include all stakeholders within the geographic area of interest who have a stake in the outcomes of the study.This meeting can help to make sure that the results and outcomes of the study and dissemination meeting can be used to strengthen the network.
Like any research method, the decision to use ONA is based on it being the most appropriate method to achieve the study goal and objective and answer the research questions.Similarly, the actual tasks carried out in an ONA will been to be adapted to the specific questions and context.

Ensuring findings are actionable
To insure that findings from the ONA are used to make decisions, communication about the activity starts from Day 1 and continues throughout the course of the activity.Central to ensuring that the results are used is to involve stakeholders throughout the planning, implementation, analysis and communication process.First, identify who the stakeholders are who need the information and results.A useful tool in the process is the Stakeholder Engagement Tool from MEASURE Evaluation.
Then, determine what information is the most salient to them and how to communicate that information to them.Different stakeholders have different needs for information because they have different roles in decision making.
Answers to the following questions can help determine each of your audiences and how to tailor communication with them (Foreit, Moreland & LaFond, 2006): • Who will benefit from the ONA results, and what questions are they seeking to answer?
• Who has influence and resources that can support the ONA process and/or the recommendations emerging from the results?
• Who will be directly or indirectly affected by the outcome of ONA?
• Who will support the ONA process and/or the emerging recommendations?Who will oppose it?Why?
• How can we best leverage their insights or assuage their objections?
When stakeholders are involved throughout the process, it leads to increased relevance, ownership, and use of the data.

Experiences Implementing ONA
From 2009 to 2014, MEASURE Evaluation implemented the ONA approach in Ethiopia, Thailand, and Malawi.There were many differences across the three countries in terms of services and client population (table 1).All three experiences had similar objectives, however, which can be summarized as: working to better understand how to link together multiple services, mainly through client referral, to improve client access to needed services.

Ethiopia:
In two sub-cities of Addis Ababa, Ethiopia, MEASURE Evaluation implemented an organizational network assessment of home and community based care and family planning networks.
We selected two distinct geographically defined communities with the intention to test whether using the results from an ONA could be used to strengthen the network of services available to people living with HIV.We worked with FHI 360's staff in Ethiopia who, with funds from the U.S. Agency for International Development (USAID), provided support to and built capacity of local NGOs in provision of community-based services and home-based care for people living with HIV.The organizations included in the study were NGOs; government; and faith-based or private organizations that provide a number of services including HIV prevention, HIV care and treatment, and family planning.We also conducted client interviews with female clients ages 18-49 of one large provider of home-based care services operating in both sub-cities to understand the clients' service use, care needs, and quality of life.By identifying all providers across sectors and their services, stakeholders were able to launch a process of developing the referral system.CBO= community-based organization; FBO=faith-based organization; MSM=men who have sex with men; NGO=nongovernmental organization; NACS=nutrition assessment, counseling, and support; ES/L/FS=economic strengthening, livelihoods, food security; n/a=not applicable.

Malawi:
In Balaka district, Malawi, MEASURE Evaluation tailored the ONA tool to align with the Livelihoods and Food Security Technical Assistance (LIFT) project objectives.1 LIFT sought to assess the network ties between nutrition assessment, counseling, and support (NACS) clinical sites at the facility level and economic strengthening, livelihoods, and food security (ES/L/FS) programs and projects in the community.NACS providers at specific NACS sites screen clients and then refer them to available community ES/L/FS services based on their identified needs.Clients of NACS services are also people living with HIV; however, any client with nutritional needs coming through the network would also be provided referrals for the ES/L/FS services.The objectives of the organizational data collection were to understand the links that currently exist both between NACS sites and ES/L/FS services, as well as between ES/L/FS services in the community themselves.Data from clients, at least two from each organization, were collected to understand client needs for the various ES/L/FS services, types of services they were aware of, their use of services (specifically ES/L/FS services) in the last three months, barriers to access services, and client satisfaction with services.
Thailand: In Chang Mai, Thailand, MEASURE Evaluation carried out a study to understand the network of services for men-who-have-sex-with-men (MSM) and transgender people.Organizations included in the study provided HIV counseling and testing, health services, outreach, HIV prevention education, advocacy, sexual reassignment, hormone therapy, and other supports services to MSM and transgender people.Data from clients were collected to understand clients' service needs, clients' perceived access to services, clients' reports of concerns with services, and clients' reports of perceptions of whether organizations were communicating and collaborating.
It is not the aim of this report to provide details about the study methodologies and findings, rather to draw lessons learned from across the applications.
More information is available about the work in Ethiopia at: http://www.cpc.unc.edu/measure/networks/organizational-networks/organizational-networks-in-ethiopia and http://www.cpc.unc.edu/measure/publications/ja-14-172Follow-up results were forthcoming as this report was being published.

Types of ONA analysis and results
In all three countries, data were collected from organizational representatives who could speak to the characteristics of their organization and whether their organization refers clients, shares resources (e.g., office space, written materials, pamphlets, posters, supplies, seed, agricultural instruments, equipment, staff, etc.), shares information (e.g., technical, training, educational, etc., and includes formal and informal communications such as notices or gossip), or conducts joint programming with other organizations.Information was also collected about the frequency of those connections over an average month and the quality of those connections.
To conduct the network analysis, data from organizations were analyzed using R (R Development Core Team, 2011), UCINET (Borgatti, Everett & Freeman, 2002), or Gephi (Bastian, Heymann & Jacomy, 2000).Data from organizations and client interviews were also analyzed in Stata (StataCorp, 2013) to yield descriptive information (means, proportions) and to conduct multivariate analyses.Open-ended questions yielded qualitative data were transcribed, reviewed, and major and minor themes identified.
We worked closely with in country partners to orient them to the concept of organizational network analysis, fundamentals of social network analysis, and analyzing network data in UCINET.
We produced results of the key attributes of organizations, network characteristics including the number of links, density, centralization, in-degree and out-degree of referrals sent and received, and reciprocity.
We assessed many types of relational data including resources shared, information shared, joint programming, and clients referred or received.We also produced sociograms and statistics of these relationships (figure 1).
From the data we produced a gap analysis that mapped what services each organization provided to what the others in the network provided (figure 2).Another output produced was the service directory describing where all the organizations were located, how to contact them, and services provided.We were also able to produce maps of the locations of the organizations utilizing the global positioning system (GPS) coordinates that were collected at each organization in the network.

Dissemination and use of the ONA results
In all three countries, the results were disseminated to stakeholders during a two-day workshop.The workshops included orientations to the concepts of organizational network analysis and how to interpret the results.Together with our in-country partners, we presented the preliminary results of the organizational and client data.The remainder of each workshop was structured to elicit high participation from participants to validate, interpret, and develop plans for use of the results.
In the workshops, participants validated the results of the ONA.For the most part, participants agreed with the linkages depicted.In some cases, participants noted where linkages no longer existed or ones had been made since the data collection.The meetings also allowed participants to discuss whether certain types of organizations, such as pharmacies, informal providers, or private providers that fell outside the original inclusion criteria, should be included in the network.Participants gained an understanding of the network they were a part of and discussed how to improve connections.We provided a directory of organizations in the network, the results of the gap analysis, and a map of the locations of the organizations.In general, we found that the client information, service directory, and gap analysis were the most useful to stakeholders.
In Thailand and Ethiopia, the results from the clients were very useful and provided a more comprehensive view to providers of the service provision successes and gaps than what they were aware of.
In Ethiopia, clients had unmet needs for services, mainly social services such as job training and funds for school fees, that were not fully understood by representatives of organizations prior to the study.Further, client data from Ethiopia revealed some problems with adherence to antiretroviral therapy and important quality of life concerns that served as call to participants to determine how to address in their programming.
In Thailand, the client interview results were the most valuable to stakeholders, who had not been fully aware of some concerns clients expressed.For example, stakeholders learned that pharmacies played an important role in care for MSM and transgender people.Also, there was a significant gap in clients' perceived quality of care compared with providers' perspectives.Confidentiality and privacy were seen by clients as a main driver of whether or not a client saw one provider versus another, or went to receive a service at all.
In Thailand, the organizational network was small and well-established, thus the ONA results reveal little new about this network, with one exception.The density of referrals by 'client sent' compared with that for 'clients received' was much higher.Stakeholders discussed potential problems with the referral measure (see below); however, participants learned that many clients mentioned that they had problems accessing services for reasons of lack of trust of the provider, lack of sufficient confidentiality measures at the referral site, or lack of understanding their specific needs as MSM and/or transgender people.
Thailand was the only country where we experienced some issues with organizations' participation in the study.Some organizations (hospitals) refused to participate as they viewed HIV services as relatively minor in their overall scope of care, or they were excluded from the study because their own internal review boards required additional review, beyond what was already sought for the study.This would have caused significant delays in study implementation.
In Malawi, client results were less useful to stakeholders than in Thailand and Ethiopia.During dissemination workshops, the most important information to participants was the identification of previously unknown network members.Recall that in this study, ONA was undertaken to identify clinical HIV and nutrition services and community-based food, livelihoods, and economic strengthening activities, where previously there were no or few linkages across sectors.The most important output to stakeholders was the information was compiled in the service directory, which served as a concrete resource.Many of the statistics associated with the sociograms (centrality, density, etc.) were not as useful for the network members and were seen as too complex for a burgeoning network of providers.
In Malawi, the organizational information was used to help build the network and establish referral patterns.The dissemination meeting served as a platform to bring all the organizational representatives and providers together to build consensus for subsequent activities that are being developed through the LIFT project in Balaka.Since then, the LIFT project used these results to understand the best way to engage stakeholders to become a network as well as to develop system to enable providers to respond to the clients' needs in a more effective and efficient manner.Subsequent iterations of the ONA (though highly adapted) have been used by LIFT in Namibia, Tanzania, Democratic Republic of Congo, and Lesotho since January 2013.
In all three countries of this study, the sociograms were useful to participants to visualize the connections between organizations.Although the content of the dissemination meetings included important information about the fundamentals of social network analysis and served as an opportunity to raise awareness about the method, some participants found the sociograms and particularly their statistics difficult to interpret.Measure of centrality and density were also not very useful to the stakeholders with the exception of Ethiopia.There, participants used the density statistic to set a target goal following the intervention (see below).The sociograms and statistics were useful to plan how to strengthen the network and the measures were useful in follow up measures to quantify changes over time.
Across all three contexts, participants generally agreed that increasing referral could result in better services for their clients and would be one strategy to alleviate the burden on the busy service provider to take on more responsibilities for service provision.Moreover, many providers are directed by government structures and/or program funding to work in a particular area.Participants agreed that better connections and collaboration with other organizations may lead to resource mobilization, capacity building, and more comprehensive services for clients.
Discussion in Thailand among stakeholders also revealed the need to define better what is meant by 'sending' and 'receiving' clients (i.e., referrals).The way the question was posed during the interview could lead the respondent to report a referral simply if he or she gave information about another organization, as opposed to whether the client actually visited the other organization and received services.Participants were also concerned that the respondents to the organizational survey, as administrators, may not actually be the most knowledgeable sources to speak about the linkages, particularly the frequency of linkages, since they are not always on the frontline of service provision.Thus, participants recommended developing more precise methods to measure referrals sent and received, and the frequency of referrals.
In Ethiopia, the results of the ONA were used to design an intervention to strengthen the network.A second ONA was conducted six months later to assess changes in the network.More about the intervention is described in the next section.

An intervention in Ethiopia
Following the dissemination meeting in Ethiopia, which included organizational representatives from both sub-cities, we implemented an organizational network strengthening intervention in one of the two sub-cities.The intervention consisted of three two-day meetings attended by representatives of the organizations, spaced approximately two months apart.The meeting content was adapted from organizational development materials developed by Management Sciences for Health (MSH) (Alterescu, 2012).The content was delivered by local Ethiopians who had been trained in organizational development and group facilitation.During the workshops, participants learned about and practiced leading and managing to face challenges and achieve results.A summary of the content of each work shop is found in table 2.

Workshop Purpose and Aactivities
First two-day meeting

Purpose:
Scanning, focusing and planning for results

Activities:
• Reviewed networks results • Reviewed the gaps analysis • Received an updated organizational directory • Set a goal to increase the number of referrals • Strategized to achieve the goal and identify barriers

Second two-day meeting
Purpose: To improve network effectiveness and ability to monitor and evaluate network activities Activities: • Developed an action plan to improve referrals

Purpose:
To increase and sustain the capacity of the network to face challenges and achieve measureable results

Activities:
• Reviewed and updated (if necessary) of action plan Participants relied heavily on the gap analysis and directory of organizations to assess barriers and develop strategies to make new connections with organizations.At the first workshop, participants selected a goal to increase the number of referral by 50% as measured by the referral density.Over the course of the workshops, participants discussed strategies and successes for making new connections, sought advice for barrier encountered, and engaged in planning for the next two-month period.
In addition to the meetings, motivational text messages sent between meetings.Text messages were also used to remind participants of subsequent meetings.At each meeting, each participant filled out an evaluation form to document new referrals made, barriers/facilitators to connecting, and added/subtracted services.
Following the intervention, a second round of organizational and client data were collected in both subcities.Comparison of baseline and follow-up data show that the number of referrals in the intervention sub-city surpassed the goal of increased referrals with organizations by 50%, whereas referrals in the comparison sub-city had declined.We also noted a decrease in reported service needs by clients in the intervention sub-city and fewer service needs among clients in the intervention sub-city compared with clients in the comparison sub-city.
The experience suggests that the network strengthening activity appears to have worked and ONA has the potential to generate data that can be used to improve referrals between organizations and quantitatively document changes.A stronger study design with multiple intervention and control communities is needed to validate the conceptual model and understand with greater precision the effect of the ONA approach on strengthened referrals (figure 3).A more representative sample of all clients will help understand the effect of improved referrals on services received, unmet care needs, and other outcomes such as ART adherence or contraceptive use.

Ethical Considerations
Any research study needs appropriate oversight by the relevant ethics committees (institutional review boards).However, we take the time to note some considerations for ONA studies, as protection of human rights, particularly with highly stigmatized, socially marginalized, or criminalized populations (such as some "key populations"), cannot be reiterated enough.These realities create environments that pose risks to human rights and undermine access and provision of needed health services.ONA studies typically collect information, such as contact information or geospatial information about organizations, that could pose risks to clients if misused.(For client interviews, however, there has not been a need to document any identifying information such as names, address, or to map locations, thus this information is less sensitive.)When working in such situations, member of key populations should be meaningfully engaged in all parts of the research process, and researchers should assess the human rights risks of activities and propose measures to address that risk (UNAIDS, 2011).
In general, measures of confidentiality and protection of human rights apply to ONA as they do to any study.These include the ethical principle of beneficence (i.e., do no harm), and guides the research such that the act of data collection and use does not facilitate or exacerbate arrests and prosecutions, harassment and violence, worsen discrimination and stigma, or otherwise cause harm to key populations.
Research studies need safeguards in place to ensure the confidentiality of the data collected and to secure the data after collection; to establish a system to supervise the research and report harmful incidents; to obtain informed consent from all participants; and to train all people involved in the study in ethics, the purpose of the study, and study procedures.

Lessons Learned and Recommendations
ONA is an approach to collecting data in a systematic way that will give organizations information about how they are currently connecting with other organizations and how they can improve those connections to benefit their clients.Information that appears to be most useful in our contexts was (1) the identification of organizations in the network and the resulting directories of organizations and services offered; (2) the visualization of the relationships between organizations in terms of client referral; and (3) information obtained from clients about their health needs and constraints to seeking referrals.The fundamental act of the dissemination meetings, where all organizations had an opportunity to interact, was an important aspect for the organizations to meet each other, understand their role, and plan to strengthen the network.
Our work in the three countries, including drawing on experiences in the literature, have revealed that ONA can: • provide quantitative measures of connections with other organizations; • provide visual representations of the relationships between organizations; • provide baseline information on which actions can be taken to improve connections; • provide follow-up quantitative measures to describe how the connections have changed over time; • shift the burden of health service integration from the provider level to the organization level through improved client referrals; • facilitate "network weaving" where new interactions are created between network clusters (Bloom et al., 2006); • identify organizations that should be included in the network (e.g., pharmacies, informal services) to aid in developing a more comprehensive network; • facilitate discussion among stakeholders to identify a mix of needed services, strategies of how to improve connections, and how to reduce barriers to making connections; • identify specific strategies to improve referral of clients between organizations; • identify specific strategies to fill gaps in service delivery; and • provide relevant new information about clients to organizations that serve them.
While there is no "right" way a network should look, more work is needed on methods and measure of effective networks.Our main measure of network effectiveness was related to the density of client referral patterns, as referrals are a key strategy to increase client access to needed services.We also relied on qualitative interpretation from network members to validate the network.Other concepts of network effectiveness that are proposed in the literature need more testing in developing country contexts.
Networks may be more effective where (Provan & Lemaire, 2012): • Multiple individuals within an organization are committed to the network (as opposed to a single individual).
• There is more than one type of tie between organizations (multiplexity).
• The network works together to address a "targeted and appropriate" issue rather than trying to take on too much.
• Network governance is shared between network members and they are focused on network level goals.
• Both internal and external legitimacy (i.e., credibility) of the network is acknowledged.
• A core of the network is stable over time (but allowing for some flexibility among members of the network).
ONA results and discussions with stakeholders revealed barriers to making connections with other organizations.These included lack of opportunities to develop relationships with people from other organizations, high staff turnover, no feedback mechanism between organizations when clients are referred, and competition between organizations to serve more clients.Other challenges identified in the network literature include varied commitment or network goals between organizations; clashes in the cultures of decision making, treatment methods, or training strategies; loss of autonomy; high time and effort associated with coordination; reduced accountability (or in our development world, reduced ability to attribute changes to the donor); and the complexity associated with management (Provan & Lemaire, 2012).
Stakeholders suggested to ameliorate the problems by establishing an inter-organizational referral feedback system, developing memoranda of understanding between organizations, creating opportunities for organization representatives to meet and share experiences, implementation of workshops on how to network effectively, and the identification of new connections in which both organizations could benefit.
There are a number of next steps to advance the ONA field.In general, approaches to make the data collection and analysis more accessible to representatives in developing countries without technical support from international organizations is needed.Work has already begun to adapt the questionnaire we used (provided in the appendix) to a digital tablet-based format.In Malawi, FHI 360 has modified it to make it a tablet-based data collection tool utilizing Open Data Kit or ODK, and MEASURE Evaluation has adapted it to a tablet-based version using Magpi, a DataDyne Group system for collecting data on mobile devices.Based on feedback, particularly from Thailand, more work is needed on the methods and measure of client referral.Broadly, more work is needed to define and test measures of organization network effectiveness.Finally, more research is needed to document an evidence based link between a strengthened network and improved client access to services and health.
The connections the ONA revealed helped people visualize and join a network they may not have been able to find on their own.In order to provide more holistic, life-long, and comprehensive care for clients, especially for those with complex needs, helping organizations understand their role in the network of care has the potential to increase their incentive for coordinating care to better meet the needs of the community.

Appendix. Sample Organizational Network Data Collection Tool
The following data collection instrument is a sample of the types of questions that could be used to gather information about the organizations, following an enumeration period to identify members of the network.Specifically, Section II collects data about the relationships between network members that can be used in network analysis.It is expected that in future applications of this tool, it will be adapted as needed to fit the specific context, including articulating specific inclusion criteria and informed consent procedures.

_____________________________
Section II: Facility/organization/bureau/office Linkages READ: I have a list of facilities, organizations, bureaus, and offices in (area of interest) that we believe are involved in some way in the provision of services, care and support to (population(s) of interest) .We would like to know the extent to which your facility/organization/bureau/office is involved with, or linked to, the others in the list for providing a range of services to (population(s) of interest).We have listed 9 types of involvement your facility/organization/bureau/office might have with these other programs and/or agencies.These include links through: • Shared funding (money) • Client referrals • Shared resources, including time, office space, written materials, pamphlets, posters, supplies, seed, agricultural instruments, equipment, staff, etc. (think of: things you can touch that you buy with money) • Shared information such as technical, training, educational, etc., includes formal and informal communications (e.g., notices or gossip).
• Joint programming.I will go through the list and ask you to indicate those which your facility/organization/bureau/office has been involved with for the provision of services to (population(s) of interest) in an average month.Please, indicate your level of involvement for each type of relationship in terms of "Never," "less than once a month," 1-3 times a month," 4-8 times a month," or "more than 8 times a month".Also you can tell me that you have no relationship with the facility/organization/bureau/office [check (√) the box labeled "not applicable" (N/A)}.

Never
Less than once a month 1-3 times a month

4-8 times a month
More than 8 times a month 0 1 2 3 4 Finally, I will ask you to rate the overall quality of the working relationship you have with each agency you have checked.For example, can you rely on the other agency to keep its word, to do a good job, and to respond to your program's needs and the needs of its clients?To do this, please circle the number that best reflects relationship quality using a scale where: 1= poor relationship (little reliability), 2= fair relationship (some reliability), 3= good relationship (reliable), 4= excellent relationship (high reliability).Again, if you have no relationship with a listed agency, I will leave it blank.At the end, I will ask you to add any programs and/or agencies you are involved with that are not listed but that you believe are valuable to your program in helping it address service provision to MSM in the community.

4.
Interview organization representatives: Conduct interviews with individuals representing each network member or organization to investigate: (a) their input into network members, (b) the quality and quantity of interactions between network members, and (c) their perspective on why connections do or do not exist.

Figure 3 .
Figure 3. Theoretical model of organization network change.

•
Discussed results achieved, where the problems are, and what is missing in terms of actions, allies, support or other • Conceptualized a positive "network climate" • Discussed capacity building in data analysis and interpretation • Discussed and practiced how to build trust • Discussed importance of and criteria for priority setting • Discussed approaches to mobilizing stakeholders to commit resources