homelessness prevention: the effect of a shallow rent subsidy program on housing outcomes among...
TRANSCRIPT
ORIGINAL PAPER
Homelessness Prevention: The Effect of a Shallow Rent Subsidy
Program on Housing Outcomes among People with HIV or AIDS
Lisa K. Dasinger Æ Richard Speiglman
Received: 12 July 2006 / Accepted: 30 April 2007 / Published online: 19 June 2007Ó Springer Science+Business Media, LLC 2007
Abstract This paper presents results of an evaluation of
Project Independence (PI), a shallow rent subsidy program
with services coordination support for very low income
people with HIV or AIDS who live in Alameda County in
the San Francisco Bay Area. By providing a small rental
subsidy to eligible individuals and their families who are
already stably housed, the philosophy of the program is to
prevent homelessness before it starts. The housing out-
comes of 185 PI clients were compared to those of 218
people who were not enrolled in the program but were
presumed eligible for it, controlling for sociodemographic,
HIV disease, and behavioral health characteristics. Using
survival analysis techniques, non-program participants
were found to be more likely to leave their rental housing
at any given point in time compared to PI program par-
ticipants. After one year of follow-up, while 99% of PI
clients remained stably housed in their program-subsidized
rental unit, only 32% of comparison group clients were still
in rental housing. At two years, 96% of PI participants
remained independently housed, compared to only 10% of
non-participants. The success of the program suggests that
Project Independence should be replicated and evaluated in
other jurisdictions where a relatively high incidence and
prevalence of HIV/AIDS is combined with a lack of
affordable housing for low income households.
Keywords Housing subsidy �Homelessness �HIV/AIDS �
Case management � Survival analysis � Hazard regression
Introduction
Until recently, AIDS was considered a terminal disease
from which most infected persons usually died within two
years of diagnosis. With the development of new classes
of antiretroviral medications and treatment regimens
(HAART, or Highly Active Anti-Retroviral Therapy), im-
proved clinical management, and earlier detection of the
disease, HIV-infected and AIDS-diagnosed individuals
now have a greater chance of experiencing better health
outcomes and living longer than previously thought pos-
sible. However, even with these significant medical
breakthroughs, the challenges facing people with HIV or
AIDS are still numerous. One of the greatest of these is the
challenge of maintaining a stable and affordable place to
live.
The ability to stay employed and maintain adequate
health insurance can be seriously hampered for persons
with HIV disease (Kass et al., 1994; Yelin, Greenblatt,
Hollander, & McMaster, 1991). Loss of employment and
health insurance can mean lack of financial resources to
meet rent or mortgage payments, which may lead to
homelessness, an inability to get or maintain proper med-
ical care, and the subsequent worsening of the disease.
Further, lack of stable housing, among persons living with
HIV/AIDS as well as others, is associated with high rates
of drug use and risky sex behavior (Aidala, Cross, Stall,
Harre, & Sumartojo, 2005; Sethi et al., 2004), in turn
increasing risks among the general population. There is
also evidence of an association between change for the
better in housing status and decline in risk-taking behavior
L. K. Dasinger (&)Division of Workers’ Compensation, Department of IndustrialRelations, State of California, 1515 Clay Street, 18th floor,Oakland, CA 94612, USAe-mail: [email protected]
R. SpeiglmanSpeiglman Norris Associates, Oakland, CA, USA
123
AIDS Behav (2007) 11:S128–S139
DOI 10.1007/s10461-007-9250-7
(Aidala et al., 2005). A number of recent studies suggest
that persons living with HIV/AIDS who are homeless or
unstably housed are less likely to receive adequate medical
care than those who are stably housed, putting them at
risk for poorer health outcomes (Aidala, Davis, Abramson,
& Lee, 2001; Aidala & Lekas, 1998; Aidala, Messeri,
Abramson, & Lee, 2001; Smith et al., 2000). When they do
obtain medical care, individuals with HIV/AIDS who are
unstably housed or homeless find adherence to the HAART
medication regimen particularly challenging (National
Health Care for the Homeless Council, nd).
Lack of housing has other effects on the costs of care
and the system of care as a whole. Marginally housed and
homeless individuals are more likely than stably-housed
persons to use more expensive forms of care, such as
emergency rooms (Arno et al., 1995) and inpatient hospi-
talization (Smith et al., 2000), and are less likely to use
ambulatory care (Arno et al., 1995). Weissman et al.
(1996) found that patients being treated for HIV/AIDS
were more likely to be hospitalized if they were homeless.
Bonuck and Arno (1997) also found that lack of housing
was a significant factor affecting hospital discharge for
persons with HIV/AIDS.
Aidala and Lee (2000) examined factors predicting
moving from an unstable to a stable housing situation, or
maintaining a stable housing situation. Homeless persons
were significantly more likely to get housing by their next
interview if they received rental assistance or mental health
services. Additionally, rental assistance and other housing
services were the strongest predictors of maintaining
housing. Substance use services, for persons with a drug
history, contributed significantly to maintaining housing.
In a multivariate analysis of the effects of services and
housing assistance for unstably housed persons with HIV/
AIDS, services had some impact, but receipt of rental
assistance was the strongest predictor of both obtaining
housing and staying in housing once placed. In summary,
‘‘rental assistance was a necessary, but often not sufficient,
condition for getting and maintaining housing among HIV/
AIDS-infected individuals’’ (Aidala, 2001).
Recognizing the need to assist persons living with AIDS
or HIV to either obtain housing or keep their current
housing, AIDS and housing advocates nationwide have
argued that special resources needed to be created for
people living with AIDS that could be made available more
quickly than traditional housing programs typically allow.
To this end, a number of local jurisdictions have developed
and implemented what are being called ‘‘shallow rent’’
subsidy programs. The term ‘‘shallow’’ is in contrast to the
notion of a traditional ‘‘deep’’ subsidy – one that reduces
the recipient’s housing expenses to a level that is deemed
‘‘affordable’’ by the federal government. In a deep subsidy
program, such as the Section 8 Housing Choice Voucher
program or Shelter Plus Care, the household receiving the
subsidy typically pays the landlord no more than 30% of its
gross income, adjusted for certain factors such as number
of dependents, physical disability, extraordinary child care
or health care expenses, and the cost of utilities. The sub-
sidy program then pays the difference between the tenant
payment and the full rent. In this way the housing is made
‘‘affordable’’ to the participating household.
A shallow subsidy program, in contrast, sets a defined
payment which the program provides, and the subsidized
household must pay everything additional to complete the
full rent, generally with no cap. Essentially, in a shallow
subsidy program the program pays the same subsidy level
for every household of comparable size, while in a deep
subsidy program the amount paid by the program may vary
significantly depending on the actual household income of
each subsidy holder. In the case where the amount of the
shallow subsidy is enough to allow the participating
household to pay 30% or less of its adjusted gross income
for housing, it is effectively acting as a deep subsidy
program for that household. In most shallow subsidy
programs, however, the amount the program pays is not
enough to bring most participating households’ portion of
the rent down to the affordability standard.
Given the fact that waiting lists for Section 8 in many
communities are five or more years long, coupled with the
fact that people living with HIV/AIDS are at greater risk
for faster progression of the disease and mortality if not
living in a stable housing situation, AIDS and housing
advocates have argued that shallow rent subsidy programs
may provide a viable alternative to deep rent subsidy
programs for some individuals. In addition, it was argued
that with limited resources, providing the minimum-
required subsidy to keep a larger number of persons with
AIDS housed may be preferable to providing deeper sub-
sidies to fewer people. In response to the above state of
affairs and advocacy efforts, the federal Department of
Housing and Urban Development (HUD) created the
Housing Opportunities for Persons With AIDS (HOPWA)
program in 1992. Among other types of housing assistance
programs, HOPWA funds are being used by some com-
munities to cover the cost of shallow rent subsidy programs
on the condition that communities offer a range of pro-
grams, including deep subsidies for those with greater
needs, and that the status of persons receiving shallow
subsidies be regularly reviewed to ensure that they are
being adequately supported.
This report presents the results of an evaluation of
Project Independence (PI), a shallow rent subsidy program
with services coordination support for very low income
people with HIV or AIDS and their families who live in
Alameda County, California. The evaluation covers about
the first four and a half years of the program, from March
AIDS Behav (2007) 11:S128–S139 S129
123
1997 to October 2001. Two characteristics of the County
during that time—a high AIDS case rate and a paucity
of affordable rental housing for low income fami-
lies—underscored the need, first identified in 1995, for a
housing subsidy program that would help low-income HIV/
AIDS-infected individuals at risk for homelessness remain
stably and independently housed. The Oakland Eligible
Metropolitan Area (EMA), which covers both Alameda
County and nearby Contra Costa County, is among the
fifty-one metropolitan areas in the nation that are dispro-
portionately affected by HIV/AIDS and receive emergency
relief funding through the Ryan White CARE Act to
provide primary care and support services for people with
HIV disease. According to the Alameda County Health
Department, there were 1,854 people with AIDS and an
estimated 6,400 with HIV disease living in the County at
the end of 1997. During the study period, the AIDS case
rate in Alameda County exceeded both the state rate and
the national rate. In 1999, the County rate was 17.0 new
cases per 100,000 population, compared to a state rate of
11.5 and a nationwide rate of 9.3. Oakland was the hardest
hit area of the County, accounting for 51% to 65% of new
AIDS cases each year between 1980 and 2000 (Alameda
County Public Health Department, 2001).
In addition to the relatively high AIDS case rate, Ala-
meda County suffers from a shortage of affordable housing
for low income households. Fair Market Rents (FMR) for a
one bedroom apartment in the county increased 16% dur-
ing the study period from $633 per month in 1997 to $734
per month in 2001 (U.S. Department of Housing and Urban
Development). It is easy to see that the limited income of
many people with HIV/AIDS, many of whom collect
Supplemental Security Income (SSI) or Social Security
Disability Income (SSDI), simply is not enough to cover
the cost of a rental unit in Alameda County. In 2001, for
example, the monthly SSI payment was $531. Assuming no
other income, for a single person household in 2001, the
Section 8 Housing Choice Voucher program would require
the renter to pay 30% of that income, or $159, toward rent.
For a one-bedroom apartment renting at $734 per month,
Section 8 would therefore pay $421 toward the rent. In
contrast, Project Independence pays a fixed amount ranging
from $175 to $425 a month, depending on the size of the
household and the number of bedrooms. For the same
rental unit, PI would pay $225 per month, therefore leaving
$509—virtually the entire monthly SSI/SSDI benefit—to
be paid by the individual toward rent.
At the time this study was begun, Project Independence
was one of eleven shallow subsidy programs operating
across the nation, of which all but one served people living
with HIV/AIDS. Five of the programs are in the State of
California. The oldest of these programs began in 1996,
indicating that this is still a relatively new programmatic
approach. Many of the programs serving persons living
with HIV/AIDS rely on HOPWA funding, although other
sources being utilized include Ryan White AIDS CARE
Act funds, city general funds, and private funding. The
number of households served varied considerably, ranging
from 16 families in Monterey, California to 565 households
in Chicago. Subsidy amounts ranged from a low of $45 a
month to as high as $500 a month, with the majority falling
in the $150 to $300/month range. There is variation in the
amount of time each program provides subsidies. Some
programs provide only short-term subsidies. Chicago’s
subsidies lasted only three months, though they could be
extended for up to six months. In contrast, Denver’s
Shallow Rent Program subsidies lasted as long as partici-
pants continued to qualify, though they were to be re-cer-
tified every 21 weeks. Like recipients of deep subsidy
rental assistance, Project Independence clients may remain
enrolled in the program for as long as they remain eligible
for it.
Project Independence is just one component of the AIDS
housing continuum in Alameda County. The program
developed out of a year-long community-based needs
assessment and planning process which included the par-
ticipation of AIDS services providers, housing developers,
government staff, and persons living with AIDS. The
program was developed in response to the findings of a 600
person survey conducted in 1995 of people living with HIV
in the County, which found that almost half were at risk of
losing their rental housing if their incomes dropped by as
little as $100 a month. By providing a small rental subsidy
to eligible individuals and their families, the philosophy of
the program is to prevent homelessness before it starts.
The evaluation was designed to answer the following
questions: (1) How long do people in the program stay
independently housed?; (2) Do people in the program stay
independently housed longer than people who are not in the
program but are eligible for it?; and (3) Besides program
participation, are there other factors which predict how
long low income people with HIV/AIDS remain stably and
independently housed?
Description of Project Independence
Overview
Project Independence (PI) is a rental housing assistance
program for very low income people living with HIV/AIDS
in Alameda County, California. In existence since 1997, PI
targets HIV positive individuals and their families who are
already stably and independently housed in a rental unit in
the County, but whose low income puts them at risk for
homelessness.
S130 AIDS Behav (2007) 11:S128–S139
123
Subsidy Amounts
Since its inception, at any given time the PI program has
provided between 100 and 125 qualifying households with
a monthly rental subsidy. The subsidy amount, unchanged
during the program’s life, varies depending on household
size, rent, and the income of the eligible household, but
averages $225 per month. The maximum rental subsidy for
PI is calculated using a simple matrix based on the number
of bedrooms and household size, as shown in Table 1.1
Program Eligibility Criteria, Activities, and
Requirements
To be eligible for Project Independence, an applicant must
have a diagnosis of HIV or AIDS, be 18 years of age or
older (or be an emancipated minor), have an income at or
below 50% of the area median income, have a lease on a
rental unit within Alameda County, and be willing to en-
gage in services as needed that promote independent living.
At least 65% of households in the program are expected to
have incomes below 25% of the area median income.
Single individuals, adult couples, and families with chil-
dren are all eligible for the program, although three-quar-
ters of participants have been single adults. PI participants
do not have to be disabled, have substance abuse issues, or
have other problems to be eligible for the program, al-
though having such presenting issues does not preclude
enrollment in the program.
Before receiving a subsidy, the rental unit must pass a
federal housing quality standards (HQS) inspection, which
determines that the unit meets minimum material and
equipment conditions in order to ensure habitability, dura-
bility, health, and life safety. If an applicant’s unit does not
pass the inspection, it can be repaired and re-inspected, or
the applicant can choose to locate and move to another unit.
In addition to the rental subsidy, participants are pro-
vided with limited case management services—termed
‘‘services coordination’’—provided by one of three non-
profit community-based ‘‘hub’’ agencies. Participants also
have access to a range of community-based services for
persons living with HIV/AIDS, including more intensive
case management, primary medical care, food, mental
health treatment, substance abuse treatment, peer coun-
seling and other supportive services. These services do not
differ from those available to other persons living with
HIV/AIDS in Alameda County, but Project Independence
was designed to maximize the probability that participants
will be made aware of the range of possible services and be
assisted with gaining access to them. Contact with a ser-
vices coordinator has been required on a quarterly basis, at
minimum.
Program Administration
Project Independence is a collaboration of the County
Housing and Community Development (HCD) Department
and three non-profit ‘‘hub’’ agencies. Since the founding of
Project Independence these three agencies have received
Ryan White CARE Act funds, which are used to provide
primary care and support services for people with HIV
disease. The hub agencies accept program applications,
issue the rental subsidy checks to landlords, and provide
tenants with services coordination, including referrals to
supportive services. HCD approves applications, coordi-
nates the overall program, and, at the time of this evalua-
tion, conducted HQS inspections. New applications are
reviewed and processed on a first-come first-served basis as
existing clients leave the program.
Program Funding
At the time this evaluation was conducted, Project Inde-
pendence was funded by the Department of Housing and
Urban Development (HUD) under the Housing Opportu-
nities for Persons with AIDS (HOPWA) Special Project of
National Significance (SPNS) competitive program. Pro-
ject Independence was first funded in 1996 for a three-year
period, and the first subsidies were provided in March,
1997. PI has been renewed for three additional three-year
terms and is currently funded under a HOPWA Permanent
Supportive Housing grant. For the first two of the three
renewals the grant included funding to conduct a research
study of the impact and effectiveness of the PI program;
this study reports on the first of those.
Method
Research Design
Using a quasi-experimental retrospective longitudinal
comparison group design, the study compares the housing
Table 1 Project independence subsidy amounts
Studio/0 bedroom 1 bedroom 2 bedroom 3 + bedroom
1 person $175 $225
2 people $275 $325
3 people $325 $375 $425
1 The subsidy is actually the lesser of the maximum subsidy and thedifference between the rent and 30% of the household’s adjustedgross income. In 80% of cases, the qualifying household receives themaximum. Only if the maximum subsidy causes the household to payless than 30% of its adjusted income for rent and utilities is thesubsidy lower.
AIDS Behav (2007) 11:S128–S139 S131
123
outcomes of PI program participants to those of individuals
in a comparison group who appear to meet the requirements
of the program but are not enrolled in it. The outcome
evaluated is the duration of time individuals in both groups
remain in rental housing while continuing to meet the
income requirements of the program, controlling for soci-
odemographic, HIV diagnosis, and behavioral health char-
acteristics. The maximum length of follow-up for any
individual was four years eight months, the time from
program inception to final data collection for the evaluation.
Measures
An electronic database maintained by HCD provided
demographic and program participation data for Project
Independence participants, including age, gender, race/
ethnicity, stage of HIV disease at program entry, city of
residence, household size, total monthly household income,
income level ( £ 25% area median income (AMI) or 26–
50% AMI), size of rental unit (e.g., studio, 1-bedroom),
total monthly rent, amount of rental subsidy, program entry
and exit dates, and reason for exit, if applicable. Additional
information on Project Independence clients was obtained
from the Alameda County Public Health Department Office
of AIDS Administration (OAA) database, which contains
data on people who receive Ryan White-, HOPWA-, and/or
county-funded HIV/AIDS-related direct services. This
database was also used to obtain a comparison group for the
study. The OAA database consists of data collected on
‘‘Client Intake Forms’’ that are submitted by each HIV/
AIDS service provider in the county to OAA once each
fiscal year for each client who received services at that
agency during the year. Entry of this document into the
database creates a separate record for each client for each
agency for each contract year in the database. In addition to
providing demographic information comparable to that
available from the HCD database, the OAA database con-
tained the following variables that were used in this eval-
uation: primary language, HIV transmission categorie(s),
month/year of first positive HIV test, household composi-
tion, presence of other presenting issues (substance abuse,
mental health, STD, hepatitis, tuberculosis, homelessness),
and current housing status. An anonymous client ID, which
appears both in the HCD and the OAA databases, was used
to identify records that pertained to Project Independence
cases to obtain supplemental information about them, and to
exclude them from the comparison group.
Participants
HCD provided a final dataset for this evaluation on October
26, 2001, by which time a total of 256 client-households
had been served by the program since March 1, 1997.
The comparison group was drawn from the OAA dataset,
which consisted of 19,003 intake records representing 5,565
unique client IDs. Intake dates ranged from January 1997 to
July 2001. The comparison group was selected in the fol-
lowing way. First, the earliest intake record in the OAA
database for which all of the following were true was used
to select a group of individuals eligible for the comparison
group: (1) intake date ‡3/1/97, the first month PI subsidies
were issued; (2) intake occurred at one of the three hub
agencies; (3) HIV positive or AIDS diagnosis; (4) ‡18 years
of age; (5) household size £ 5, the largest household size of
PI households; (6) total household income £ 50% AMI,
adjusted for household size and intake date; and (7) living in
rental housing or apartment/house. Cases were excluded if,
on the same intake record, any of the following applied: (1)
‘‘living with relatives/friends,’’ suggesting the individual
may have been doubled up with others rather than living
independently; (2) client IDs partially matching those of PI
clients’, to safeguard against including a PI client who had
more than one client ID in use, due to typos and other
factors. Cases without at least one later intake record where
housing status and income were known were excluded. For
remaining cases, the most recent intake record for which
housing status and income were known, where no inter-
vening records existed with unknown housing status and
income, was chosen. This was so that a longitudinal mea-
sure of length of time in rental housing and the income
eligibility of the individual for the program could be cal-
culated for members of the comparison group.
After application of the above criteria to the OAA
database, only 238 client IDs, or 4.3% of all client IDs,
remained for inclusion in the comparison group. Most
clients were excluded from the comparison group because
they were never in rental housing or their housing status
was unknown. Fewer clients were excluded because they
had only one intake record in the OAA database or had
missing data on income or household composition, which
made the determination of income level, and hence PI
eligibility, impossible.
It is important to note that eligibility of comparison
group members for PI is unknown in four areas: whether the
client held the lease on the rental unit, whether the rental
unit met HUD housing quality standards, the rent amount
and its relationship to HUD fair market rents and the client’s
income, and the willingness of the client to participate in
services needed for independent living. The comparison
group may therefore have contained some individuals who
would not have been eligible for the program.
Data Analysis
The analytic technique used for the evaluation is Cox
proportional hazards regression, a time-to-event or survival
S132 AIDS Behav (2007) 11:S128–S139
123
analysis technique (Cox & Oakes, 1984). As its name
implies, survival analysis was developed in the context of
studying time to death, disease onset, or disease recurrence
among clinical populations, typically in evaluating the
effectiveness of a given treatment. Clinical trials typically
measure the length of time from study entry to a disease
endpoint—the ‘‘event’’—for a treatment and a control
group. Time-to-event (survival) analysis allows inclusion
of patients who fail to complete the trial or do not reach the
study endpoint before the study ends (censored data) by
making comparisons between the number of survivors in
each group at multiple points in time. Survival analysis
techniques are especially important since excluding
patients who do not reach the endpoint from the analysis
could introduce considerable bias because the length of
survival for these patients prior to study exit is important
information that contributes to the power and validity of
the study.
The main results of a Cox analysis are typically de-
scribed in two ways. Time-to-event curves, also called
cumulative survival curves, provide estimates of the pro-
portion of individuals in each group still ‘‘surviving’’ at
any given point in time after study entry through the
study’s maximum follow-up period, where everyone’s
‘‘start time’’ is standardized to time ‘‘zero.’’ The ‘‘relative
hazard,’’ also called the ‘‘hazard ratio,’’ is a number that
indicates the instantaneous increase or decrease in risk of
experiencing the study event at any given point in time
relative to a baseline reference group, e.g., the treatment
versus control group. The hazard ratio cannot be used to
describe how much faster/sooner this event may occur, but
only the instantaneous relative risk of it occurring for one
group compared to another. Hazard ratios greater than one
indicate an increased risk of the event occurring and ratios
less than one a decreased risk. An assumption of propor-
tional hazards regression is that the hazard ratio is constant
over time.
In the context of this evaluation, the outcome of interest
is the time an individual remains independently housed in a
rental unit after entrance into the study. The event of
interest is therefore loss of independent, rental housing. For
the Project Independence and the comparison group, loss of
housing included living with relatives/friends, in a hotel/
motel, or in transitional housing, being homeless (in
emergency shelter or on the streets), residing in a psychi-
atric, substance abuse treatment, hospital or other medical
facility, residing in jail/prison, or ‘‘other’’ at last obser-
vation. For Project Independence clients, loss of housing
also included moving into Section 8, Shelter Plus Care, or
some other form of deep subsidized housing, being evicted
or losing the lease, or living in ineligible housing. The
study follows individuals from the time they either entered
the program (PI participants) or the study (comparison
group), when first observed to be housed in a rental unit
and eligible for the program, until their housing status
changed, or until follow-up data are no longer available.
For those still independently housed in a rental unit at
the time of last observation, the ‘‘true’’ survival time is
only known to be a value greater than the time observed
from study entry to the last observation. For these indi-
viduals, the time-to-event duration is unknown, and the
final observation is said to be ‘‘censored.’’ We do not know
how much longer the individual remained or will remain in
their current housing situation after the final observation.
Housing outcomes were also considered censored if (1) the
individual became income ineligible for the Project
Independence program at last observation, i.e., household
income exceeded 50% AMI, (2) the individual left the
County, or (3) the individual died. The latter two condi-
tions were only known for PI clients. Individuals with
censored housing outcomes contribute data to the compu-
tation of the survival curve up until the time right before
their final observation date.
All available variables were individually tested for their
association with time remaining in rental housing using the
Cox method. Variables with a significant association with
housing outcome in bivariate survival analyses (P £ .10)
or, as with age, which are commonly used as a control
variable, were retained for the final analyses presented
here. All analyses were conducted using SPSS version
11.0. The research was approved by the institutional review
boards serving the Public Health Institute (Oakland, CA)
and Speiglman Norris Associates (Oakland, CA).
Results
We first describe sociodemographic, income, HIV status,
and behavioral health characteristics of PI participants and
members of the comparison group. We then present the re-
sults of a multivariate survival analysis of the effect of pro-
gram involvement (Project Independence vs. Comparison
Group) on the length of time in independent rental housing
while controlling for other key factors. Deidentified names
are used for the hub agencies (A, B, and C) so as not to single
out any particular hub agency as ‘‘better’’ or ‘‘worse’’ in
terms of producing a positive housing outcome.
Both the descriptive and multivariate analyses are lim-
ited to individuals with no missing values on all the vari-
ables of interest. This resulted in a sample size of 187 for
the Project Independence group and 219 for the compari-
son group. Most (91%) of the 69 PI clients lost from the
analysis had no data on other presenting issues in the OAA
database during the time they were enrolled in PI. Since
including these cases in the analysis and dropping the
variables for other presenting issues did not change the
AIDS Behav (2007) 11:S128–S139 S133
123
significance and/or effect size of other variables in the
analysis, the model with more variables and fewer cases is
presented here.
Table 2 shows that the PI group and the comparison
group did not statistically differ in terms of age, presence
of substance abuse issues, or hepatitis. Sociodemographi-
cally, members of the comparison group were more likely
to be female (32.4% vs. 21.4%), black (72.6% vs. 59.9%),
living with others (54.8% vs. 23.0%), have a household
income at or below 25% of the area median income (90.0%
vs. 64.7%), and be unemployed during their time in the
study (87.7% vs. 76.5%). Regarding health characteristics,
they were more likely to have AIDS (92.7% vs. 59.9%),
and to have purportedly contracted HIV through intrave-
nous drug use (34.7% vs. 18.7%), but less likely to have
self-identified as having a mental health issue (21.9% vs.
39.6%). In addition, almost all comparison group clients’
baseline intake was conducted at hub agency A (94.1%)
compared to 38% of PI clients. No comparison group client
had an initial study intake at hub agency C. This distri-
bution of comparison group clients among the hub agencies
is not surprising, given the distribution of intakes among
the three hub agencies represented in the OAA database.
To the degree that each of the variables in Table 2 has an
impact on length of time in independent rental housing, it is
important to control for these variables in the multivariate
analysis.
Table 3 shows the results of the multivariate survival
analysis. Since there were no statistically significant dif-
ferences between White, Asian/Pacific Islander, and Native
American clients on housing outcome in the bivariate
analysis, and the sample size of the latter two categories
was too small to treat them as separate groups, these were
placed into a single category. The analysis also excludes
clients with ‘‘other’’ gender, due to small sample size,
resulting in 185 PI clients and 218 comparison group cli-
ents included in the final analysis.
By far the most significant predictor of time remaining
in independent rental housing, while controlling for other
variables in the analysis, is program status. Members of the
comparison group were 3.83 times more likely to lose their
rental housing at any point in time after entrance into the
study than Project Independence participants (P < .001).
Smaller but significant effects were seen for the impact of
hub agency, race/ethnicity, employment, mental health,
and hepatitis on the risk of leaving rental housing. Age,
gender, living alone, household income level, history of
injection drug use, stage of HIV disease, and having a
substance abuse issue, although significant in bivariate
analyses, were not associated with the risk of losing rental
housing in the multivariate analysis.
The hub agency through which individuals entered the
study was significantly associated with the risk of losing
rental housing, independent of enrollment in PI. PI and
comparison group individuals who enrolled through Hub
Agency B had a decreased risk of 0.53 times that of Hub
Agency A clients of losing their rental housing at any given
point in time (P < .05). The risk of leaving rental housing
among clients of Hub Agency C was 0.16 times the risk
among Hub Agency A clients (P < .01). Possible reasons
for these observed differences are the following: (1) clients
served by each agency may have differed systematically in
ways relating to housing outcome which were not mea-
sured in this study, e.g., more disadvantaged, harder to
serve, or more precariously housed, (2) hub agencies may
have differed in levels of case management, services
coordination, other operating practices, and/or the provi-
sion of other services that affect a client’s ability to remain
housed, and (3) in terms of PI clients, the hub agencies may
have applied differential selection criteria when recom-
mending clients for the program, biasing the results toward
a more or less favorable outcome.
Compared to White, Asian/Pacific Islander, and Native
American clients, the risk of losing rental housing was 1.54
times greater for Black clients (P < .10) and 2.61 times
greater for Hispanic clients (P < .01). Whether these dif-
ferences are attributable to cultural differences in preferred
living arrangements, discrimination, the kinds of services
received, or other factors is unknown. Among members of
the comparison group, Blacks and Hispanics who lost their
rental housing were more likely to end up living with rel-
atives and friends than Whites, Asian/Pacific Islanders and
Native Americans (data not shown).2 Of Blacks who left
rental housing, 36.3% indicated ‘‘lives with relatives/
friends’’ on the final client intake form used for the last
observation in the analysis, and of Hispanics, 55.6%
mentioned this living arrangement. Only 13.6% of Whites,
Asian/Pacific Islanders, and Native Americans who left
rental housing had this final housing outcome. Although
from the perspective of the goals of Project Independence,
doubling up with friends and relatives is a ‘‘negative’’
outcome, it may be preferred by some individu-
als—perhaps among certain racial/ethnic group lines—to
living alone and/or independently.
Clients who were not employed during their time of
observation in the study had an increased risk of leaving
rental housing that was 1.41 times that of individuals who
had a part-time or full-time job at some point during par-
ticipation in the study (P < .05). Employment may be
associated with a number of factors which increase the
ability of an individual to remain stably housed, for
example, greater levels of self-sufficiency, health, and self-
advocacy that enable the client to secure needed services
and supports to stay housed.
2 Comparable data on the PI participants are not available.
S134 AIDS Behav (2007) 11:S128–S139
123
The risk of losing rental housing for individuals with a
mental health issue was 0.59 times the risk for individuals
without a mental health issue (P < .05). Similarly, pre-
senting with hepatitis was associated with a decreased risk
of losing rental housing of 0.44 times (P < .01). One
possible explanation of these findings is that these indi-
viduals, by virtue of self-disclosing a mental health or
hepatitis issue, may be more likely to get help with these
Table 2 Characteristics ofProject Independence versuscomparison group clients
* P \ .05, ** P \ .01,*** P \ .001 (Pearson chi-square)
Project Independence (N = 187) Comparison Group (N = 219) P
% N % N
Age
18–30 9.6 18 7.8 17
31–50 78.6 147 82.6 181
51+ 11.8 22 9.6 21
Gender *
Female 21.4 40 32.4 71
Male 77.5 145 67.1 147
Other 1.1 2 0.5 1
Race/Ethnicity *
White 27.8 52 19.6 43
Black 59.9 112 72.6 159
Hispanic 8 15 5.9 13
Asian/Pacific Islander 3.7 7 0.5 1
Native American 0.5 1 1.4 3
Lives Alone ***
Yes 77 144 45.2 99
No 23 43 54.8 120
Household Income ***
£ 25% AMI 64.7 121 90 197
26%–50% AMI 35.3 66 10 22
Employed PT or FT **
Yes 23.5 44 12.3 27
No 76.5 143 87.7 192
HIV Status ***
HIV+ 40.1 75 7.3 16
AIDS 59.9 112 92.7 203
Injection Drug User ***
Yes 18.7 35 34.7 76
No 81.3 152 65.3 143
Substance Abuse Issue
Yes 25.7 48 22.4 49
No 74.3 139 77.6 170
Mental Health Issue ***
Yes 39.6 74 21.9 48
No 60.4 113 78.1 171
Hepatitis
Yes 16 30 13.2 29
No 84 157 86.8 190
Hub Agency ***
A 38 71 94.1 206
B 30.5 57 5.9 13
C 31.6 59 0 0
AIDS Behav (2007) 11:S128–S139 S135
123
problems. The help received, in the form of additional
case management or other services, may in turn facilitate
staying stably housed, on its own or through its association
with other services received. The group of people who did
not present with a mental health or hepatitis issue may have
included individuals who actually had these problems, but
by lack of self-disclosure, did not receive needed help.
Self-disclosure of a behavioral health issue may be asso-
ciated with a greater level of self-advocacy.
Figure 1 shows the estimated proportion of people
remaining independently housed in a rental unit over time
as a function of program status (Project Independence vs.
comparison group), while controlling for all other factors
in the multivariate analysis. The graph presents a striking
picture of the positive impact of Project Independence on
keeping individuals stably housed. After half a year of
follow-up, 99% of Project Independence clients were still
in rental housing compared to 65% of comparison group
clients. After one year of follow-up, while 99% of Project
Independence clients were still in their PI-subsidized
unit, only 32% of comparison group clients were still in
rental housing. At two years follow-up, the percentage
of Project Independence clients estimated to be in rental
housing remained high at 96%, while the percentage of
comparison group clients still in rental housing had
dropped to 10%. Unadjusted for other variables, the
average time in rental housing for program participants
was 1,416 ± 77 days, or 3.9 years. The corresponding
figure for individuals in the comparison group was
372 ± 58 days, or 1.0 years
Table 3 Results of multivariate survival analysis of the effect ofprogram participation and other variables on time in independent,rental housing
Variable HazardRatio
95% ConfidenceInterval
Program-related variables
Program status
Project Independence 1
Comparison Group 3.83*** 2.79–5.25
Hub agency
A 1
B 0.53* 0.29–0.97
C 0.16** 0.05–0.54
Sociodemographic variables
Age
18 – 30 1
31 – 50 1.48 0.79 – 2.76
51+ 1.21 0.56 – 2.61
Gender
Female 1
Male 1.21 0.83 – 1.77
Race/Ethnicity
White, Asian, or Nat. Am. 1
Black 1.54 0.97 – 2.44
Hispanic 2.61** 1.29 – 5.28
Client Lives Alone
No 1
Yes 1.28 0.91 – 1.80
Household Income
£ 25% MFI 1
26%–50% MFI 1.01 0.79 – 1.29
Employed PT or FT
Yes 1
No 1.41* 1.03 – 1.92
HIV-related variables
Injection drug user
No 1
Yes 1.25 0.86 – 1.81
Stage of HIV Disease
AIDS Diagnosis 1
HIV+, not AIDS 1.21 0.71 – 2.06
Other Presenting Issues
Substance Abuse
No 1
Yes 1 0.62 – 1.60
Mental Health
No 1
Yes 0.59* 0.36 – 0.95
Hepatitis
No 1
Yes 0.44** 0.24 – 0.80
* P < .05, ** P < .01, *** P < .0001
Time in rental housing (years)
43210
la
vivr
uS
evit
alu
mu
C
1.0
.8
.6
.4
.2
0.0
Program Status
Project Independence
Comparison Group
Fig. 1 Survival function by program participation, adjusted for allother variables
S136 AIDS Behav (2007) 11:S128–S139
123
Discussion
The results of this evaluation provide strong support for the
success of the Project Independence Program in meeting its
objective of keeping low income people with HIV/AIDS
and their families independently housed. The average time
in rental housing for program participants, unadjusted for
other variables, was 3.9 years, for the nearly five years
since the program began in March 1997 to October 2001.
When adjusted for sociodemographic, HIV disease, and
behavioral health characteristics, members of a comparison
group of individuals who met program eligibility criteria
but were not enrolled in the program were 3.8 times more
likely to leave their rental housing at any given point in
time compared to Project Independence program partici-
pants. After one year of follow-up, an estimated 99% of
Project Independence clients were still living in their rental
unit, compared to only 32% of comparison group clients,
adjusted for sociodemographic, HIV-related, and behav-
ioral health factors. At 2 years follow-up, only 10% of
comparison group clients were still housed in a rental unit,
compared to 96% of Project Independence participants.
The hub agency through which the client was enrolled
into the study was also found to be associated with time
remaining independently housed in a rental unit. Further
investigation of the reasons accounting for the observed
hub agency differences in housing outcomes is needed in
order to maximize the success of all clients in maintaining
their housing. Although the independent influence of re-
ceipt of services coordination support, case management,
or any other kind of service, on housing outcome could not
be studied due to lack of readily available data, evidence
from another study provides some support for the added
beneficial effect of services. In the study by Aidala and Lee
(2000) of 600 people with HIV/AIDS living in New York
City, social services were found to have some impact on
the maintenance of a stable housing situation, although
receipt of a rental subsidy had a bigger impact. Further
studies are needed to determine the degree to which addi-
tional client characteristics and the uptake of the services
coordination component of the program, as well as utili-
zation of other kinds of services, affect housing outcomes.
In a later evaluation of the Project Independence pro-
gram, which studied housing outcomes among PI clients
only over a longer time frame (from March 1997 to
December 2005), the effect of hub agency was no longer
significant (Speiglman & Dasinger, 2006). This study
found that clients who enrolled in the program later were
more likely to remain successfully housed for a longer
period of time than those who had enrolled earlier. Inter-
views with hub agency and other program staff suggest that
program implementation or start-up issues, and a more
open application process, with less attention to or under-
standing of who would best be served by the program, may
partly account for the hub agency differences seen in the
program’s earlier years. A third potential factor mentioned
by key informants is that treatment of the disease was not
as good in earlier as in subsequent years.
In addition to program-related factors, a number of other
client characteristics significantly predicted length of time
in rental housing, namely, race/ethnicity, employment, and
presenting with a mental health or hepatitis issue. His-
panics were 2.6 times more likely to leave rental housing at
any given point in time and Blacks were the 1.5 times more
likely compared to Whites, Asian/Pacific Islanders and
Native Americans. Many factors, whether a product of
client attitudes, expectations, and behaviors, attitudes and
behavior of landlords, or other, ‘‘environmental’’ condi-
tions, could have contributed to observed differences by
race/ethnicity.
However, it is noteworthy that findings from other
examinations of what is termed ‘‘housing dynamics’’ differ
from ours. Generally, Black and Hispanic recipients of
housing assistance are less likely than others to leave the
housing subsidies. In a sense, they ‘‘conserve’’ the avail-
able housing resources. Using a hazard model from late
1980’s Survey of Income and Program Participation (SIPP)
data to estimate tenure in housing programs, Hungerford
(1996) found that, compared to others, Blacks were sig-
nificantly less likely to depart subsidized public housing
(P < .10) and, though not significantly so, also less likely
to leave subsidized rental housing. More recently, relying
on stratified, random samples of recipients of assistance in
the 1995–2000 period from the Department of Housing and
Urban Development through the Multifamily Tenant
Characteristics System and Tenant Rental Assistance Cer-
tification System databases, Ambrose (2005) found stron-
ger results: African-American households have a much
lower probability of leaving both public housing and ten-
ant-based assisted housing programs, compared with White
households. Hispanics, compared with non-Hispanics, are
significantly less likely to depart assisted housing. They are
also less likely to leave public housing (P < .10). Finally,
Olsen, Davis, and Carrillo (2005) report similar findings in
their study of HUD data for the 1995 to 2002 period. The
interpretation is that Blacks and Hispanics, like other
groups with relatively fewer social options, exhibit greater
tendencies to retain subsidies.
Although we have no data to directly interpret our
contrary race/ethnicity finding, we believe research on this
matter is critically important. Very low income people
living with HIV or AIDS receiving housing assistance
through the Project Independence program arguably face
even more challenges than other populations of individuals
who receive housing assistance. In that context, compared
to the situation for Whites, an array of biographical and
AIDS Behav (2007) 11:S128–S139 S137
123
societal conditions may have a greater limiting influence
on Black and Hispanic clients’ ability to access the kinds
of services and/or supports necessary to maintaining
stable—and subsidized—housing.
Having a part- or full-time job was associated with a
decreased risk of losing rental housing, as was having a
disclosed mental health or hepatitis issue. These variables
may each be associated with a higher level of self advocacy
and/or self-sufficiency, which in turn are associated with
obtaining the kinds of services and supports that are nec-
essary to maintaining stable housing.
The main limitation of the study is that the compara-
bility of the comparison group and the Project Indepen-
dence group is not entirely known. Among members of the
comparison group, no information was available about
rent amounts, quality and size of the unit, or whether the
client’s name was actually on the lease. Nor was it known
whether comparison group members were committed to
engaging in activities that would facilitate living indepen-
dently. Comparison group members may have been paying
a larger share of their household income toward rent, in
comparison to Project Independence participants both be-
fore and after receiving the rental subsidy, which would put
them at more of an economic disadvantage. Although an
attempt was made to limit the comparison group to clients
who were living independently by excluding individuals
whose initial eligible intake indicated ‘‘living with rela-
tives/friends,’’ this was no guarantee that the client was
actually on a rental lease agreement and living indepen-
dently. Members of the Project Independence group may
also have been in stable housing before entering the pro-
gram longer than comparison group clients when they were
entered into the study. Interestingly, most (94%) compar-
ison group members were clients of the hub agency which
showed the poorest housing outcomes. The housing out-
comes for clients at that hub agency may have been con-
founded with other client and/or hub agency characteristics
that were not measured in this study, but which were
associated with poorer housing outcomes.
Another study limitation is the relative lack of precision
in the measurement of time in rental housing for compar-
ison group clients versus Project Independence clients.
Unlike the PI clients, whose continued participation in the
program was dependent upon hub agencies’ verification of
the client’s housing situation throughout program partici-
pation, the housing situation of comparison group clients
was only known at the moment of intake at the community-
based agencies at which they received services. It was
therefore not possible to know exactly when a comparison
group member’s housing status changed. The program
departures of PI clients who left the county were treated as
non-failure outcomes, assuming that the clients’ moves
represented greater self-sufficiency. In fact, we have no
data on the situations in which former clients found
themselves. It is probable that some departures were to less
independent and less stable settings. These measurement
errors produce both under- and overestimates of time in
rental housing among both groups.
Despite the limitations of the study, the large difference
in housing outcomes observed between the Project Inde-
pendence group and the comparison group strongly sup-
port the conclusion that Project Independence is successful
at helping low income HIV/AIDS-affected households
remain independently and stably housed. The importance
of a Project Independence-type subsidy also rests in the
fact that it can be made available relatively quickly to a
large number of individuals. In comparison, a deep rental
subsidy such as Section 8 requires a much larger sum of
money to help the same number of people. Nevertheless,
the Project Independence evaluation did not benefit from a
randomized, controlled design. Given the ethical chal-
lenges and other difficulties in fielding such rigorous
studies, we believe it would be appropriate to repli-
cate—and evaluate—the Project Independence program in
other areas of the state and country experiencing a rela-
tively high incidence and prevalence of HIV/AIDS com-
bined with a lack of affordable housing for low income
households. We note that the model of a shallow rental
subsidy plus services coordination may prove relevant for
other populations. Demonstration programs which include
an evaluative component might apply the Project Inde-
pendence model to any number of client groups. We would
encourage prospective studies, with more comprehensive
data collection across a number of factors, including some
follow-up of clients departing the program.
Project Independence and other housing programs can
be evaluated not only in terms of effects on stable housing
and the prevention of homelessness but also in terms of
their effects on maintaining or improving health and
avoiding unnecessary high cost health care such as hospi-
talization. Elsewhere, housing has been found to predict
lower HIV risk and less risk of transmission of HIV as well
as improved health outcomes. Another approach, which has
been considered but not yet implemented in Alameda
County, would involve studies that examine clients’ versus
controls’ access to and utilization of health-related and
other services and measure the cost-effectiveness of Project
Independence. Each of these topics is addressed in a recent
policy paper from the National AIDS Housing Coalition
(2007), which considers housing a ‘‘promising structural
intervention to prevent and treat HIV’’ (p. 5). Whether
considering housing outcomes, HIV transmission, the
promotion of positive health outcomes, appropriate use
of outpatient, emergency, and inpatient services, or cost
effectiveness, shallow rent subsidy programs with associ-
ated supportive services should be examined in comparison
S138 AIDS Behav (2007) 11:S128–S139
123
to other housing models: deep subsidy programs, housing-
first programs, and programs with more, and less, sup-
portive services. Where causal effectiveness in preserving
housing and health—of HIV seropositive people as well as
others—as a result of these programs can be demonstrated,
additional support will be generated and more individuals
at risk of losing housing will be assisted so that they can
keep their homes and protect their own and the commu-
nity’s health.
Acknowledgments This paper is based on an evaluation of ProjectIndependence that was completed by the authors in 2002 withfunding from the Alameda County Community DevelopmentAgency, Housing and Community Development Department (HCD),through a Housing Opportunities for Persons with AIDS Program(HOPWA) Special Project of National Significance (SPNS)competitive grant from the Department of Housing and UrbanDevelopment (HUD) to HCD. HCD subcontracted with the PublicHealth Institute to perform this initial evaluation (Contract# SPNS-99-04). Additional funding for this paper was provided by a HOP-WA Program Permanent Supportive Housing Renewal grant fromHUD to HCD and a subcontract with Speiglman Norris Associatesfor a follow-up evaluation of Project Independence (Contract #C-2005-207). We would like to thank the following current orformer HCD and Alameda County Office of AIDS Administrationofficials who assisted with the evaluations: Katharine Gale, LindaGardner, Laura Hansen, Tom Mosmiller, Sandra Pastermack, HazelWeiss, and Riley Wilkerson. Over the course of this study several ofthe Alameda County officials served as invaluable sounding postsand, along with the editor and two anonymous reviewers, veryhelpful critics.
References
Aidala, A. A. (2001). Personal communication, November.Aidala, A. A., Davis, N., Abramson, D., & Lee, G. (2001). Housing
status and health outcomes among persons living with HIV/
AIDS. New York: Columbia University.Aidala, A. A., & Lee, G. (2000). Housing services and housing
stability among persons living with HIV/AIDS. New York:Columbia University.
Aidala, A. A., & Lekas, M. (1998). Top client-identified unmet needs
for medical and social services. New York: Columbia Univer-sity.
Aidala, A. A., Messeri, P., Abramson, D., & Lee, G. (2001). Housingand health care among persons living with HIV/AIDS. NewYork: Columbia University.
Aidala, A., Cross, J. E., Stall, R., Harre, D., & Sumartojo, E. (2005).Housing status and HIV risk behaviors: Implications forprevention and policy. AIDS and Behavior, 9(3), 251–265.
Alameda County Public Health Department (2001). AIDS epidemi-ology report, Alameda County, California, 1980–2000. Oakland.
Ambrose, B. W. (2005). A hazard rate analysis of leavers and stayersin assisted housing programs. Cityscape: A Journal of Policy
Development and Research, 8(2), 69–93.Arno, P. S., Bonuck, K. A., Green, J., Fleishman, J., Bennett, C. L.,
Fahs, M. C., Maffeo, C., & Drucker, E. (1995). The impact ofhousing status on health care utilization among persons with HIVdisease. Journal of Health Care for the Poor and Underserved,
7(1), 36–49.Bonuck, K. A., & Arno, P. S. (1997). Social and medical factors
affecting hospital discharge of persons with HIV/AIDS. Journalof Community Health, 22(4), 225–232.
Cox, D. R., & Oakes, D. (1984). Analysis of Survival Data. London:Chapman and Hall.
Hungerford, T. L. (1996). The dynamics of housing assistance spells.Journal of Urban Economics, 39(2), 193–208.
Kass, N. E., Munoz, A., Chen, B., Zucconi, S. L., & Bing, E. G.(1994). Changes in employment, insurance, and income inrelation to HIV status and disease progression. Journal of
Acquired Immune Deficiency Syndrome, 7(1), 86–91.National AIDS Housing Coalition. (2007). Transforming fact into
strategy – developing a public health response to the housingneeds of persons living with and at risk of HIV/AIDS. PolicyPaper from the Second National Housing and HIV/AIDSResearch Summit. Washington, D.C.: The National AIDSHousing Coalition.
National Health Care for the Homeless Council (nd). The health careof homeless persons: A manual of communicable diseases andcommon problems in shelters and on the streets. RetrievedAugust 12, 2005.
Olsen, E. O., Davis, S. E., & Carrilo, P. E. (2005). Explaining attritionin the housing voucher program. Cityscape: A Journal of Policy
Development and Research, 8(2), 95–113.Sethi, A. K., Celentano, D. D., Gange, S. J., Gallant, J. E., Vlahov, D.,
& Farzadegan, H. (2004). High-risk behavior and potentialtransmission of drug-resistant HIV among injection drug users.Journal of AIDS, 35(5), 503–510, cited in National AIDSHousing Coalition, nd.
Smith, M. Y., Rapkin, B. D., Winkel, G., Springer, C., Chhabra, R., &Feldman, I. S. (2000). Housing status and health care serviceutilization among low-income persons with HIV/AIDS. Journalof General Internal Medicine, 15(10), 731–738.
Speiglman, R., & Dasinger, L. K. (2006). Alameda county projectindependence evaluation, phase 2: A longitudinal study of ashallow rent subsidy program for people with HIV/AIDS.Oakland: Speiglman Norris Associates.
U.S. Department of Housing and Urban Development (2006) FairMarket Rents. Accessed September 24, 2006. http://www.hud-user.org/datasets/fmr.html.
Weissman, J. S., Cleary, P. D., Seage, G. R., Gatsonis, C., Haas, J. S.,Chasan-Taber, S., & Epstein, A. M. (1996). The influence ofhealth-related quality of life and social characteristics on hospitaluse by patients with AIDS in the Boston Health Study. Medical
Care, 34(10), 1037–1054.Yelin, E. H., Greenblatt, R. M., Hollander, H., McMaster, J. R.
(1991). The impact of HIV-related illness on employment.American Journal of Public Health, 81(1), 79–84.
AIDS Behav (2007) 11:S128–S139 S139
123