Take a look at the Recent articles

Abstract

Introduction : Serum transaminases rose significantly in 7 patients with chronic hepatitis C, genotypes 2 and 3, who were treated with pegylated interferon and ribavirin.

Methods : 219 patients with chronic hepatitis C, genotypes 2 and 3, were treated between 2005 and 2011 following the same protocol. For the 7 patients presented in this paper, the initial liver screen revealed chronic hepatitis C infection only. The same liver screen was repeated following the transaminase rise during the treatment period and failed to reveal additional comorbidity.

Results : 5 male and 2 female patients with chronic hepatitis C experienced a rise in serum transaminases after commencement on treatment with pegylated interferon and ribavirin. They all achieved rapid and end of treatment virological responses. 3 of the patients achieved sustained virological response and 4 relapsed. There was no evidence to suggest that steatosis, development of autoimmunity or intercurrent illness was the cause of the liver injury. In 3 out of 7 patients, the level of transaminases exhibited a downward trend after pegylated interferon was changed to non pegylated interferon. Additionally, it is evident that in those patients whose treatment was temporarily or permanently aborted, the rise in transaminases rapidly improved and returned to baseline.

Conclusion : Our experience suggests the possibility of a toxic reaction to polyethylene glycol in a small number of patients being treated with pegylated interferon, resulting in an acute hepatitic response which resolved when therapy was stopped or switched to non-pegylated interferon.

Key words

pegylated interferon; hepatitis C; toxic hepatitis

Abbreviations

HCV: Hepatitis C; SVR: Sustained Virological Response; LFT: Liver Function Tests; PEG: Polyethylene Glycol

Introduction

Hepatitis C (HCV) infection is a major public health problem, affecting 3% of the world’s population [1]. The infection persists in up to 85% of patients and may result in cirrhosis and hepatocellular carcinoma. Since 1999, the current standard therapy consists of combination therapy of pegylated interferon and ribavirin with sustained virological response (SVR) rates ranging between 42-82 % depending on the viral genotype [2]. Viral kinetic studies reveal that most patients with chronic hepatitis C (HCV) infection who achieve an SVR following standard treatment, clear HCV from serum within a few weeks [3,4]. This pattern of response to antiviral treatment is usually associated with a prompt reduction in elevated serum transaminases to normal.

The use of this combination of drugs is associated with a variety of side effects, including, occasionally, elevation of serum transaminases, which are rarely severe. Mild rises may reflect ongoing viral activity in treatment non-responders. More significant rises are infrequently observed despite a virological response and may be due to an immuno modulating effect of interferon, induction of steatohepatitis [5], direct toxicity in susceptible patients [6,7] and intercurrent illness.

In this report, we present 7 patients with chronic hepatitis C, genotypes 2 and 3, whose serum transaminases rose 1.5-14 x baseline during treatment with pegylated interferon α2a or α2b and ribavirin despite all achieving a rapid viral response and end of treatment response. In 3 out of 7 patients, the level of transaminases exhibited a downward trend after pegylated interferon was changed to non-pegylated interferon. It is also evident that in those patients whose treatment was temporarily or permanently aborted, their transaminases rapidly improved and returned to baseline.

Patient management summary

Between 2005 and 2011, a total of 219 patients with chronic hepatitis C, genotypes 2&3, were treated at Gartnavel General Hospital following the same protocol. 6 (2.8%) of these patients developed a hepatitic reaction during therapy and an additional patient was treated at the Victoria Infirmary. Prior to commencement on treatment, a full liver screen was performed in order to exclude co morbidities. That included full blood count, ferritin, coagulation profile, renal function indices, liver function tests (LFT), alpha-fetoprotein, caeruroplasmin, α1-antitrypsin, immunoglobulins, autoantibodies (ANA, AMA, SMA, anti-LKM, ANCA screen), hepatitis A, B, C, E, HIV antibodies, HCV RNA viral load and genotype and where indicated HBV DNA. Liver ultrasound was carried out in all patients. For the 7 patients presented in this report, the initial liver screen revealed chronic hepatitis C infection only. The same liver screen was repeated following the serum transaminase rise during treatment and failed to reveal additional comorbidities. The results are summarised in Table 1.

Table 1. Response to pegylated interferon & ribavirin in the seven patients with chronic hepatitis C

Case	Gender	Age	Ethnicity (Asian/ Caucasian)	Cirrhosis	Genotype	Pre-treatment ALT (U/L)	Peak ALT (U/L)	Start of ALT rise (weeks)	Duration of therapy (weeks)	PEG IFN α2a or α2b	Switch to non- PEG IFN α2a	Outcome (RVR, EVR, EOTR, SVR)
1	M	50	Caucasian	No	2b	52	398	2	24	α2a	No	RVR, EOTR, SVR
2	M	42	Asian	No	3	116	364	4	9	α2a	No	RVR, EOTR, no SVR
						144	404	2	8	α2a	No	RVR, EOTR, no SVR
						131	310	2	48	α2b	Yes	RVR, EOTR, no SVR
3	M	41	Asian	Yes	3	170	473	4	48	α2a	Yes	RVR, EOTR, no SVR
4	M	48	Caucasian	Yes	3	29	401	2	24	α2a	Yes	RVR, EOTR, no SVR
5	M	47	Caucasian	No	2a	39	123	8	24	α2a	No	RVR, EOTR, no SVR
6	F	45	Caucasian	Yes	3	111	187	4	7	α2a	No	RVR, SVR
7	F	34	Asian	No	3	209	368	4	16	α2a	No	RVR, EOTR, SVR

Case 1

In 2004, a 50 year old Caucasian man was found by chance to be hepatitis C positive with genotype 2b and the only risk factor for acquisition that was identified was a tendon repair in Saudi Arabia 15 years prior to presentation. He led a healthy lifestyle and did not abuse alcohol. He was systemically well and did not have cirrhosis.

He was commenced on treatment in October 2005 with pegylated interferon α2a (Pegasys, Roche) 180μgm weekly and ribavirin 800mg daily for 24 weeks. At the outset of treatment AST was 95 U/L, ALT 52 U/L, GGT 37 U/L, Alk Phos 207 U/L and bilirubin 10μmol/L. His transaminases started rising 2 weeks after commencement on treatment and reached a peak 22 weeks later with an AST of 668 U/L, ALT 398 U/L, GGT 541 U/L, Alk Phos 386 U/L and bilirubin 18μmol/L (Figure 1). He achieved a rapid virological response and therapy was continued for 24 weeks while monitoring the LFT.

Figure 1. Case 1, treatment with Peg IFN α2a and ribavirin for 24 weeks

He remained well throughout the period of treatment and his LFT returned rapidly to normal 2 months after cessation of treatment. He achieved an end of treatment response and SVR.

Case 2

In 2007, a 42 year old man of Pakistani origin, was diagnosed with hepatitis C, genotype 3, following investigation of abnormal LFT. The only risk factor for acquiring hepatitis C that was identified was multiple therapeutic inoculations in Pakistan as a child. He had been vaccinated for hepatitis B and was teetotal. His LFT at presentation were AST of 99 U/L, ALT 116 U/L, GGT 54 U/L and Alk Phos 65 U/L and he had already developed cirrhosis.

 He was commenced on treatment in March 2008 with pegylated interferon α2a 180mcg once weekly and ribavirin 800mg daily and was HCV RNA negative at week 4, at which point his liver function started deteriorating and the patient developed general malaise and nausea. As a result, the dose of interferon was reduced at week 8 to 90mcg. At week 9, treatment was stopped, as ALT rose to 423 U/L, AST to 435 U/L and GGT to 485 U/L (Figure 2). His bilirubin was 19 μmol/L and Alk Phos 103 U/L.

Figure 2. Case 2, first treatment attempt with Peg IFN α2a & Ribavirin for 9 weeks

6 weeks later, his liver function tests slowly improved, but ALT was still 2 and a half times the pre- treatment level and his symptoms resolved. 9 months after the treatment had stopped, AST dropped to 132 U/L, ALT to 150 U/L, GGT to 98 U/L but he was HCV RNA positive again.

He was offered further treatment in August 2009 with pegylated interferon α2a weekly and ribavirin daily after his liver function returned to baseline, with ALT of 144 U/L, AST 120 U/L, bilirubin 19 μmol/L. However, he had to stop again at 8 weeks following a rise of AST to 404 U/L, ALT to 364 U/L, GGT to 329 U/L and recurrence of symptoms. Again, he achieved rapid virological and end of treatment response.

At this stage he was referred to our clinic for consideration of further treatment options. His LFT returned to baseline 4 months following discontinuation of treatment, but his HCV RNA was again positive. Further antiviral treatment was attempted in June 2010 and at that time his AST was 131 U/L, ALT 131 U/L, GGT 91 U/L, Alk Phos 91 U/L and bilirubin 15 μmol/L. On this occasion, pegylated interferon α2b (Viraferon PEG, Schering-Plough) was used instead of α2a at a dose of 1.5 mcg/kg weekly and ribavirin 1000mg daily with a view to treating for 48 weeks. He had a rapid virological response and was PCR negative at 12 weeks. During this attempt, the LFT progressively deteriorated again and at week 12 his treatment was changed to non-pegylated interferon α2a (Roferon A, Roche) in addition to ribavirin. 4 weeks later, the transaminases started improving and at the end of 48 weeks of treatment in May 2011, they returned to baseline and his HCV RNA was negative (Figure 3). Unfortunately, he relapsed at 6 months following cessation of the last treatment attempt.

Figure 3. Case 2, treatment with Peg IFN α2b & Ribavirin, followed by treatment with non-Peg IFN α2a & Ribavirin in total for 48 weeks

Case 3

In May 2009, a 41 year old male of Pakistani origin was diagnosed with hepatitis C, genotype 3. He had a past medical history of jaundice as a child of unknown cause and had been vaccinated against hepatitis B, following his mother’s death from hepatitis C infection. He consumes less than a unit of alcohol per week. His baseline LFTs were deranged, with a bilirubin of 29 μmol/L, AST 146 U/L, ALT 227 U/L, GGT 61 U/L, Alk Phos 58 U/L. Anti-HBs was positive. Immunoglobulins showed IgG 19.3 g/L, IgA 1.63 g/L and IgM 0.83 g/L. He had cirrhosis, portal hypertension, oesophageal varices and a platelet count of 77x10⁹/L

He was started on pegylated interferon α2a 180mcg weekly and ribavirin 800mg daily in December 2010 with a plan to continue therapy for 48 weeks. Almost immediately after the onset of treatment, he developed a marked rise in serum transaminases and at week 12, his AST was 330 U/L, AST 473 U/L, bilirubin 52μmol/L, GGT 203 U/L and Alk Phos 77U/L (Figure 4). He remained asymptomatic. Repeat liver screening only showed a raised IgG level the titre of which was similar to the baseline (20.3 g/L).

Figure 4. Case 3, treatment with Peg IFN α2a and Ribavirin followed by treatment with non-Peg IFN α2a and Ribavirin in total for 48 weeks

 At that point, his interferon was changed to non-pegylated interferon α2a (Roferon A, Roche) with subsequent improvement in serum transaminases. At the end of treatment in November 2011, AST was 60/L, ALT was 58 U/L, GGT was 50 U/L, Alk Phos 73U/L and bilirubin 72μmol/L. He was HCV RNA negative at weeks 12, 24 and 48 but relapsed at the 6 month post treatment test.

Case 4

In February 2011, a 48 year old Caucasian man was referred for consideration of treatment for hepatitis C genotype 3 which was diagnosed in 2009. He had injected drugs for approximately 30 years but had been abstinent for 5 years prior to presentation. He did not abuse alcohol. He had developed cirrhosis with portal hypertension. His baseline LFT was as follows: AST 54 U/L, ALT 29 U/L, Alk Phos 117 U/L and bilirubin 7μmol/L.

He was commenced on treatment with pegylated interferon α2a weekly and ribavirin daily in August 2011. 2 weeks after initiation of treatment, his liver function deteriorated rapidly and reached a peak at week 8. AST rose to 554 U/L, ALT to 322 U/L, Alk Phos to 136 U/L, and bilirubin was 7μmol/L (Figure 5). Despite the rise in transaminases, he appeared to tolerate treatment well.

Figure 5. Case 4, treatment with Peg IFN α2a and Ribavirin followed by treatment with non-Peg IFN α2a and Ribavirin in total for 24 weeks

At week 13, his interferon was changed to a non-pegylated interferon α2a (Roferon A, Roche). At week 24, his AST was 45 U/L, ALT was 20 U/L, Alk Phos 82 U/L and bilirubin 3 μmol/L. He had a rapid virolological response and completed 24 weeks of therapy with an end of treatment response. However, he relapsed at the 6 month post treatment test.

Case 5

A 47 year old Caucasian man was diagnosed with chronic hepatitis C genotype 2 infection in 2001, the origin being blood transfusion in 1975. Liver biopsy in 2001 showed chronic hepatitis with a mild to moderate activity without fibrosis.

He was started on treatment in August 2005 with pegylated interferon α2a 180mcg per week and 800mg ribavirin daily for 24 weeks. His AST at that time was 30 U/L, ALT 30 U/L, GGT 71 U/L, Alk Phos 134 U/L, bilirubin 12 μmol/L. At week 8 of treatment, his liver function started deteriorating and at week 14 AST rose to 63 U/L, ALT to 118 U/L, GGT to 216 U/L with the rest of the liver function markers within normal limits. He achieved a rapid virological response. He was merely observed as he remained asymptomatic and gradually his liver function normalised. At the end of treatment in January 2006, he was HCV RNA negative. However, he relapsed at 6 months.

Case 6

In May 2010, a 45 year old Caucasian lady, who used to inject drugs, was diagnosed with hepatitis C genotype 3. She smoke approximately twenty cigarettes per day and drank very little alcohol. Her abdominal ultrasound showed fatty infiltration of the liver but no evidence of cirrhosis or portal hypertension. The serum HCV RNA showed a viral load less than 1000 U/L. The patient was very keen to be started on treatment which was commenced in August 2010. Prior to the onset of therapy, AST was 99 U/L, ALT 111 U/L, GGT 275 U/L, Alk Phos 111 U/L and bilirubin 8 μmol/L.

She received pegylated interferon α2a and ribavirin for 7 weeks until mid-October 2010 when the treatment was discontinued due to deterioration in her LFT which started rising almost immediately after treatment commenced. At that time, AST was 261 U/L, ALT 177 U/L, GGT 912 U/L, Alk Phos 244 U/L, and bilirubin 15 μmol/L. Her HCV RNA was negative at week 4 and at the end of treatment.

She was then seen again in March 2011. By that stage, her LFT had normalised with an AST of 34 U/L, ALT 28 U/L, GGT 133 U/L, bilirubin 8 μmol/L and Alk Phos 124 U/L. She remained HCV RNA negative. However, she underwent a fibroscan which was suggestive of cirrhosis.

Case 7

In July 2011, a 34 yea2021 Copyright OAT. All rights reservagnosed with chronic hepatitis C, genotype 3 infection. She had been living in the UK for 6 years prior to presentation and no risk factor for acquisition of hepatitis C could be identified. She was a non-smoker and did not abuse alcohol. She underwent a fibroscan which was in keeping with moderate fibrosis.

She was commenced on treatment with pegylated interferon α2a and ribavirin in October 2011. Her baseline LFT was abnormal with an ALT of 209 U/L, AST 126 U/L, Alk Phos 90 U/L and bilirubin 9 μmol/L. 8 weeks into treatment her liver function markers deteriorated despite having achieved a rapid virological response. She remained well but her LFT continued to rise and as a result treatment was aborted at week 16 when AST was 209 U/L, ALT 266 U/L, Alk phos 109 U/L and bilirubin 8 μmol/L . The repeat liver screen revealed a polyclonic rise in the immunoglobulins (IgG 24.40 g/L, IgM 2.76 g/L, IgA 3.32 g/L) but specific autoantibody tests were negative. Her liver function tests gradually improved and returned to baseline 6 weeks after the treatment was stopped. She achieved SVR and remains well.

Discussion

Seven patients with chronic hepatitis C are presented who sustained rapid and significant rises in serum transaminases during therapy with pegylated interferon α2a and ribavirin despite a rapid and end of treatment virological response. Three of the patients achieved sustained virological response and four relapsed. Interestingly, they all had non genotype 1 infection. Intercurrent illness, reactivation of hepatitis B and autoimmune hepatitis were excluded as possible causes.

During treatment with pegylated interferon, significant serum transaminase elevations above baseline are not common but are occasionally clinically significant. This biochemical response has been little studied. The transaminase elevations that occurred in 35% of patients treated with pegylated interferon and ribavirin for chronic HCV infection between October 2000 and October 2003 [5]. Most commonly, minor rises (< 1.5x baseline) occurred early in the course of treatment and were associated with failure to achieve a virological response. More significant transaminase elevations (>1.5x baseline) were observed later in the course of treatment and were associated with similar virological response rates to patients who did not experience toxicity. On univariate and multivariate analysis, increased body weight and increased severity of steatosis on pre-treatment liver biopsy were associated with transaminase elevations. Steatosis is a common finding in patients with chronic hepatitis C [8], and the pathogenesis is multifactorial. A causative role for HCV in the development of steatosis is supported by the observation that it occurs more commonly with HCV genotype 3 infections and improves after viral eradication [5]. Even though 2 of the patients presented in this paper had hypertension and diabetes, there was not enough biochemical or other evidence to suggest that non-alcoholic fatty liver disease contributed to the sustained liver injury.

Interferon has several antiviral mechanisms, but its role in hepatitis C seems to be related to its immunomodulatory effect [9]. Interferon therapy for patients with hepatitis C infection has been reported to induce or exacerbate autoimmune hepatitis [10,11]. Kogure et al. [12] have reported fulminant hepatic failure secondary to autoimmune hepatitis with development of anti LKM-1 and ANA antibodies that was confirmed on liver biopsy following a 12 week course of treatment with pegylated interferon alpha-2b for chronic hepatitis C (1a genotype) [12]. One of the explanations for the occurrence of autoimmunity in HCV patients is the loss of self-tolerance due to molecular mimicry between viral proteins and self-antigen [13]. Merli et al. [14] have reported development of de novo autoimmune hepatitis in patients who had HCV infected liver grafts (genotypes 1a &1b) and received treatment with pegylated interferon and ribavirin [14]. Parana et al. [15] reported the development of anti-Golgi complex and progression of liver disease in 3 patients during pegylated interferon treatment for HCV infection (genotypes 1a). In this study it was suggested that a rise in transaminases could be secondary to temporary autoimmune phenomenon triggered by treatment, however the lack of histological confirmation raised the question of hepatotoxicity of pegylated interferon α2a. In our own cases, autoantibody markers were at all times negative, which is against this hypothesis.

Caroleo et al. [7] have reported an increase in serum transaminase levels 4 months after the start of pegylated interferon α2a treatment for chronic hepatitis C (genotype 1b) secondary to a possible direct toxic effect as seen on immunostaining following liver biopsy. In contrast, transaminases normalised following change of treatment to pegylated interferon α2b as shown in the same case report [7]. It was postulated that because of the molecular weight of the PEG chain of pegylated interferon α2a (40kDa), it accumulates in the liver, has higher distribution in plasma volume and its half-life is approximately twice that of pegylated interferon α2b. Therefore, pegylated interferon α2a is able to accumulate in tissues such as liver which have high plasma volumes. In our report, the use of pegylated interferon α2b did not result in normalisation of transaminases, which was achieved only when switched to non-pegylated interferon α2a.

Mishra et al. [6] have reported a marked rise in transaminases with both pegylated forms of interferon [6], raising the question of toxicity secondary to polyethylene glycol. PEGylation provides benefits that are derived from the covalent attachment of polyethylene glycol (PEG), a synthetic polymer of ethylene oxide, to bioactive species. When PEG is attached to the surface of a protein, the resulting increase in size reduces renal clearance and increases stability by enhancing resistance to proteolysis, which prolongs protein circulating times [16]. PEGs, however, contain potential impurities such as ethylene oxide and 1,4 dioxane which are toxic. In 3 of the cases, the levels of the liver enzymes exhibited a downward trend once pegylated interferon was changed to non-pegylated interferon α. Additionally, it is evident that in those patients whose treatment was temporarily or permanently aborted, their transaminases decreased slowly to baseline.

Unfortunately, no liver biopsy was done during the hepatitic responses in our patients. This is a potential weakness as it is possible that this might have aided in the determination of the mechanism of this reaction and helped to rule out other conditions. However, the timing of the response in relation to pegylated interferon was so rapid, reproducible and resolved with switching to non-pegylated interferon or cessation of therapy as to leave no doubt as to the relationship to the therapy.

In summary, our report suggests the possibility of a toxic reaction to polyethylene glycol in 2.8% of patients being treated with pegylated interferon resulting in an acute hepatitic response which resolved when therapy was stopped or switched to non-pegylated interferon. It was interesting, but possibly coincidental, that this reaction only occurred in non-genotype 1 infection, that it was more common in Asian patients and that all patients had a rapid viral response and end of treatment response while on therapy. No other explanation for the acute hepatitis reaction was found in these patients despite extensive investigation. This rare toxic response to polyethylene glycol caused occasional symptoms, concern on the part of the medical teams but no serious illness despite 3 patients having cirrhosis. It rapidly resolves following cessation of therapy with no apparent longer-term sequelae.

References

1. WHO (World Health Organisation) data on “global prevalence of hepatitis C” 2012.
2. Ghany MG, Strader DB, Thomas DL, Seeff LB; American Association for the Study of Liver Diseases (2009) Diagnosis, management, and treatment of hepatitis C: an update. Hepatology 49: 1335-1374. [Crossref]
3. Fried MU, Hadziyannis SJ, Shiffman ML, Messinger D, Zeuzem S. Rapid virological response is the most important predictor of sustained virological response across genotypes in patients with hepatitis C virus infection. J.Hepatol 2011; 55:69-75
4. Lutchman G, Hoofnagle JH (2003) Viral kinetics in hepatitis C. Hepatology 37: 1257-1259. [Crossref]
5. Thurairajah PH, Thorburn D, Hubscher S, White A, Lai WK, O’Donnell K, Mutimer D : Incidence and characterization of serum transaminases elevation in pegylated interferon and ribavirin treated patients with chronic hepatitis C. Aliment Pharmacol Ther 2007;25:1293-1300
6. Mishra P, DeVoss A, Pai R, Hart J, Jensen DM (2009) Marked flare in hepatic aminotransferases during treatment with pegylated interferon for chronic hepatitis C, genotype 2: a case report. Dig Dis Sci 54: 1369-1372. [Crossref]
7. Caroleo B, Gallelli L, Staltari O,.De Sarro G, Guadagnino V.: Serum transaminase elevations during pegylated interferon treatment of chronic HCV hepatitis probably induced by polyethylene glycol. Intervirology 2008; 51:407-409
8. Goodman ZD, Ishak KG (1995) Histopathology of hepatitis C virus infection. Semin Liver Dis 15: 70-81. [Crossref]
9. Luxon BA, Grace M, Brassard D, Bordens R (2002) Pegylated interferons for the treatment of chronic hepatitis C infection. Clin Ther 24: 1363-1383. [Crossref]
10. Shindo M, Di Bisceglie AM, Hoofnagle JH (1992) Acute exacerbation of liver disease during interferon alfa therapy for chronic hepatitis C. Gastroenterology 102: 1406-1408. [Crossref]
11. Todros L, Saracco G, Durazzo M, Abate ML, Touscoz G, et al. (1995) Efficacy and safety of interferon alfa therapy in chronic hepatitis C with autoantibodies to liver-kidney microsomes. Hepatology 22: 1374-1378. [Crossref]
12. Kogure T, Ueno Y, Fukushima K, Nagasaki F, Inoue J, Kakazu E, Matsuda T, Kido O, Nakagome Y, Kimura O, Obara N, Wakui Y, Iwasaki T, Shimosegawa T. Fulminant hepatic failure in a case of autoimmune hepatitis in hepatitis C during peg-interferon alpha-2b plus ribavirin treatment. World J Gastroenterol 2007;13:4394-4397
13. Kita H, Mackay IR, Van De Water J, Gershwin ME (2001) The lymphoid liver: considerations on pathways to autoimmune injury. Gastroenterology 120: 1485-1501. [Crossref]
14. Merli M, Gentili F, Giusto M, Attili AF, Corradini SG, et al. (2009) Immune-mediated liver dysfunction after antiviral treatment in liver transplanted patients with hepatitis C: allo or autoimmune de novo hepatitis? Dig Liver Dis 41: 345-349. [Crossref]
15. Paraná R, Schinoni MI, de Freitas LA, Codes L, Cruz M, et al. (2006) Anti-Golgi complex antibodies during pegylated-interferon therapy for hepatitis C. Liver Int 26: 1148-1154. [Crossref]
16. Kang JS, Deluca PP, Lee KC (2009) Emerging PEGylated drugs. Expert Opin Emerg Drugs 14: 363-380. [Crossref]