Fulvestrant with or without Palbociclib and Avelumab in Treating Patients with Hormone Receptor Positive, HER2 Negative Metastatic or Recurrent Breast Cancer That Cannot Be Removed by Surgery Previously Treated with CDK and Endocrine Therapy

Inclusion Criteria

• Participants must have histologically confirmed hormone receptor positive (HR+) HER2 negative metastatic or locally recurrent unresectable invasive breast cancer; both measurable and non-measurable disease are allowed; ER, progesterone receptor (PR) and HER2 measurements should be performed according to institutional guidelines, in a Clinical Laboratory Improvement Act (CLIA)-approved setting; cut-off values for positive/negative staining should be in accordance with current ASCO/CAP (American Society of Clinical Oncology/College of American Pathologists) guidelines
• Ongoing monthly gonadotrophin releasing hormone (GNRH) agonist is required in pre-menopausal women or male participants for at least 4 weeks prior to study entry
• Participants must have radiological or objective evidence of progression on an endocrine and CDK 4/6 inhibitor regimen in the metastatic setting, and/or relapse/progression during or within 12 months of completion of an endocrine and CDK4/6 inhibitor regimen in the adjuvant setting * Participants must have previously been exposed to CDK4/6 inhibitor therapy in combination with endocrine therapy; exposure to any prior CDK4/6 inhibitor, (including palbociclib, abemaciclib, and ribociclib) is allowed; patients may have a line of endocrine therapy after combination endocrine and CDK4/6 inhibitor exposure * Participants must have remained on prior endocrine and CDK4/6 therapy in the metastatic setting without progression for at least 6 months prior to study entry * It is not mandatory to have a CDK 4/6 inhibitor containing regimen as the most recent treatment * Participants may have had no more than 1 prior line of endocrine and CDK4/6 inhibitor therapy in the metastatic setting
• Participants may have 0-1 prior lines of cytotoxic chemotherapy in the metastatic setting
• Prior endocrine therapy in the metastatic setting may include any aromatase inhibitor (AI) or tamoxifen; in the metastatic setting, 1-2 prior lines of endocrine therapy are allowed; prior fulvestrant is permitted if the participant received < 2 doses without concurrent endocrine therapy prior to enrollment and without evidence of disease progression
• Participants may have received radiotherapy for palliative purpose, but must not be experiencing > grade 1 treatment related toxicities, and must have completed treatment > 14 days prior to registration
• Eastern Cooperative Oncology Group (ECOG) performance status 0-1
• Absolute neutrophil count >= 1,500/uL
• Platelets >= 100,000/uL
• Hemoglobin >= 9 g/dL
• Total bilirubin =< 1.5 x institutional upper limit of normal (ULN)
• Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase [SGOT])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase [SGPT]) =< 2.5 x institutional ULN, or =< 5 x ULN for subjects with documented metastatic disease to the liver
• Creatinine =< institutional ULN or creatinine clearance >= 30 mL/min/1.73 m^2 for subjects with creatinine levels above institutional ULN
• Women of childbearing age, women who are made postmenopausal through use of GNRH agonists, and men must agree to use adequate contraception for the duration of protocol treatment and for at least 90 days after the last dose of palbociclib if the risk of contraception exists; adequate contraception is defined as one highly effective non-hormonal form of contraception or two effective forms of non-hormonal contraception by the participant and/or partner * Highly effective non-hormonal contraception ** Methods of birth control which result in a low failure rate (i.e., less than 1% per year) when used consistently and correctly are considered highly-effective forms of contraception ** The following non-hormonal methods of contraception are acceptable: *** True abstinence when this is in line with the preferred and usual lifestyle of the participant; (periodic abstinence [e.g., calendar, ovulation, symptothermal post-ovulation methods] and withdrawal are not acceptable methods of contraception) *** Male sterilization (with appropriate post-vasectomy documentation of the absence of sperm in the ejaculate); for female participants, the vasectomized male partner should be the sole partner OR * Effective non-hormonal contraception ** Alternatively two of the following effective forms of contraception may be used instead: *** Placement of non-hormonal intrauterine device (IUD) or intrauterine system (IUS); consideration should be given to the type of device being used, as there is higher failure rates quoted for certain types, e.g., steel or copper wire *** Condom with spermicidal foam/gel/film/cream/suppository *** Occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream/suppository *** The use of barrier contraceptives should always be supplemented with the use of spermicide; the following should be noted: **** Failure rates indicate that, when used alone, the diaphragm and condom are not highly effective forms of contraception; therefore, the use of additional spermicides does confer additional theoretical contraceptive protection **** However, spermicides alone are ineffective at preventing pregnancy when the whole ejaculate is spilled; therefore, spermicides are not a barrier method of contraception and should not be used alone * It should be noted that two forms of effective contraception are required; a double barrier method is acceptable, which is defined as condom and occlusive cap (diaphragm or cervical/vault caps) with spermicidal foam/gel/film/cream/suppository
• Premenopausal women must have a negative serum or urine pregnancy test; pregnancy testing does not need to be pursued in female participants who are: * Age > 60 years; or * Age < 60 with intact uterus and amenorrhea for 12 consecutive months or more AND estrogen (estradiol) levels within postmenopausal range; or * Status-post bilateral oophorectomy, total hysterectomy, or bilateral tubal ligation
• Participant must be able to swallow and retain oral medication
• Ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria

• Participants who have had endocrine, chemotherapy, and/or biologic therapy < 7 days (other than fulvestrant) prior to entering the study or those who have not recovered from any prior treatment-related toxicities (must recover to no more than grade 1; alopecia, sensory neuropathy grade =< 2, or other grade =< 2 toxicity not constituting a safety risk based on investigator’s judgment are acceptable)
• Participants who have had prior therapy with any anti-PD-1, anti-PD-L1, L2, or anti-CTLA4 immunotherapy
• Participants who are receiving concurrent therapy with other investigational agents
• Rapidly progressive, symptomatic, visceral spread of disease placing participant at risk of life-threatening complications in the short term
• Participants with active brain metastases; stable treated brain metastases are allowed (this includes participants who have documented radiologic stability at least 4 weeks after radiotherapy, and do not require systemic steroids for management of symptoms from central nervous system [CNS] metastatic lesions)
• Participants who have discontinued prior palbociclib for toxicity, or have needed more than two dose reductions for toxicity from prior palbociclib therapy; if a participant previously required dose reductions during prior palbociclib therapy and tolerated it well, for example prior dosing at 100 mg or 75 mg qd 3 weeks on 1 week off schedule, than that dose may be selected for this trial
• History of allergic reactions to palbociclib or attributed to compounds of similar chemical or biologic composition to palbociclib
• Known prior severe hypersensitivity to investigational product or any component in its formulations, including known severe hypersensitivity reactions to monoclonal antibodies (National Cancer Institute [NCI] Common Terminology Criteria for Adverse Events [CTCAE] version [v]4.03 grade >= 3)
• Participants receiving any medications or substances that are strong inhibitors or inducers of CYP3A isoenzymes are ineligible; because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated list ; medical reference texts such as the Physicians’ Desk Reference may also provide this information; as part of the enrollment/informed consent procedures, the participant will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the participant is considering a new over-the-counter medicine or herbal product
• Prior organ transplantation including allogenic stem-cell transplantation
• Current use of immunosuppressive medication, except for the following: a. intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection); b. systemic corticosteroids at physiologic doses =< 10 mg/day of prednisone or equivalent; c. steroids as premedication for hypersensitivity reactions (e.g., computed tomography [CT] scan premedication)
• Uncontrolled intercurrent illness including, but not limited to: ongoing or active infection requiring systemic therapy, clinically significant cardiovascular disease including: cerebral vascular accident/stroke (< 6 months prior to enrollment), myocardial infarction (< 6 months prior to enrollment), unstable angina, congestive heart failure (>= New York Heart Association Classification class II), or serious cardiac arrhythmia requiring medication, uncontrolled diabetes mellitus, or psychiatric illness/social situations that would limit compliance with study requirements; ability to comply with study requirements is to be assessed by each investigator at the time of screening for study participation
• Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent; participants with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible
• Known history of testing positive for human immunodeficiency virus (HIV) or known acquired immunodeficiency syndrome, or need to receive combination antiretroviral therapy for HIV
• Known history of colitis, inflammatory bowel disease, pneumonitis, or pulmonary fibrosis
• Hepatitis B virus (HBV) or hepatitis C virus (HCV) infection at screening (positive HBV surface antigen or HCV RNA if anti-HCV antibody screening test positive)
• Live vaccination within 4 weeks of the first dose of avelumab
• Pregnant women are excluded from this study because effect of palbociclib and avelumab on a developing fetus is unknown; breastfeeding should be discontinued prior to entry onto the study
• Individuals with a history of a different malignancy are ineligible except for the following circumstances; individuals with a history of other malignancies are eligible if they have been disease-free for at least 3 years and are deemed by the investigator to be at low risk for recurrence of that malignancy; individuals with the following cancers are eligible if diagnosed and treated within the past 5 years: ductal carcinoma in situ of the breast, cervical cancer in situ, and basal cell or squamous cell carcinoma of the skin