Previous studies suggest that in Alzheimer's disease (AD) the Apolipoprotein E (APOE) ε4 allele is associated with greater vulnerability of medial temporal lobe structures. However, less is known about its effect on the whole cortical mantle. Here we aimed to identify APOE-related patterns of cortical atrophy in AD using an advanced computational anatomy technique. We studied 15 AD patients carriers (ε4+, age: 72 ± 10 SD years, MMSE: 20 ± 3 SD) and 14 non-carriers (ε4−, age: 69 ± 9, MMSE: 20 ± 5) of the ε4 allele and compared them to 29 age-and-sex matched controls (age: 70 ± 9, MMSE: 28 ± 1). Each subject underwent a clinical evaluation, a neuropsychological battery, and high-resolution MRI. UCLA's cortical pattern matching technique was used to identify regions of local cortical atrophy. ε4+ and ε4− patients showed similar performance on neuropsychological tests (p > .05, t-test). Diffuse cortical atrophy was detected for both ε4+ (p = .0001, permutation test) and ε4− patients (p = .0001, permutation test) relative to controls, and overall gray matter loss was about 15% in each patients group. Differences in gray matter loss between carriers and non-carriers mapped to the temporal cortex and right occipital pole (20% greater loss in carriers) and to the posterior cingulate, left orbitofrontal and dorsal fronto-parietal cortex (5–15% greater loss in non-carriers). APOE effect in AD was not significant (p > .74, ANOVA), but a significant APOE by region (temporal vs fronto-parietal cortex) interaction was detected (p = .002, ANOVA), in both early and late-onset patients (p < .05, ANOVA). We conclude that the ε4 allele modulates disease phenotype in AD, being associated with a pattern of differential temporal and fronto-parietal vulnerability.