Children with early focal brain injury show remarkable plasticity in language development. However, little is known about how early brain injury influences mathematical learning. Here, we examine early number understanding, comparing cardinal number knowledge of typically developing children (TD) and children with pre- and perinatal lesions (BI) between 42 and 50 months of age. We also examine how this knowledge relates to the number words children hear from their primary caregivers early in life. We find that children with BI, are, on average, only slightly behind TD children in both cardinal number knowledge and later mathematical performance, and show only slightly slower learning rates than TD children in cardinal number knowledge during the preschool years. We also find that parents’ “number talk” to their toddlers predicts later mathematical ability for both TD children and children with BI. These findings suggest a relatively optimistic story in which neural plasticity is at play in children’s mathematical development following early brain injury. Further, the effects of early number input suggest that intervening to enrich the number talk that children with BI hear during the preschool years could narrow the math achievement gap.