Originally the Vrouwekerk (Church of Our Lady) was a chapel of a neighboring village. When part of that area was incorporated into the city of Leiden, the church was enlarged as a parish church for the northwestern side of town. Construction saw several enlargements from the fourteenth up to the sixteenth century. Before the Reformation this was a neighborhood church that included numerous devotional and trade guild chapels. Among them was the chapel of the surgeons' guild, dedicated to SS. Cosmas and Damian. The altarpiece of that chapel is probably the triptych (now in Vienna) painted by Cornelis Engebrechtsz., perhaps assisted by one or more of his three sons, Cornelis, Pieter, and Lucas. The painting displays the words "Het Comt [van Go]D"(It Comes from God), which must refer both to sickness and recovery. The bills, however, came from guild members.

After the Reformation, the church was assigned to the congregation of French-speaking refugees, the Walloons (Huguenots). This was a large congregation, ca. 6,000 at one time in the 17th century. These refugees came from many different towns and regions in what is now Belgium and northern France, and many had been in exile in England or Germany before coming to Leiden. The church differed from other Walloon congregations because Leiden's University included a theological seminary, the Walloon College, where French-speaking students were trained. Professor Johannes Polyander, closely associated with this theological school, became a personal friend of the Pilgrims' pastor John Robinson.

Among the Walloons were several who joined the Pilgrims, emigrating with them to Plymouth in New England. The most famous among them were Philippe de la Noye, later known as Philip Delano, and Philip's aunt and uncle, Esther Mahieu and Franchoys Coucke (Hester Mayhew and Francis Cooke). Francis was a passenger on the Mayflower. These people eventually became ancestors of American presidents, Ulysses Grant, Franklin Delano Roosevelt, George H. W. Bush, and George W. Bush.

Leiden's Walloon congregation, inspired by the Pilgrims, was also the source of a migration in 1622-1624, by which 55 families left Leiden to become settlers in Guyana and on Manhattan Island. The Vrouwekerk, even though in ruins, has become a symbol of Leiden's role in the movement of refugees to the city and again outward to New England and New Netherland (present-day New York).