He taught in Chiswick before moving to Ludlow, then proceeded to Balliol College, Oxford. Williams chose the career of a public-school master, commencing his career at Winchester. He returned to Wales when offered the vicarage of Lampeter, where St David’s College was established in 1822. Such was his reputation that the second son of Sir Walter Scott was sent to Lampeter as a private pupil. In 1824, Williams was invited to become the headmaster of the new Academy founded at Edinburgh, whose aims were to raise the standard of classical education and especially of Greek learning. He retired from the rectorship of the Academy in 1847. Sir Walter Scott was to eulogise him as “a heaven-born teacher” and “the best schoolmaster in Europe”. Within a few weeks of his retirement he was appointed the first warden of Llandovery. Ill health caused him to retire in 1853, but not before he had raised Llandovery to a foremost position among the schools of Wales.
The second warden was to be in office for only 15 months. The Reverend David James was appointed by a majority vote of the trustees, but their choice was highly controversial, causing furore the community. His suitability was questioned as he was no classical scholar without real experience of schoolmastering. There was a marked decline in pupil enrolment from 80 to 25.
The Reverend Evan Owen Phillips became the third Warden in 1854 at the age of 27. His kindness and forbearance was evident, but he was firm and fair with pupils and gained their respect. Pupil enrolment increased from 20 to 80 by 1861 and the school enjoyed considerable academic success. On his departure in 1861 the tributes to Phillips’ wardenship were laudatory, and the efficient state of the school being particularly stressed.
At the age of 25, the Reverend William Watkins was appointed warden in 1861. He was an old boy of the school, being one of John Williams first pupils. From Llandovery he gained a mathematical scholarship at Cambridge, and taught at Eton. He was a fine teacher, able to arouse great enthusiasm amongst his pupils. However, the fabric of the College declined during Watkins’ 14 year tenure, and by 1875 the buildings were in need of considerable repair. There was considerable disillusionment with the wardenship, and the numbers sank to about 40. There was also a lack of discipline, and Watkins resigned in 1875.
The task facing the new Warden was a daunting one, and the Reverend A.G.Edwards, formerly Second Master, was appointed. He has properly been described as the second founder of Llandovery. He abolished the boarding-out system, and renovated rooms in the College to provide suitable accommodation. The number of pupils increased from 20 in 1875 to 178 in 1881. Under Edwards , pupils gained great success at Oxford and Cambridge. However, his wardenship did see almost total neglect of the Welsh language. Motivated by a misguided belief that academic success as measured in the public school league was irreconcilable with the study of the Welsh language, Edwards utterly ignored the provisions of the trust deed. For many years Llandovery College was to follow the path laid out by its fifth warden. Edwards moved to Carmarthen in 1885, later to become Archbishop of Wales.