Secretary of the Interior Caleb B. Smith was the only cabinet member who joined Seward in explicitly calling for a withdrawal from Sumter. Smith asserted that Sumter could not be successfully defended and that its evacuation was "a necessity."
Smith, however, rejected General Scott's recommendation about Fort Pickens. Instead, he called for "the most vigorous measures for the defense" of Pickens and other forts. He explained that the country could accept the abandonment of Sumter if they understood that it was done "from necessity," and if at the same time Pickens and other forts in the government's possession "shall be defended, and the power of the government vindicated."
Bibliography: Lincoln, Works, eds. Nicolay and Hay, 6: 229-30.